A pre-training method and related methods and devices

Abstract

This invention provides a pre-training method and related methods and devices. The pre-training method includes: acquiring a sequence of character information and a sequence of phoneme information corresponding to a training text, and alignment information between the character information sequence and the phoneme information sequence at the whole-word level; combining the alignment information, performing a mixed information sequence at the whole-word level to obtain a mixed information sequence, wherein, during the mixed processing, only one type of information, character information or phoneme information, is retained for the same whole word; and training an initial language model based on the mixed information sequence. Because this invention pre-trains the language model based on a mixed information sequence containing both character information and phoneme information, the language model can learn both pronunciation information and semantic information through training, resulting in a language model with better representational capabilities.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a pre-training method and related methods and devices. BACKGROUND

[0002] The general way to obtain a task model (such as a speech synthesis model) based on an encoder-decoder framework is to first obtain an encoder through pre-training, that is, to train an initial language model, and then to use the trained language model as the encoder to build a model including the encoder and a decoder, and then to fine-tune the built model using training data of a specified task, thereby obtaining a final task model (for example, fine-tuning the built model using training data of a speech synthesis task to obtain a speech synthesis model).

[0003] The current pre-training method is generally a pre-training method based on phoneme information, that is, a sequence of phoneme information corresponding to training text is obtained, and the initial language model is trained using the sequence of phoneme information corresponding to the training text. However, when the language model is trained using the sequence of phoneme information corresponding to the training text, the language model can only learn pronunciation information and cannot learn other important information, which leads to poor results when the trained language model is used as an encoder to apply to a downstream task (such as a speech synthesis task). SUMMARY

[0004] Therefore, the present application provides a pre-training method and related methods and devices to solve the problem that the language model can only learn pronunciation information and cannot learn other important information when the language model is pre-trained using the current pre-training method, which leads to poor results when the trained language model is used as an encoder to apply to a downstream task. The technical solution is as follows:

[0005] A pre-training method comprises:

[0006] obtaining a sequence of grapheme information and a sequence of phoneme information corresponding to training text, and alignment information between the sequence of grapheme information and the sequence of phoneme information at the word level, wherein each grapheme information in the sequence of grapheme information is information of a segmented unit obtained by segmenting the training text, and each phoneme information in the sequence of phoneme information is phoneme information of a whole word in the training text;

[0007] combining the alignment information, performing mixed processing on the sequence of grapheme information and the sequence of phoneme information at the word level to obtain a sequence of mixed information, wherein when performing the mixed processing, only one of the grapheme information and the phoneme information is retained for the same whole word;

[0008] The initial language model is trained based on the mixed information sequence.

[0009] Optionally, obtain the character information sequence and phoneme information sequence corresponding to the training text, including:

[0010] The training text is segmented at the sub-word level. For each sub-word in the segmented sub-word sequence, the sub-word is mapped to an identifier representing the position of the sub-word in the dictionary, which serves as the character identifier corresponding to the sub-word. The sequence composed of the character identifiers corresponding to each sub-word in the sub-word sequence is determined as the character information sequence corresponding to the training text.

[0011] For each whole word in the training text, the phoneme sequence corresponding to the whole word is obtained, and each phoneme in the phoneme sequence corresponding to the whole word is mapped to an identifier representing the position of the phoneme in the dictionary. The resulting identifier sequence is used as the phoneme identifier corresponding to the whole word. The sequence composed of the phoneme identifiers corresponding to each whole word in the training text is determined as the phoneme information sequence corresponding to the training text.

[0012] The dictionary includes several word segmentation methods and phoneme sequences corresponding to each whole word.

[0013] Optionally, obtaining the alignment information between the character information sequence and the phoneme information sequence at the whole word level includes:

[0014] The position information of the whole word corresponding to each character information in the character information sequence in the training text is obtained to obtain a first position information sequence, and the position information of the whole word corresponding to each phoneme information in the phoneme information sequence in the training text is obtained to obtain a second position information sequence.

[0015] The first position information sequence and the second position information sequence are determined as the alignment information between the character information sequence and the phoneme information sequence at the whole word level.

[0016] Optionally, the step of combining the alignment information to perform mixed processing of the character information sequence and the phoneme information sequence at the whole word level includes:

[0017] Based on the alignment information, a first mask is randomly generated;

[0018] Based on the first mask, the character information sequence and the phoneme information sequence are mixed at the whole word level, wherein the first mask is used to determine the combination method of character information in the character information sequence and phoneme information in the phoneme information sequence.

[0019] Optionally, the pre-training method further includes:

[0020] Obtain the punctuation information sequence corresponding to the training text, wherein the punctuation information sequence contains the punctuation information of the training text at the whole word level;

[0021] The training of the initial language model based on the mixed information sequence includes:

[0022] The initial language model is trained based on the mixed information sequence and the punctuation information sequence.

[0023] Optionally, obtaining the punctuation information sequence corresponding to the training text includes:

[0024] Obtain the punctuation information of each word in the word sequence obtained by segmenting the training text at the word granularity, wherein the punctuation information of a word is the punctuation information of the whole word to which the word belongs;

[0025] The punctuation information of each subword in the subword sequence is mapped to an identifier representing the position of the punctuation information in the dictionary, thereby obtaining the punctuation identifier corresponding to each subword in the subword sequence, wherein the dictionary includes several punctuation information;

[0026] The sequence consisting of the punctuation marks corresponding to each word in the word sequence is determined as the punctuation information sequence corresponding to the training text.

[0027] Optionally, training the initial language model based on the mixed information sequence and the punctuation information sequence includes:

[0028] At the whole-word level, partial information in the mixed information sequence and partial information in the punctuation information sequence are masked respectively, and the masked mixed information sequence is used as the first target information sequence;

[0029] The missing punctuation information indicating no punctuation in the masked punctuation information sequence is restored, and the restored punctuation information sequence is used as the second target information sequence.

[0030] The initial language model is trained using the first target information sequence and the second target information sequence.

[0031] Optionally, the step of masking partial information in the mixed information sequence and partial information in the punctuation information sequence at the whole-word level includes:

[0032] Based on the alignment information, a second mask is randomly generated;

[0033] Based on the second mask, partial information in the mixed information sequence is masked at the whole word level, and partial information in the punctuation information sequence is masked at the whole word level based on the second mask.

[0034] Optionally, training the initial language model using the first target information sequence and the second target information sequence includes:

[0035] Obtain the representation vector of the first target information sequence and the representation vector of the second target information sequence;

[0036] The representation vector of the first target information sequence and the representation vector of the second target information sequence are fused and then input into the initial language model for encoding to obtain the encoding result;

[0037] Based on the encoding result, the missing information in the first target information sequence is predicted to obtain a first prediction result, and based on the encoding result, the missing information in the second target information sequence is predicted to obtain a second prediction result;

[0038] Based on the first prediction result and the second prediction result, the parameters of the initial language model are updated.

[0039] Optionally, updating the parameters of the initial language model based on the first prediction result and the second prediction result includes:

[0040] Based on the first prediction result and the missing actual information in the first target information sequence, the first prediction loss is determined;

[0041] Based on the second prediction result and the missing actual information in the second target information sequence, the second prediction loss is determined;

[0042] The first predicted loss and the second predicted loss are fused to obtain the fused loss;

[0043] Based on the fusion loss, the parameters of the initial language model are updated.

[0044] A method for obtaining a speech synthesis model, comprising:

[0045] The initial language model is pre-trained using any of the pre-training methods described above to obtain a pre-trained language model.

[0046] The pre-trained language model is used as an encoder to construct a model including the encoder and decoder, which serves as the initial speech synthesis model.

[0047] The initial speech synthesis model is trained using training data from a speech synthesis task to obtain the final speech synthesis model.

[0048] A speech synthesis method, comprising:

[0049] Obtain the target text;

[0050] Based on the speech synthesis model, the speech corresponding to the target text is synthesized, wherein the speech synthesis model is obtained using the aforementioned method for obtaining the speech synthesis model.

[0051] A pre-training device includes: a first information acquisition module, an information processing module, and a model training module;

[0052] The first information acquisition module is used to acquire the character information sequence and phoneme information sequence corresponding to the training text, as well as the alignment information of the character information sequence and the phoneme information sequence at the whole word level, wherein each character information in the character information sequence is the information of a word segmentation unit obtained by segmenting the training text, and each phoneme information in the phoneme information sequence is the phoneme information of a whole word in the training text;

[0053] The information processing module is used to combine the alignment information to perform mixed processing on the character information sequence and the phoneme information sequence at the whole word level to obtain a mixed information sequence. During the mixed processing, only one type of information, character information or phoneme information, is retained for the same whole word.

[0054] The model training module is used to train the initial language model based on the mixed information sequence.

[0055] Optionally, the pre-training device further includes: a second information acquisition module;

[0056] The second information acquisition module is used to acquire the punctuation information sequence corresponding to the training text, wherein the punctuation information sequence includes the punctuation information of the training text at the whole word level;

[0057] When the model training module trains the initial language model based on the mixed information sequence, it is specifically used to train the initial language model based on the mixed information sequence and the punctuation information sequence.

[0058] A processing device, comprising: a memory and a processor;

[0059] The memory is used to store programs;

[0060] The processor is configured to execute the program to implement each step of the pre-training method described above, and / or to implement each step of the speech synthesis model acquisition method described above, and / or to implement each step of the speech synthesis method described above.

[0061] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the pre-training method described above, and / or implements the steps of the speech synthesis model acquisition method described above, and / or implements the steps of the speech synthesis method described above.

[0062] The pre-training method provided by this invention first obtains the character information sequence and phoneme information sequence corresponding to the training text, as well as the alignment information of the character information sequence and phoneme information sequence at the whole word level. Then, combining the alignment information, the character information sequence and phoneme information sequence are mixed at the whole word level to obtain a mixed information sequence containing character information and phoneme information. Finally, the initial language model is trained based on the mixed information sequence. Pre-training the language model based on phoneme information allows the language model to learn pronunciation information, while pre-training based on character information allows the language model to learn semantic information. Because this invention pre-trains the language model based on a mixed information sequence containing both character information and phoneme information, the language model can learn both pronunciation information and semantic information through training, resulting in a language model with better representational capabilities.

[0063] Based on the pre-training method provided in this invention, a method for obtaining a speech synthesis model is also provided. This method first pre-trains an initial language model using the pre-training method provided in this invention. Then, the pre-trained language model is used as an encoder to construct a model including an encoder and a decoder. Finally, the constructed model is trained using training data from a speech synthesis task to obtain the final speech synthesis model. Because the encoder in the speech synthesis model is pre-trained using the pre-training method provided in this invention, it has good representational capabilities, thus resulting in good performance of the speech synthesis model. Based on the method for obtaining the speech synthesis model provided in this invention, this invention also provides a speech synthesis method. Because this method uses the speech synthesis model obtained by the method for obtaining the speech synthesis model provided in this invention to synthesize the speech corresponding to the target text, it can obtain synthesized speech of good quality. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0065] Figure 1 This is a schematic diagram of a hardware architecture involved in the present invention;

[0066] Figure 2 A schematic flowchart of the pre-training method provided in an embodiment of the present invention;

[0067] Figure 3 A flowchart illustrating a specific example of the pre-training method provided in an embodiment of the present invention;

[0068] Figure 4 This is a schematic diagram illustrating the process of obtaining a first target information sequence and training a language model using the first target information sequence, as provided in an embodiment of the present invention.

[0069] Figure 5 A flowchart illustrating another specific example of the pre-training method provided in this embodiment of the invention;

[0070] Figure 6 This is a schematic diagram illustrating the process of obtaining a first target information sequence and a second target information sequence, and training a language model using the first target information sequence and the second target information sequence, as provided in an embodiment of the present invention.

[0071] Figure 7 This is a schematic diagram of the pre-training device provided in an embodiment of the present invention;

[0072] Figure 8 This is a schematic diagram of the processing device provided in an embodiment of the present invention. Detailed Implementation

[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] In the process of realizing this case, the inventors discovered that when training a language model using the phoneme information sequence corresponding to the training text, the language model can only learn pronunciation information and cannot learn semantic information. Semantic information is relatively important, and the lack of semantic information leads to poor performance when the trained language model is used as an encoder for downstream tasks (such as speech synthesis tasks).

[0075] In order to enable the language model to learn semantic information, the inventors of this case conducted research. The initial idea was to use a pre-training method based on character information, that is, to train the language model using the character information sequence corresponding to the training text. However, through research, it was found that when the language model is trained using the character information sequence corresponding to the training text, the language model can only learn semantic information and cannot learn pronunciation information. The lack of pronunciation information will also lead to poor performance when the trained language model is used as an encoder for downstream tasks.

[0076] When a language model is trained using the phoneme information sequence corresponding to the training text, the language model can learn pronunciation information. When a language model is trained using the character information sequence corresponding to the training text, the language model can learn semantic information. The inventors of this case thought of combining phoneme information and character information. Following this idea, the inventors came up with a pre-training method based on phoneme information and character information. That is, the phoneme information sequence corresponding to the training text and the character information sequence corresponding to the training text are input into the language model one after the other. During input, a masking process is performed. During masking, the character information and phoneme information of the same word are masked at the same time.

[0077] The inventors of this case studied a pre-training method based on phoneme and character information and found that although the method combines phoneme and character information, the large gap between the two makes it difficult for the language model to learn a good representation. When the trained language model is used as an encoder for downstream tasks, the effect is still unsatisfactory.

[0078] In view of this, the inventors of this case conducted further research and, through continuous research, finally proposed a pre-training method with better results. The basic concept of this pre-training method is to mix the character information sequence and the phoneme information sequence corresponding to the training text to obtain a mixed information sequence containing both character and phoneme information. The language model is then trained based on this mixed information sequence. In addition, to further improve the training effect, the punctuation information sequence corresponding to the training text can be combined for training. During the training process, some information in the mixed information sequence and the punctuation information sequence is masked. The model parameters are updated based on the prediction loss of two prediction tasks (first, predicting the masked information in the mixed information sequence; second, predicting the masked information in the punctuation information sequence).

[0079] Before introducing the pre-training method provided by this invention, the hardware architecture involved in this invention will be described first.

[0080] In one possible implementation, such as Figure 1 As shown, the hardware architecture involved in this invention may include: electronic device 101 and server 102.

[0081] For example, electronic device 101 can be any electronic product that can interact with a user through one or more means such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as personal computer, laptop, tablet, mobile phone, smart TV, etc.

[0082] It should be noted that, Figure 1 This is just one example; there can be many types of electronic devices, not limited to... Figure 1 The laptop in the middle.

[0083] For example, server 102 can be a single server, a server cluster consisting of multiple servers, or a cloud computing server center. Server 102 may include processors, memory, and network interfaces, etc.

[0084] For example, electronic device 101 can establish a connection and communicate with server 102 through a wireless communication network; for example, electronic device 101 can establish a connection and communicate with server 102 through a wired network.

[0085] Electronic device 101 can acquire training data and send the training data to server 102. Server 102 uses the acquired training data to pre-train the initial language model according to the pre-training method provided by the present invention.

[0086] In another possible implementation, the hardware architecture involved in this invention may include an electronic device. The electronic device is one with strong data processing capabilities.

[0087] For example, an electronic device can be any electronic product that can interact with a user through one or more means such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as a personal computer, laptop, tablet, mobile phone, smart TV, etc.

[0088] The electronic device can acquire training data and use the acquired training data to pre-train the initial language model according to the pre-training method provided in this invention.

[0089] Those skilled in the art should understand that the above-described electronic devices and servers are merely examples, and other existing or future electronic devices or servers that are applicable to this invention should also be included within the scope of protection of this invention, and are hereby incorporated by reference.

[0090] The pre-training method provided by the present invention will be described in the following embodiments.

[0091] First Embodiment

[0092] Please see Figure 2 The diagram illustrates a flowchart of a pre-training method provided in an embodiment of the present invention, which may include:

[0093] Step S201: Obtain the character information sequence and phoneme information sequence corresponding to the training text, as well as the alignment information of the character information sequence and the phoneme information sequence corresponding to the training text at the whole word level.

[0094] The character element information sequence corresponding to the training text represents the sequence of segmentation units obtained by segmenting the training text. Each character element in the character element information sequence corresponding to the training text represents the segmentation unit obtained by segmenting the training text. In other words, the character element information sequence corresponding to the training text includes the representation information of each segmentation unit obtained by segmenting the training text. When segmenting the training text, segmentation can be performed at the granularity of whole words or at the granularity of sub-words. Considering that sub-words are between whole words and characters, they can better balance the OOV (out-of-vocabulary, meaning words that did not appear during training but appeared during testing) problem, segmentation can be performed at the granularity of sub-words. It should be noted that if the training text is segmented at the whole-word level, the segmentation result is a sequence of whole words, and the segmentation unit mentioned above is the whole word. If the training text is segmented at the sub-word level, the segmentation result is a sequence of sub-words, and the segmentation unit mentioned above is the sub-word. It should also be noted that sub-words are obtained by segmenting whole words. A sub-word may be a part of a whole word or the whole word itself, depending on the segmentation method of the whole word. For example, segmenting the whole word "St-Petersburg" results in "St##-##Petersburg", and the sub-words "St", "-", and "Petersburg" are all part of the whole word "St-Petersburg". As another example, segmenting the whole word "of" results in "of", and the sub-word "of" is the whole word "of" itself.

[0095] Among them, the phoneme information sequence representation corresponding to the training text is the pronunciation information of each whole word in the training text. Each phoneme information in the phoneme information sequence corresponding to the training text is the representation information of each phoneme in the phoneme sequence corresponding to a whole word in the training text. That is, the phoneme information sequence corresponding to the training text includes the representation information of each phoneme in the phoneme sequence corresponding to each whole word in the training text.

[0096] Step S202: Combine the alignment information of the character information sequence and the phoneme information sequence corresponding to the training text at the whole word level, and perform mixed processing on the character information sequence and the phoneme information sequence corresponding to the training text at the whole word level to obtain a mixed information sequence.

[0097] In the process of mixing information, for the same whole word, only one type of information—either its character information or its phonetic information—is retained. That is, the mixed information sequence either contains the character information or the phonetic information of the whole word. It should be noted that the choice between retaining character information and phonetic information for the same whole word can be random.

[0098] For example, if the training text includes 5 whole words, namely word1, word2, word3, word4, and word5, then after mixing the grapheme information sequence corresponding to the training text with the phoneme information sequence corresponding to the training text at the whole word level, one possible form of the resulting mixed information sequence includes the grapheme information of word1, the phoneme information of word2, the phoneme information of word3, the grapheme information of word4, and the phoneme information of word5.

[0099] Step S203: Train the initial language model based on the mixed information sequence.

[0100] In one possible implementation, the language model can be the MPG BERT model. Of course, this embodiment is not limited to this, that is, other language models are also possible.

[0101] The language model is trained multiple times using multiple different training texts in the manner described above, until the training termination condition is met (e.g., the model converges, or the preset number of training iterations is reached).

[0102] The pre-training method provided in this embodiment of the invention first obtains the character information sequence and phoneme information sequence corresponding to the training text, as well as the alignment information of the character information sequence and phoneme information sequence at the whole word level. Then, combining the alignment information, the character information sequence and phoneme information sequence are mixed at the whole word level to obtain a mixed information sequence containing character information and phoneme information. Finally, the initial language model is trained based on the mixed information sequence. Pre-training the language model based on phoneme information allows the language model to learn pronunciation information, while pre-training based on character information allows the language model to learn semantic information. Because the pre-training method provided in this embodiment of the invention pre-trains the language model based on the mixed information sequence containing character information and phoneme information, through training, the language model can learn both pronunciation information and semantic information, resulting in a language model with better language representation capabilities.

[0103] Second Embodiment

[0104] Based on the first embodiment described above, this embodiment provides a specific example of a pre-training method.

[0105] Please see Figure 3 The flowchart illustrating a specific example of the pre-training method provided in this embodiment is shown and may include:

[0106] Step S301: Obtain the character information sequence and phoneme information sequence corresponding to the training text, as well as the alignment information of the character information sequence and the phoneme information sequence corresponding to the training text at the whole word level.

[0107] In this embodiment, each character information in the character information sequence corresponding to the training text is the information of a segmentation unit obtained by segmenting the training text into words. That is, the character information sequence corresponding to the training text includes the information of each segmentation unit obtained by segmenting the training text into words.

[0108] The process of obtaining the sequence of character information corresponding to the training text may include:

[0109] Step a1: Segment the training text at the sub-word level to obtain the sub-word sequence corresponding to the training text.

[0110] For example, the training text is "The weather of St-Petersburg was perfect again around ten knots of wind and sunny." Sub-word segmentation of this training text yields "CLSThe weather of St##-##Petersburg was perfect again around ten knots of wind and sunny##.SEP," where "CLS" indicates the beginning of the sentence and "SEP" indicates the end. In this example, the sub-word segmentation result for "The," "weather," and "of" is the same as the sub-word segmentation result for "St-Petersburg," and so on. It should be noted that the above sub-word segmentation results for each whole word are merely examples.

[0111] Step a2: For each subword in the subword sequence obtained by segmentation, map the subword to an identifier that represents the position of the subword in the dictionary, which serves as the character identifier corresponding to the subword.

[0112] In this embodiment, the character identifiers corresponding to the sub-words are used as the representation information of the sub-words.

[0113] In this embodiment, the dictionary is pre-built. In one possible implementation, the dictionary is built by obtaining an existing dictionary containing several words and their corresponding segmentation methods (such as a bert-base-cased dictionary) as an initial dictionary, and then adding a corresponding phoneme sequence to each word in the initial dictionary. For example, adding the phoneme sequence "ss ah nn ii" to the word "sunny" to obtain the final dictionary.

[0114] In the initial dictionary, a large amount of text data is acquired when adding the corresponding phoneme sequence for each word. Then, dirty data such as emoticons (e.g., ":") are removed from this text data. The text data after removing dirty data is then normalized (e.g., "10" in English text data is processed into "ten"). Next, the G2P (grapheme to phoneme) rule is used to transform the normalized text data to obtain the phoneme sequence corresponding to each whole word in the text data. Finally, based on the phoneme sequence corresponding to each whole word in these text data, the corresponding phoneme sequence is added to each word in the initial dictionary.

[0115] It should be noted that the training text in this embodiment is training text for a specified language, and the dictionary in this embodiment is a dictionary for a specified language. This dictionary should, as far as possible, encompass all whole words in the specified language, as well as the segmentation methods and phoneme sequences corresponding to each whole word.

[0116] Each sub-word obtained by segmenting the whole word according to the corresponding segmentation method can be mapped to its position in the dictionary. For example, "CLS" can be mapped to 1, "SEP" can be mapped to 2, and "sunny" can be mapped to 203. After segmenting the training text at the sub-word level to obtain the sub-word sequence, each sub-word in the sub-word sequence can be mapped to an identifier representing the position of that sub-word in the dictionary.

[0117] For example, in the subword sequence "CLS The weather of St##-##Petersburgwas perfect again around ten knots of wind and sunny##.SEP" corresponding to the training text, "CLS" can be mapped to "1", with "1" serving as the character identifier for "CLS", "The" can be mapped to "223", with "223" serving as the character identifier for "The", "weather" can be mapped to "259", with "259" serving as the character identifier for "weather", and so on. "." can be mapped to "200", with "200" serving as the character identifier for ".", and "SEP" can be mapped to "2", with "2" serving as the character identifier for "SEP". It should be noted that punctuation marks are also mapped as subwords.

[0118] Step a3: Determine the sequence consisting of the character identifiers corresponding to each sub-word in the sub-word sequence corresponding to the training text as the character information sequence corresponding to the training text.

[0119] For example, the sub-word sequence corresponding to the training text is “CLS The weather of St##-##Petersburg was perfect again around ten knots of wind and sunny##.SEP”, and the sequence composed of the character identifiers corresponding to each sub-word in this sub-word sequence is [1 223 259 231 233 305247 235 216218 249 228 261 220 235 208 203 200 2], which serves as the character information sequence corresponding to the training text.

[0120] In this embodiment, each phoneme in the phoneme information sequence corresponding to the training text is the phoneme information of a whole word in the training text, that is, the phoneme information sequence corresponding to the training text includes the phoneme information of each whole word in the training text.

[0121] The process of obtaining the phoneme information sequence corresponding to the training text may include:

[0122] Step b1: For each whole word in the training text, obtain the phoneme sequence corresponding to the whole word, and map each phoneme in the phoneme sequence corresponding to the whole word to an identifier representing the position of the phoneme in the dictionary. The resulting identifier sequence is used as the phoneme identifier corresponding to the whole word.

[0123] It should be noted that each phoneme in the phoneme sequence corresponding to each whole word can be mapped to an identifier representing its position in the dictionary. For example, the phoneme sequence corresponding to "sunny" is "ss ah nn ii". In the phoneme sequence "ss ah nnii", the phoneme "ss" can be mapped to 78, the phoneme "ah" can be mapped to 60, the phoneme "nn" can be mapped to 85, and the phoneme "ii" can be mapped to 98. 78, 60, 85, and 98 can be regarded as the representation information of each phoneme in the phoneme sequence corresponding to "sunny". In this embodiment, [7860 85 98] is used as the phoneme identifier corresponding to the whole word "sunny".

[0124] Step b2: Determine the sequence composed of the phoneme identifiers corresponding to each whole word in the training text as the phoneme information sequence corresponding to the training text.

[0125] To facilitate subsequent processing, it is also necessary to obtain the alignment information between the pixel information sequence and the phoneme information sequence corresponding to the training text. Specifically, the process of obtaining the alignment information between the pixel information sequence and the phoneme information sequence corresponding to the training text may include:

[0126] Step c1-a: Obtain the position information of the whole word corresponding to each character information in the training text in the training text, so as to obtain the first position information sequence.

[0127] For example, the sequence of character information corresponding to the training text is [1 223 259 231 233 305 247 235 216 218 249 228 261 220 235 208 203 200]. [2], where the word corresponding to the character information "1" is "CLS", and "CLS" is the 0th word in the training text. Therefore, the position information of the word corresponding to the character information "1" in the training text is "0". The word corresponding to the character information "223" is "The", and "The" is the 1st word in the training text. Therefore, the position information of the word corresponding to the character information "223" in the training text is "1". The word corresponding to the character information "259" is "weather", and "weather" is the 2nd word in the training text. Therefore, the position information of the word corresponding to the character information "259" in the training text is "2". The word corresponding to the character information "231" is "of". As the third word in the training text, the position information of the whole word corresponding to the character information "231" in the training text is "3". The whole word corresponding to the character information "233" is "St-Petersburg". "St-Petersburg" is the fourth word in the training text, so the position information of the whole word corresponding to the character information "233" in the training text is "4". Since the whole words corresponding to the character information "305" and the character information "247" are also "St-Petersburg", the position information of the whole words corresponding to the character information "305" and the character information "247" in the training text is also "4". And so on, the final first position information sequence is [0 1 2 3 4 4 456 7 8 9 10 11 12 13 14 14 15].

[0128] Step c1-b: Obtain the position information of the whole word corresponding to each phoneme in the training text in the training text to obtain the second position information sequence.

[0129] For example, the whole word corresponding to the phoneme information [78 60 85 98] is "sunny". "sunny" is the 14th whole word in the training text. Therefore, the position information of the whole word corresponding to the phoneme information [78 60 85 98] in the training text is [14 14 1414]. The position information of the whole word corresponding to each phoneme information in the phoneme information sequence corresponding to the training text is used to form the second position information sequence.

[0130] Step c2: Determine the alignment information between the first position information sequence and the second position information sequence as the pixel information sequence and phoneme information sequence corresponding to the training text.

[0131] Step S302: Combine the alignment information of the character information sequence and the phoneme information sequence corresponding to the training text at the whole word level, and perform mixed processing on the character information sequence and the phoneme information sequence corresponding to the training text at the whole word level to obtain a mixed information sequence.

[0132] In the process of mixed processing, only one type of information, either character information or phoneme information, is retained for the same whole word.

[0133] Specifically, the process of combining the alignment information of the character information sequence and the phoneme information sequence corresponding to the training text at the whole-word level, and then performing mixed processing on the character information sequence and the phoneme information sequence corresponding to the training text at the whole-word level, can include:

[0134] Step d1: Generate the first mask based on the alignment information of the character information sequence and the phoneme information sequence corresponding to the training text at the whole word level.

[0135] The first mask can be randomly generated based on the alignment information of the sequence of character information corresponding to the training text and the sequence of phoneme information corresponding to the training text at the whole word level.

[0136] Step d2: Based on the first mask, the sequence of character information corresponding to the training text and the sequence of phoneme information corresponding to the training text are mixed at the whole word level to obtain a mixed information sequence.

[0137] For adjacent whole words A and B in the training text, there are four possible combinations of phoneme information and character information at the whole word level: (character information of whole word A, character information of whole word B), (character information of whole word A, phoneme information of whole word B), (phoneme information of whole word A, character information of whole word B), and (character information of whole word A, character information of whole word B). When mixing the character information sequence and the phoneme information sequence corresponding to the training text at the whole word level based on the first mask, the combination method of phoneme information and character information is determined based on the first mask. Since the first mask is randomly generated, the final choice of which of the above four combination methods is random.

[0138] For example, “and sunny##.” is the result of word segmentation of “and sunny.” at the whole-word level. Its corresponding phoneme sequence is “ac nn dd ss ah nn ii”. At the whole-word level, phoneme information and character information are combined in the following four ways: “and sunny##.” (character information, character information), “and ss ah nn ii” (character information, phoneme information), “ac nn dd sunny##.” (phoneme information, character information), and “ac nn dd ss ah nn ii” (phoneme information, phoneme information). The above four combination forms can be distinguished by 0 / 1 strings. “and sunny##.” can be represented as [00 0 0 0 0 0 0 0 0], where the first three 0s of [0 0 0 0 0 0 0 0 0 0] represent “and”, the fourth 0 represents a space, the fifth to ninth 0s represent “sunny”, and the last 0 represents “.”. “and ss ah "nn ii" can be represented as [0 0 0 0 1 1 1 1], where the first three 0s in [0 0 0 0 11 1 1] represent "and", the fourth 0 represents a space, the first 1 represents "ss", the second 1 represents "ah", the third 1 represents "nn", and the fourth 1 represents "ii". "ac nn dd sunny##." can be represented as [1 1 1 0 0 0 00 0 0], where the first three 1s in [1 1 1 0 0 0 0 0 0 0] represent "ac", "nn", and "dd" respectively, the first 0 represents a space, the second to sixth 0s represent "sunny", and the last 0 represents ".". "ac nn dd ss ah nn ii" can be represented as [1 1 1 0 1 11 1], where [1 1 1 0 1 1 1] represents "and", the fourth 0 represents a space, the fifth 0 represents "sunny", and the sixth 0 represents ".". The first three 1s in [1] represent "ac", "nn", and "dd" respectively, and 0 represents a space. The four 1s after 0 represent "ss", "ah", "nn", and "ii" respectively. When mixing phoneme information and character information at the whole word level, the phoneme information and character information are marked with a 0 / 1 string at the whole word level based on the first mask. For example, if the 0 / 1 string is marked as [0 0 0 0 1 1 11], it means that the combination form is "and sunny##.", that is, (character information, phoneme information).

[0139] Step S303: At the whole word level, partially mask out the information in the mixed information sequence, and use the masked information sequence as the target information sequence.

[0140] The process of masking out part of the mixed information sequence at the whole word level can include: generating a second mask based on the alignment information between the pixel information sequence and the phoneme information sequence corresponding to the training text, and masking out part of the mixed information sequence based on the second mask. In one possible implementation, the second mask can be randomly generated based on the alignment information between the pixel information sequence and the phoneme information sequence corresponding to the training text, thus making the masking of which whole word information is random.

[0141] Step S304: Train the initial language model using the target information sequence.

[0142] Specifically, such as Figure 4 As shown, the process of training the initial language model using the target information sequence includes: first, obtaining the representation vector of the target information sequence; then, inputting the representation vector of the target information sequence into the initial language model for encoding to obtain the encoding result; next, predicting the missing information in the target information sequence based on the encoding result to obtain the prediction result; finally, determining the prediction loss based on the prediction result and the actual missing information in the target information sequence, and updating the parameters of the initial language model based on the prediction loss.

[0143] The language model is trained multiple times using multiple different training texts in the manner described above, until the training termination condition is met (e.g., the model converges, or the preset number of training iterations is reached).

[0144] The pre-training method provided in this embodiment of the invention first obtains the character information sequence and phoneme information sequence corresponding to the training text, as well as the alignment information of the character information sequence and phoneme information sequence at the whole word level. Then, combining the alignment information, the character information sequence and phoneme information sequence are mixed at the whole word level to obtain a mixed information sequence containing character information and phoneme information. Next, at the whole word level, some information in the mixed information sequence is masked, and the masked information sequence is used as the target information sequence. Finally, the target information sequence is used, along with a prediction task (predicting the missing information in the target information sequence), to train the initial language model. Pre-training the language model based on phoneme information allows the language model to learn pronunciation information, while pre-training based on character information allows the language model to learn semantic information. Since the pre-training method provided in this embodiment of the invention pre-trains the language model based on a mixed information sequence containing both character information and phoneme information, through training, the language model can learn both pronunciation information and semantic information, resulting in a language model with good representational capabilities.

[0145] Third Embodiment

[0146] Based on the first embodiment described above, this embodiment provides another specific example of the pre-training method. The difference between this specific example and the specific example provided in the second embodiment is that, in addition to mixing character information and phoneme information and using the mixed information sequence for training, punctuation information with prosodic boundary significance is also introduced. On this basis, multi-task prediction is performed to improve the training effect.

[0147] Please see Figure 5 The flowchart illustrating a specific example of the pre-training method provided in this embodiment is shown and may include:

[0148] Step S501: Obtain the character information sequence, phoneme information sequence, and punctuation information sequence corresponding to the training text, as well as the alignment information between the character information sequence and the phoneme information sequence at the whole word level.

[0149] Among them, the character information sequence corresponding to the training text represents the sequence of word segmentation units obtained by segmenting the training text into words. The character information sequence corresponding to the training text includes the representation information of each word segmentation unit obtained by segmenting the training text into words. The phoneme information sequence corresponding to the training text represents the pronunciation information of each whole word in the training text. The phoneme information sequence corresponding to the training text includes the representation information of each phoneme in the phoneme sequence corresponding to each whole word in the training text.

[0150] The specific process of obtaining the character information sequence and phoneme information sequence corresponding to the training text, as well as the alignment information of the character information sequence and the phoneme information sequence corresponding to the training text at the whole word level, can be found in the relevant part of the second embodiment, and will not be repeated here.

[0151] The process of obtaining the punctuation information sequence corresponding to the training text may include:

[0152] Step e1: Obtain the punctuation information of each word in the word sequence obtained by segmenting the training text at the word granularity.

[0153] Among them, the punctuation information of a sub-word is the punctuation information of the whole word to which the sub-word belongs.

[0154] For example, the word sequence obtained by segmenting the training text at the word level is "CLS The weatherof St##-##Petersburg was perfect again around ten knots of wind and sunny##.SEP". If "pcx" represents non-punctuation information and "pc" is used to concatenate the corresponding punctuation to represent punctuation information, then the punctuation information of "CLS" is represented by "pcx", the punctuation information of "The" is represented by "pcx", ..., the punctuation information of "." is represented by "pc.", and the punctuation information of "SEP" is represented by "pcx". Finally, we can obtain [pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pc.pcx].

[0155] Step e2: Map the punctuation information of each subword in the subword sequence to an identifier representing the position of that punctuation information in the dictionary, and obtain the punctuation identifier corresponding to each subword in the subword sequence.

[0156] In this embodiment, the dictionary includes not only segmentation methods and phoneme sequences corresponding to several whole words, but also punctuation information. The punctuation information included in the dictionary may include "pcx", "pc.", "pc,", "pc.", etc. The dictionary in this embodiment is also pre-built. Specifically, the dictionary in this embodiment is built by obtaining an existing dictionary containing several words and their corresponding segmentation methods (such as a bert-base-cased dictionary) as the initial dictionary, and then further adding the corresponding phoneme sequence to each word in the initial dictionary, as well as adding all possible punctuation information, thereby obtaining the final dictionary.

[0157] After obtaining the punctuation information of each subword in the subword sequence, for each subword, the punctuation information of the subword can be mapped to an identifier representing the position of the punctuation information of the subword in the dictionary. The mapped identifier is used as the punctuation identifier corresponding to the subword. For example, "pcx" is mapped to "199" and "pc." is mapped to "200".

[0158] Step e3: Determine the sequence consisting of the punctuation marks corresponding to each subword in the subword sequence as the punctuation information sequence corresponding to the training text.

[0159] For example, mapping [pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pcx pc.pcx] based on the dictionary yields [199 199 199 199 199 199 199 199 199 199 199 199 199 199 199 200 199], and [199 199 199 199 199 199 199 199 199 199 199 199 199 199 200 199] serves as the punctuation information sequence corresponding to the training text.

[0160] Step S502: Combine the alignment information of the character information sequence and the phoneme information sequence corresponding to the training text at the whole word level, and perform mixed processing on the character information sequence and the phoneme information sequence corresponding to the training text at the whole word level to obtain a mixed information sequence.

[0161] In the process of mixed processing, only one type of information, either character information or phoneme information, is retained for the same whole word.

[0162] The specific implementation process of step S502 can be found in the specific implementation process of step S302 in the second embodiment, and will not be repeated here.

[0163] Step S503: At the whole word level, partially mask out the information in the mixed information sequence, and use the masked information sequence as the first target information sequence. At the whole word level, partially mask out the punctuation information in the punctuation information sequence corresponding to the training text.

[0164] Specifically, a second mask can be generated based on the alignment information between the pixel information sequence and the phoneme information sequence of the training text. This second mask is used to mask some information in the mixed information sequence at the whole-word level, and also to mask some punctuation information in the punctuation information sequence of the training text at the whole-word level. In one possible implementation, the second mask can be randomly generated based on the alignment information between the pixel information sequence and the phoneme information sequence of the training text; thus, which whole-word information is masked is random.

[0165] It should be noted that when masking the punctuation information sequence corresponding to the training text, the punctuation information sequence corresponding to the training text is first preprocessed based on the mixed information sequence to make the preprocessed punctuation information sequence correspond in form to the mixed information sequence. For example, if part of the information in the mixed information sequence corresponds to "and ss ah nn ii", then the punctuation information corresponding to that part is processed into the form [pcx pcx pcx pcx pc.pc.pc.pc.], that is, [199 199199 199 200 200 200 200]. After obtaining the preprocessed punctuation information sequence, the preprocessed punctuation information sequence is then masked.

[0166] Step S504: Restore the missing punctuation information indicating no punctuation in the punctuation information sequence after masking, and use the restored punctuation information sequence as the second target information sequence.

[0167] Since the purpose of this embodiment is to enable the model to learn prosodic boundary information, the focus is on punctuation information indicating punctuation marks. That is, it is desirable to only mask out punctuation information indicating punctuation marks. However, when masking part of the punctuation information in the punctuation information sequence corresponding to the training text in step S503, it is very likely that punctuation information indicating no punctuation marks will be masked out. In view of this, after masking part of the punctuation information in the punctuation information sequence corresponding to the training text at the whole word level based on the second mask, the missing punctuation information indicating no punctuation marks in the masked punctuation information sequence is restored (that is, the punctuation information indicating no punctuation marks in each masked punctuation information is unmasked). Specifically, a third mask can be generated, and the masked punctuation information sequence can be further processed based on the third mask so that only the punctuation information indicating punctuation marks is masked out in the further processed punctuation information sequence, that is, only the punctuation information indicating punctuation marks is missing in the further processed punctuation information sequence.

[0168] Step S505: Train the initial language model using the first target information sequence and the second target information sequence.

[0169] like Figure 6As shown, the process of training an initial language model using a first target information sequence and a second target information sequence can include: First, obtaining the representation vector of the first target information sequence and the representation vector of the second target information sequence; then, fusing the representation vectors of the first and second target information sequences to obtain a fused vector; next, inputting the fused vector into the initial language model for encoding to obtain an encoding result; then, predicting the missing information in the first target information sequence based on the encoding result, and using the obtained prediction result as the first prediction result; and then predicting the missing information in the second target information sequence based on the encoding result, and using the obtained prediction result as the second prediction result; finally, updating the parameters of the initial language model based on the first and second prediction results.

[0170] In one possible implementation, the process of fusing the representation vector of the first target information sequence with the representation vector of the second target information sequence may include concatenating the representation vector of the first target information sequence with the representation vector of the second target information sequence.

[0171] In one possible implementation, the process of updating the parameters of the initial language model based on the first prediction result and the second prediction result may include: firstly, determining the first prediction loss based on the first prediction result and the missing actual information in the first target information sequence, and determining the second prediction loss based on the second prediction result and the missing actual information in the second target information sequence; then, fusing the first prediction loss and the second prediction loss to obtain the fused loss; and finally, updating the parameters of the initial language model based on the fused loss.

[0172] There are several ways to integrate the first prediction loss and the second prediction loss. In one possible implementation, the first prediction loss and the second prediction loss can be summed directly. In another possible implementation, the first prediction loss and the second prediction loss can be summed with weights. The weights corresponding to the first prediction loss and the second prediction loss can be set according to the actual situation.

[0173] The pre-training method provided in this embodiment of the invention first obtains the character information sequence, phoneme information sequence, and punctuation information sequence corresponding to the training text, as well as the alignment information of the character information sequence and phoneme information sequence at the whole word level. Then, combining the alignment information, the character information sequence and phoneme information sequence are mixed at the whole word level to obtain a mixed information sequence containing both character information and phoneme information. Next, at the whole word level, some information in the mixed information sequence is masked, and the masked information sequence is used as the first target information sequence. At the whole word level, some punctuation information in the punctuation information sequence corresponding to the training text is masked. Then, with the goal of only masking punctuation information indicating punctuation, the masked punctuation information sequence is further processed, and the punctuation information sequence obtained from the further processing is used as the second target information sequence. Finally, the first target information sequence and the second target information sequence are used, along with two prediction tasks (predicting the missing information in the first target information sequence and predicting the missing information in the second target information sequence), to train the initial language model. Pre-training a language model based on phoneme information enables the model to learn pronunciation information. Pre-training a language model based on character information enables the model to learn semantic information. Since the pre-training method provided in this embodiment of the invention pre-trains the language model based on a mixed information sequence containing both character and phoneme information, the language model can learn both pronunciation and semantic information through training. This results in a language model with good representational ability. In addition, the introduction of punctuation information with prosodic boundary meaning and punctuation prediction tasks during the pre-training process enables the model to learn prosodic boundary information, thereby improving the model's representational ability.

[0174] Fourth embodiment

[0175] Based on the pre-training method provided in the above embodiments, this embodiment provides a method for obtaining a speech synthesis model, which may include:

[0176] Step f1: Use the pre-training method provided in the above embodiment to pre-train the initial language model to obtain the pre-trained language model.

[0177] Step f2: Use the pre-trained language model as an encoder to construct a model including an encoder and a decoder, which serves as the initial speech synthesis model.

[0178] Step f3: Use the training data of the speech synthesis task to train the initial speech synthesis model to obtain the final speech synthesis model.

[0179] By fine-tuning the initial speech synthesis model using training data from a speech synthesis task, a better-performing speech synthesis model can be obtained.

[0180] The speech synthesis model acquisition method provided in this embodiment can yield a speech synthesis model with better performance.

[0181] Fifth embodiment

[0182] Based on the method for obtaining the speech synthesis model provided in the above embodiments, this embodiment provides a speech synthesis method, which may include:

[0183] Step g1: Obtain the target text.

[0184] Step g2: Based on the speech synthesis model, synthesize the speech corresponding to the target text.

[0185] The speech synthesis model is obtained using the speech synthesis model acquisition method provided in the above embodiments.

[0186] The speech synthesis method provided in this embodiment can synthesize speech of relatively good quality.

[0187] Sixth Embodiment

[0188] This invention also provides a pre-training device. The pre-training device provided in this invention is described below. The pre-training device described below can be referred to in correspondence with the pre-training method described above.

[0189] Please see Figure 7 The diagram shows a pre-training device provided in an embodiment of the present invention. The pre-training device may include: a first information acquisition module 701, an information processing module 702, and a model training module 703.

[0190] The first information acquisition module 701 is used to acquire the character information sequence and phoneme information sequence corresponding to the training text, as well as the alignment information of the character information sequence and the phoneme information sequence at the whole word level.

[0191] Wherein, each character information in the character information sequence is the information of a word segmentation unit obtained by segmenting the training text, and each phoneme information in the phoneme information sequence is the phoneme information of a whole word in the training text;

[0192] The information processing module 702 is used to combine the alignment information and perform mixed processing on the character information sequence and the phoneme information sequence at the whole word level to obtain a mixed information sequence. During the mixed processing, only one type of information, character information or phoneme information, is retained for the same whole word.

[0193] The model training module 703 is used to train the initial language model based on the mixed information sequence.

[0194] In one possible implementation, the first information acquisition module 701, when acquiring the character information sequence and phoneme information sequence corresponding to the training text, is specifically used for:

[0195] The training text is segmented at the sub-word level. For each sub-word in the segmented sub-word sequence, the sub-word is mapped to an identifier representing the position of the sub-word in the dictionary, which serves as the character identifier corresponding to the sub-word. The sequence composed of the character identifiers corresponding to each sub-word in the sub-word sequence is determined as the character information sequence corresponding to the training text.

[0196] For each whole word in the training text, the phoneme sequence corresponding to the whole word is obtained, and each phoneme in the phoneme sequence corresponding to the whole word is mapped to an identifier representing the position of the phoneme in the dictionary. The resulting identifier sequence is used as the phoneme identifier corresponding to the whole word. The sequence composed of the phoneme identifiers corresponding to each whole word in the training text is determined as the phoneme information sequence corresponding to the training text.

[0197] The dictionary includes several word segmentation methods and phoneme sequences corresponding to each whole word.

[0198] In one possible implementation, when the first information acquisition module 701 acquires the alignment information of the character information sequence and the phoneme information sequence at the whole word level, it is specifically used for:

[0199] The position information of the whole word corresponding to each character information in the character information sequence in the training text is obtained to obtain a first position information sequence, and the position information of the whole word corresponding to each phoneme information in the phoneme information sequence in the training text is obtained to obtain a second position information sequence.

[0200] The first position information sequence and the second position information sequence are determined as the alignment information between the character information sequence and the phoneme information sequence at the whole word level.

[0201] In one possible implementation, when the information processing module 702 combines the alignment information to perform mixed processing of the character information sequence and the phoneme information sequence at the whole-word level, it is specifically used for:

[0202] Based on the alignment information, a first mask is randomly generated;

[0203] Based on the first mask, the character information sequence and the phoneme information sequence are mixed at the whole word level, wherein the first mask is used to determine the combination method of character information in the character information sequence and phoneme information in the phoneme information sequence.

[0204] In one possible implementation, the pre-training device provided in this embodiment of the invention may further include: a second information acquisition module 704.

[0205] The second information acquisition module 704 is used to acquire the punctuation information sequence corresponding to the training text, wherein the punctuation information sequence includes the punctuation information of the training text at the whole word level.

[0206] When training the initial language model based on the mixed information sequence, the model training module 703 is specifically used to train the initial language model based on the mixed information sequence and the punctuation information sequence.

[0207] In one possible implementation, when training the initial language model based on the mixed information sequence and the punctuation information sequence, the model training module 703 is specifically used for:

[0208] At the whole-word level, partial information in the mixed information sequence and partial information in the punctuation information sequence are masked respectively, and the masked mixed information sequence is used as the first target information sequence;

[0209] The missing punctuation information indicating no punctuation in the masked punctuation information sequence is restored, and the restored punctuation information sequence is used as the second target information sequence.

[0210] The initial language model is trained using the first target information sequence and the second target information sequence.

[0211] In one possible implementation, when the model training module 703 performs masking processing on partial information in the mixed information sequence and partial information in the punctuation information sequence at the whole-word level, it is specifically used for:

[0212] Based on the alignment information, a second mask is randomly generated;

[0213] Based on the second mask, partial information in the mixed information sequence is masked at the whole word level, and partial information in the punctuation information sequence is masked at the whole word level based on the second mask.

[0214] In one possible implementation, when the model training module 703 trains the initial language model using the first target information sequence and the second target information sequence, it specifically performs the following:

[0215] Obtain the representation vector of the first target information sequence and the representation vector of the second target information sequence;

[0216] The representation vector of the first target information sequence and the representation vector of the second target information sequence are fused and then input into the initial language model for encoding to obtain the encoding result;

[0217] Based on the encoding result, the missing information in the first target information sequence is predicted to obtain a first prediction result, and based on the encoding result, the missing information in the second target information sequence is predicted to obtain a second prediction result;

[0218] Based on the first prediction result and the second prediction result, the parameters of the initial language model are updated.

[0219] In one possible implementation, when updating the parameters of the initial language model based on the first prediction result and the second prediction result, the model training module 703 is specifically used for:

[0220] Based on the first prediction result and the missing actual information in the first target information sequence, the first prediction loss is determined;

[0221] Based on the second prediction result and the missing actual information in the second target information sequence, the second prediction loss is determined;

[0222] The first predicted loss and the second predicted loss are fused to obtain the fused loss;

[0223] Based on the fusion loss, the parameters of the initial language model are updated.

[0224] Pre-training a language model based on phoneme information enables the model to learn pronunciation information. Pre-training a language model based on character information enables the model to learn semantic information. Since the pre-training method and apparatus provided in this embodiment of the invention pre-trains the language model based on a mixed information sequence containing both character and phoneme information, the language model can learn both pronunciation and semantic information through training. This results in a language model with good representational ability. In addition, the introduction of punctuation information with prosodic boundary meaning and punctuation prediction tasks during the pre-training process enables the model to learn prosodic boundary information, thereby improving the model's representational ability.

[0225] Seventh Embodiment

[0226] This invention also provides a processing device; please refer to [link / reference]. Figure 8 The diagram shows the structure of the processing device, which may include: a processor 801, a communication interface 802, a memory 803, and a communication bus 804.

[0227] In this embodiment of the invention, the number of processor 801, communication interface 802, memory 803, and communication bus 804 is at least one, and processor 801, communication interface 802, and memory 803 communicate with each other through communication bus 804.

[0228] The processor 801 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0229] The memory 803 may include high-speed RAM, or it may also include non-volatile memory, such as at least one disk storage device;

[0230] The memory stores a program, which the processor can call. The program is used for:

[0231] Obtain the character information sequence and phoneme information sequence corresponding to the training text, as well as the alignment information of the character information sequence and the phoneme information sequence at the whole word level. Each character information in the character information sequence is the information of a word segmentation unit obtained by segmenting the training text, and each phoneme information in the phoneme information sequence is the phoneme information of a whole word in the training text.

[0232] Combining the alignment information, the character information sequence and the phoneme information sequence are mixed at the whole word level to obtain a mixed information sequence. During the mixing process, only one type of information, character information or phoneme information, is retained for the same whole word.

[0233] The initial language model is trained based on the mixed information sequence.

[0234] Optionally, the refined and extended functions of the program can be found in the description above.

[0235] Eighth embodiment

[0236] This invention also provides a readable storage medium that stores a program suitable for execution by a processor, the program being used for:

[0237] Obtain the character information sequence and phoneme information sequence corresponding to the training text, as well as the alignment information of the character information sequence and the phoneme information sequence at the whole word level. Each character information in the character information sequence is the information of a word segmentation unit obtained by segmenting the training text, and each phoneme information in the phoneme information sequence is the phoneme information of a whole word in the training text.

[0238] Combining the alignment information, the character information sequence and the phoneme information sequence are mixed at the whole word level to obtain a mixed information sequence. During the mixing process, only one type of information, character information or phoneme information, is retained for the same whole word.

[0239] The initial language model is trained based on the mixed information sequence.

[0240] Optionally, the refined and extended functions of the program can be found in the description above.

[0241] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0242] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0243] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pre-training method, characterized in that, The method comprises the following steps: obtaining a sequence of grapheme information and a sequence of phoneme information corresponding to a training text, and alignment information of the sequence of grapheme information and the sequence of phoneme information at a word level, wherein each grapheme information in the sequence of grapheme information is information of a subword unit obtained by segmenting the training text, and each phoneme information in the sequence of phoneme information is phoneme information of a whole word in the training text, the subword unit is a subword or a whole word, and the granularity of the subword is between a whole word and a character; combining the alignment information, performing mixed processing on the sequence of grapheme information and the sequence of phoneme information at the word level to obtain a sequence of mixed information, wherein, when performing the mixed processing, only one of the grapheme information and the phoneme information is retained for the same whole word; training an initial language model based on the sequence of mixed information, wherein the language model learns pronunciation information based on the phoneme information, and the language model learns semantic information based on the grapheme information.

2. The pre-training method of claim 1, wherein, The method comprises the following steps: segmenting the training text at a subword granularity, mapping each subword in a sequence of subwords obtained by the segmentation to an identifier representing the position of the subword in a dictionary as a grapheme identifier corresponding to the subword, and determining a sequence composed of the grapheme identifiers corresponding to the subwords in the sequence of subwords as a sequence of grapheme information corresponding to the training text; for each whole word in the training text, obtaining a phoneme sequence corresponding to the whole word, mapping each phoneme in the phoneme sequence corresponding to the whole word to an identifier representing the position of the phoneme in the dictionary to obtain an identifier sequence as a phoneme identifier corresponding to the whole word, and determining a sequence composed of the phoneme identifiers corresponding to the whole words in the training text as a sequence of phoneme information corresponding to the training text; wherein the dictionary comprises a plurality of subword segmentation manners and phoneme sequences corresponding to a plurality of whole words respectively.

3. The pre-training method of claim 1, wherein, The method comprises the following steps: obtaining position information of each whole word corresponding to each grapheme information in the sequence of grapheme information in the training text to obtain a first sequence of position information, and obtaining position information of each whole word corresponding to each phoneme information in the sequence of phoneme information in the training text to obtain a second sequence of position information; determining the first sequence of position information and the second sequence of position information as alignment information of the sequence of grapheme information and the sequence of phoneme information at the word level.

4. The pre-training method of claim 1, wherein, The method comprises the following steps: generating a first mask randomly based on the alignment information; based on the first mask, performing mixed processing on the sequence of grapheme information and the sequence of phoneme information at the word level, wherein the first mask is used to determine the combination mode of the grapheme information in the sequence of grapheme information and the phoneme information in the sequence of phoneme information.

5. The pre-training method of any one of claims 1-4, wherein, The method further comprises the following steps: obtaining a punctuation information sequence corresponding to the training text, wherein the punctuation information sequence comprises punctuation information of the training text at a whole word level; the training of the initial language model based on the mixed information sequence comprises: training of the initial language model based on the mixed information sequence and the punctuation information sequence.

6. The pre-training method of claim 5, wherein, the obtaining of the punctuation information sequence corresponding to the training text comprises: obtaining punctuation information of each subword in a subword sequence obtained by performing subword granularity segmentation on the training text, wherein the punctuation information of a subword is the punctuation information of a whole word to which the subword belongs; mapping the punctuation information of each subword in the subword sequence to an identifier representing the position of the punctuation information in a dictionary to obtain a punctuation identifier corresponding to each subword in the subword sequence, wherein the dictionary comprises a plurality of punctuation information; a sequence composed of the punctuation identifiers corresponding to the subwords in the subword sequence is determined as the punctuation information sequence corresponding to the training text.

7. The pre-training method of claim 5, wherein, the training of the initial language model based on the mixed information sequence and the punctuation information sequence comprises: performing masking processing on part of the information in the mixed information sequence and part of the information in the punctuation information sequence at the whole word level, respectively, the mixed information sequence after the masking processing serving as a first target information sequence; performing recovery processing on the missing punctuation information indicating no punctuation in the punctuation information sequence after the masking processing, the punctuation information sequence after the recovery processing serving as a second target information sequence; training of the initial language model using the first target information sequence and the second target information sequence.

8. The pre-training method of claim 7, wherein, the performing of the masking processing on part of the information in the mixed information sequence and part of the information in the punctuation information sequence at the whole word level comprises: randomly generating a second mask based on the alignment information; performing masking processing on part of the information in the mixed information sequence at the whole word level based on the second mask, and performing masking processing on part of the information in the punctuation information sequence at the whole word level based on the second mask.

9. The pre-training method of claim 7, wherein, the training of the initial language model using the first target information sequence and the second target information sequence comprises: obtaining a representation vector of the first target information sequence and a representation vector of the second target information sequence; inputting the representation vector of the first target information sequence and the representation vector of the second target information sequence after fusion into the initial language model for encoding to obtain an encoding result; based on the encoding result, predicting the missing information in the first target information sequence to obtain a first prediction result, and based on the encoding result, predicting the missing information in the second target information sequence to obtain a second prediction result; performing parameter updating of the initial language model based on the first prediction result and the second prediction result.

10. The pre-training method of claim 9, wherein, the performing of the parameter updating of the initial language model based on the first prediction result and the second prediction result comprises: determining a first prediction loss based on the first prediction result and actual information missing in the first target information sequence; determining a second prediction loss based on the second prediction result and actual information missing in the second target information sequence; and performing parameter updating of the initial language model based on the first prediction loss and the second prediction loss. determine a second prediction loss based on the second prediction result and actual information missing in the second target information sequence; fuse the first prediction loss and the second prediction loss to obtain a fused loss; update a parameter of an initial language model based on the fused loss.

11. A method of acquiring a speech synthesis model, characterized by, The method comprises: pre-training an initial language model by using the pre-training method according to any one of claims 1-10 to obtain a pre-trained language model; using the pre-trained language model as an encoder to construct an initial speech synthesis model comprising the encoder and a decoder model; training the initial speech synthesis model by using training data of a speech synthesis task to obtain a final speech synthesis model.

12. A speech synthesis method characterized by, The method comprises: obtaining a target text; synthesizing speech corresponding to the target text based on a speech synthesis model, wherein the speech synthesis model is obtained by using the method for obtaining a speech synthesis model according to claim 11.

13. A pre-training device, characterized by The method comprises: a first information obtaining module, an information processing module, and a model training module; The first information obtaining module is configured to obtain a grapheme information sequence and a phoneme information sequence corresponding to a training text, and alignment information between the grapheme information sequence and the phoneme information sequence at a word level, wherein each grapheme information in the grapheme information sequence is information of a tokenization unit obtained by tokenizing the training text, each phoneme information in the phoneme information sequence is phoneme information of a whole word in the training text, and the tokenization unit is a sub-word or a whole word, and the granularity of the sub-word is between that of a whole word and that of a character; The information processing module is configured to combine the alignment information to perform mixed processing on the grapheme information sequence and the phoneme information sequence at the word level to obtain a mixed information sequence, wherein, during the mixed processing, only one kind of information between grapheme information and phoneme information is retained for a same whole word; The model training module is configured to train an initial language model based on the mixed information sequence, wherein the language model learns pronunciation information based on the phoneme information, and the language model learns semantic information based on the grapheme information.

14. The pre-training apparatus according to claim 13, characterized in that, The method further comprises: a second information obtaining module; The second information obtaining module is configured to obtain a punctuation information sequence corresponding to the training text, wherein the punctuation information sequence comprises punctuation information of the training text at the word level; The model training module is configured to train the initial language model based on the mixed information sequence and the punctuation information sequence when training the initial language model based on the mixed information sequence.

15. A processing device, characterized by The method comprises: a memory and a processor; The memory is configured to store a program; The processor is configured to execute the program to implement each step of the pre-training method according to any one of claims 1-10, and / or implement each step of the method for obtaining a speech synthesis model according to claim 11, and / or implement each step of the speech synthesis method according to claim 12.

16. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements each step of the pre-training method according to any one of claims 1-10, and / or implements each step of the method for obtaining a speech synthesis model according to claim 11, and / or implements each step of the speech synthesis method according to claim 12.

Images