Polyphonetic character pronunciation prediction network training method, speech generation method and device
By constructing the target vocabulary list and sample text identification conversion technology, the problem of low accuracy in pronunciation determination of polyphonic pronunciation is solved, efficient pronunciation prediction of polyphonic pronunciation is achieved, and the accuracy and storage efficiency of pronunciation are improved.
Patent Information
- Application Number
- CN202210712858.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-22
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-06-22
AI Technical Summary
In pronunciation synthesis, the accuracy of pronunciation of polyphonic characters is low. The existing methods require a lot of storage space and are easy to misidentify, making it difficult to quickly and accurately determine the pronunciation of polyphonic characters in the text context.
By constructing a target vocabulary list, different text identifiers are assigned to polyphonic characters with different pronunciations, and the sample text is converted into sample text identifiers, and input them into the polyphonic word pronunciation prediction network for training. The pronunciation prediction results and labels are used for network training to improve the Accuracy of pronunciation prediction of polyphonic characters.
It improves the accuracy of pronunciation prediction of polyphonic characters, saves storage space, simplifies the training process, and only relies on polyphonic characters to predict the prediction network without additional neural networks, improving the accuracy of speech generation.
Smart Images

Figure CN115273809B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a training method for a polyphonetic character pronunciation prediction network, a speech generation method and a device. Background Art
[0002] In speech synthesis, the text to be generated needs to be recognized for pronunciation first to determine the pronunciation of the text, and then the audio corresponding to the text is synthesized based on the determined pronunciation of the text. In the case where the text contains polyphones, it is necessary to determine the pronunciation of the polyphones in the context of the text.
[0003] In the related art, the pronunciations of polyphones in different phrases are generally preset, and the pronunciations of polyphones are determined by identifying phrases in the text. However, this method requires a large amount of storage space to store the pronunciations of polyphones in different phrases, and it is easy for polyphones and adjacent Chinese characters to be mistaken for preset phrases, and the accuracy rate is low. Therefore, how to quickly and accurately determine the pronunciations of polyphones in text has become a problem that needs to be solved urgently. Summary of the invention
[0004] The embodiments of the present disclosure at least provide a method for training a polyphonetic character pronunciation prediction network, a method for generating speech, and a device.
[0005] In a first aspect, an embodiment of the present disclosure provides a method for training a polyphonetic character pronunciation prediction network, comprising:
[0006] Acquire a sample text containing polyphones and a label corresponding to the sample text, wherein the label corresponding to the sample text is used to represent a text identifier of the polyphone in the sample text in a pre-constructed target vocabulary, and different pronunciations of the same Chinese character in the target vocabulary correspond to different text identifiers;
[0007] Converting the sample text into a corresponding sample text identifier; wherein different pronunciations of the same Chinese character in the sample text identifier correspond to the same text identifier;
[0008] Inputting the sample text identifier into the polyphone pronunciation prediction network to be trained to obtain a pronunciation prediction result corresponding to the sample text; wherein the pronunciation prediction result is used to characterize the probability that each character contained in the sample text is the Chinese character corresponding to each text identifier in the target vocabulary;
[0009] The polyphone pronunciation prediction network to be trained is trained based on the pronunciation prediction result and the label corresponding to the sample text.
[0010] In the above method, in the target vocabulary, polyphones with different pronunciations correspond to different text identifiers, so the different pronunciations of polyphones can be distinguished by text identifiers; when the sample text is converted into the corresponding sample text identifier, since the text identifiers corresponding to different pronunciations of the same Chinese character in the sample text identifier are the same, the polyphone pronunciation prediction network can predict the different pronunciations of the same character, and further, the polyphone pronunciation prediction network to be trained is trained according to the pronunciation prediction results and the labels corresponding to the sample texts (i.e., the text identifiers of the correct pronunciations corresponding to the polyphones), thereby improving the ability of the polyphone pronunciation prediction network to classify polyphones as correct pronunciations and improving the accuracy of the polyphone pronunciation prediction network in predicting the pronunciations of polyphones.
[0011] In a possible implementation, the polyphonetic characters in the sample text carry pronunciation marks corresponding to the pronunciations;
[0012] Obtaining the label corresponding to the sample text includes:
[0013] Based on the target vocabulary and the pronunciation marks of the polyphonetic characters in the sample text, text identifiers of the polyphonetic characters in the sample text are determined.
[0014] In a possible implementation, the polyphonetic character pronunciation prediction network includes an initial feature extraction module, an encoding module and a probability prediction module, and the polyphonetic character pronunciation prediction network is a pre-trained model;
[0015] Before inputting the sample text identifier into the polyphonetic character pronunciation prediction network to be trained, the method further includes initializing the polyphonetic character pronunciation prediction network to be trained according to the following method:
[0016] Determining initial parameters corresponding to the target Chinese character in the polyphonetic character pronunciation prediction network; wherein the target Chinese character corresponds to multiple pronunciations;
[0017] The parameters of the fully connected layer and the masked language layer of the polyphone pronunciation prediction network are adjusted based on the initial parameters corresponding to the target Chinese character.
[0018] By adopting this structure, training can be performed on the basis of an existing pre-trained natural language model, and the entire polyphone pronunciation prediction network can be directly trained without accessing other neural networks, thereby improving the training efficiency.
[0019] In a possible implementation, adjusting the parameters of the fully connected layer and the masked language layer of the polyphonetic character pronunciation prediction network based on the initial parameters corresponding to the target Chinese character includes:
[0020] Using the initial parameters corresponding to the target Chinese characters as the initial parameters corresponding to the updated Chinese characters; wherein the updated Chinese characters are the target Chinese characters carrying different pronunciation marks;
[0021] The initial parameters corresponding to the updated Chinese characters are added to the parameters of the fully connected layer and the masked language layer of the polyphonetic character pronunciation prediction network.
[0022] Since the target Chinese character and the updated Chinese character are the same Chinese character, the features of the target Chinese character and the updated Chinese character are relatively similar, so the updated Chinese character inherits the weight and bias parameters of the target Chinese character. Compared with randomly initializing the weight and bias parameters of the updated Chinese character, the number of training times of the polyphone pronunciation prediction network can be reduced, thereby improving training efficiency.
[0023] In a possible implementation manner, converting the sample text into a corresponding sample text identifier includes:
[0024] Determine a mask corresponding to the sample text, and replace the polyphonetic characters with pronunciation marks in the sample text based on the mask;
[0025] Eliminate pronunciation marks in the sample text based on the mask;
[0026] The sample text with the pronunciation mark removed is converted into a corresponding sample text identifier.
[0027] By adopting this method, the polyphones with different pronunciations in the sample text can be converted into text tags that do not distinguish pronunciations, so that the polyphone pronunciation prediction network can perform feature extraction on polyphones that do not carry pronunciation tags, avoid the influence of the pronunciation tags of polyphones on feature extraction, and improve the accuracy of polyphone pronunciation prediction.
[0028] In a possible implementation, the method further includes constructing the target vocabulary according to the following method:
[0029] Obtaining an initial vocabulary; wherein the initial vocabulary includes text identifiers corresponding to each Chinese character;
[0030] Determining polyphonetic characters in the initial vocabulary and the respective pronunciations of the polyphonetic characters;
[0031] Adding pronunciation marks to the polyphones under different pronunciations, and determining the text identifiers corresponding to the polyphones under different pronunciations;
[0032] Based on the polyphones with pronunciation marks added, and the text identifiers corresponding to the polyphones under different pronunciations, the initial vocabulary is updated to obtain the target vocabulary.
[0033] By using this method, polyphones can be split into multiple Chinese characters according to different pronunciations, so that the polyphones with different pronunciations can be converted into different text identifiers based on the target vocabulary, thereby training the polyphone pronunciation prediction network to predict different pronunciations.
[0034] In a possible implementation manner, converting the sample text into a corresponding sample text identifier includes:
[0035] Based on the target vocabulary, the sample text is converted into a corresponding sample text identifier.
[0036] In a possible implementation manner, the polyphonetic character pronunciation prediction network to be trained is a pre-trained network;
[0037] The training of the polyphonetic character pronunciation prediction network to be trained based on the pronunciation prediction result and the label corresponding to the sample text includes:
[0038] Determine a target prediction result corresponding to the polyphonetic character in the pronunciation prediction result;
[0039] The polyphone pronunciation prediction network to be trained is trained based on the target prediction result and the label corresponding to the sample text.
[0040] By adopting this method, the polyphonetic character pronunciation prediction network can be trained through the prediction results of the polyphonetic characters (i.e., the target prediction results) of the polyphonetic character pronunciation prediction network, thereby improving the accuracy of the polyphonetic character pronunciation prediction network in predicting polyphonetic characters.
[0041] In a possible implementation manner, the pronunciation prediction result includes a probability matrix;
[0042] The determining of the target prediction result corresponding to the polyphonetic character in the pronunciation prediction result includes:
[0043] Determining a probability vector corresponding to the polyphonetic character in the probability matrix;
[0044] The training of the polyphone pronunciation prediction network to be trained based on the target prediction result and the label corresponding to the sample text includes:
[0045] The polyphone pronunciation prediction network to be trained is trained based on the probability vector corresponding to the polyphone and the label corresponding to the sample text.
[0046] In a second aspect, an embodiment of the present disclosure provides a speech generation method, including:
[0047] Get the text to be detected;
[0048] Based on the polyphonetic character pronunciation prediction network trained by the polyphonetic character pronunciation prediction network training method described in the first aspect or any possible implementation method of the first aspect, the text to be detected is recognized to obtain a corresponding text detection result in the text to be detected, wherein the text detection result includes the text identifiers corresponding to each Chinese character in the text to be detected in the target vocabulary;
[0049] Determining a target pronunciation corresponding to the text detection result based on a predetermined pronunciation mapping relationship;
[0050] A target speech is synthesized based on the target pronunciation.
[0051] In the above method, the text to be detected can be obtained, and the text to be detected can be input into the polyphone pronunciation prediction network, and the pronunciation of the polyphone in the text to be detected can be accurately predicted based on the polyphone pronunciation prediction network, so that the target pronunciation can be determined based on the text detection result containing the accurate polyphone pronunciation, and the correct target speech can be generated. This method improves the accuracy of the target speech, and the pronunciation prediction is performed only by the polyphone pronunciation prediction network, without the need for other additional neural networks, saving storage space and improving the efficiency of pronunciation prediction.
[0052] In a third aspect, the present disclosure also provides a training device for a polyphonetic character pronunciation prediction network, comprising:
[0053] A first acquisition module is used to acquire a sample text containing polyphones and a label corresponding to the sample text, wherein the label corresponding to the sample text is used to represent a text identifier of the polyphone in the sample text in a pre-constructed target vocabulary, and different pronunciations of the same Chinese character in the target vocabulary correspond to different text identifiers;
[0054] A conversion module, used for converting the sample text into a corresponding sample text identifier; wherein different pronunciations of the same Chinese character in the sample text identifier correspond to the same text identifier;
[0055] A prediction module, used for inputting the sample text identifier into the polyphone pronunciation prediction network to be trained to obtain a pronunciation prediction result corresponding to the sample text; wherein the pronunciation prediction result is used to characterize the probability that each character contained in the sample text is the Chinese character corresponding to each text identifier in the target vocabulary;
[0056] A training module is used to train the polyphone pronunciation prediction network to be trained based on the pronunciation prediction results and the labels corresponding to the sample texts.
[0057] In a possible implementation, the polyphonetic characters in the sample text carry pronunciation marks corresponding to the pronunciations;
[0058] The first acquisition module, when acquiring the label corresponding to the sample text, is used to:
[0059] Based on the target vocabulary and the pronunciation marks of the polyphonetic characters in the sample text, text identifiers of the polyphonetic characters in the sample text are determined.
[0060] In a possible implementation, the polyphonetic character pronunciation prediction network includes an initial feature extraction module, an encoding module and a probability prediction module, and the polyphonetic character pronunciation prediction network is a pre-trained model;
[0061] Before inputting the sample text identifier into the polyphone pronunciation prediction network to be trained, the prediction module is further used to initialize the polyphone pronunciation prediction network to be trained:
[0062] Determining initial parameters corresponding to the target Chinese character in the polyphonetic character pronunciation prediction network; wherein the target Chinese character corresponds to multiple pronunciations;
[0063] The parameters of the fully connected layer and the masked language layer of the polyphone pronunciation prediction network are adjusted based on the initial parameters corresponding to the target Chinese character.
[0064] In a possible implementation, the prediction module, when adjusting the parameters of the fully connected layer and the masked language layer of the polyphonetic character pronunciation prediction network based on the initial parameters corresponding to the target Chinese character, is used to:
[0065] Using the initial parameters corresponding to the target Chinese characters as the initial parameters corresponding to the updated Chinese characters; wherein the updated Chinese characters are the target Chinese characters carrying different pronunciation marks;
[0066] The initial parameters corresponding to the updated Chinese characters are added to the parameters of the fully connected layer and the masked language layer of the polyphonetic character pronunciation prediction network.
[0067] In a possible implementation manner, the conversion module, when converting the sample text into a corresponding sample text identifier, is used to:
[0068] Determine a mask corresponding to the sample text, and replace the polyphonetic characters with pronunciation marks in the sample text based on the mask;
[0069] Eliminate pronunciation marks in the sample text based on the mask;
[0070] The sample text with the pronunciation mark removed is converted into a corresponding sample text identifier.
[0071] In a possible implementation manner, the first acquisition module is further configured to construct the target vocabulary according to the following method:
[0072] Obtaining an initial vocabulary; wherein the initial vocabulary includes text identifiers corresponding to each Chinese character;
[0073] Determining polyphonetic characters in the initial vocabulary and the respective pronunciations of the polyphonetic characters;
[0074] Adding pronunciation marks to the polyphones under different pronunciations, and determining the text identifiers corresponding to the polyphones under different pronunciations;
[0075] Based on the polyphones with pronunciation marks added, and the text identifiers corresponding to the polyphones under different pronunciations, the initial vocabulary is updated to obtain the target vocabulary.
[0076] In a possible implementation manner, the conversion module, when converting the sample text into a corresponding sample text identifier, is used to:
[0077] Based on the target vocabulary, the sample text is converted into a corresponding sample text identifier.
[0078] In a possible implementation manner, the polyphonetic character pronunciation prediction network to be trained is a pre-trained network;
[0079] The training module, when training the polyphone pronunciation prediction network to be trained based on the pronunciation prediction result and the label corresponding to the sample text, is used to:
[0080] Determine a target prediction result corresponding to the polyphonetic character in the pronunciation prediction result;
[0081] The polyphone pronunciation prediction network to be trained is trained based on the target prediction result and the label corresponding to the sample text.
[0082] In a possible implementation manner, the pronunciation prediction result includes a probability matrix;
[0083] The training module, when determining the target prediction result corresponding to the polyphone in the pronunciation prediction result, is used to:
[0084] Determining a probability vector corresponding to the polyphonetic character in the probability matrix;
[0085] The training module is used to train the polyphone pronunciation prediction network to be trained based on the target prediction result and the label corresponding to the sample text:
[0086] The polyphone pronunciation prediction network to be trained is trained based on the probability vector corresponding to the polyphone and the label corresponding to the sample text.
[0087] In a fourth aspect, the present disclosure also provides a speech generating device, including:
[0088] The second acquisition module is used to acquire the text to be detected;
[0089] A recognition module, for recognizing the text to be detected based on the polyphonetic character pronunciation prediction network trained by the polyphonetic character pronunciation prediction network training method described in the first aspect or any possible implementation method of the first aspect, and obtaining a corresponding text detection result in the text to be detected, wherein the text detection result includes the text identifiers corresponding to each Chinese character in the text to be detected in the target vocabulary;
[0090] A determination module, used to determine the target pronunciation corresponding to the text detection result based on a predetermined pronunciation mapping relationship;
[0091] A synthesis module is used to synthesize a target speech based on the target pronunciation.
[0092] In a fifth aspect, an embodiment of the present disclosure further provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the first aspect or any possible implementation of the first aspect are executed, or the steps of the second aspect or any possible implementation of the second aspect are executed.
[0093] In a sixth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the steps of the above-mentioned first aspect, or any possible implementation of the first aspect, or executes the above-mentioned second aspect, or any possible implementation of the second aspect.
[0094] For a description of the effects of the training of the polyphonetic character pronunciation prediction network, the speech generation device, the computer equipment, and the computer-readable storage medium, please refer to the description of the training method and the speech generation method of the polyphonetic character pronunciation prediction network, which will not be repeated here.
[0095] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure.
[0096] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0097] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.
[0098] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.
[0099] Figure 1 A flowchart of a method for training a polyphonetic character pronunciation prediction network provided by an embodiment of the present disclosure is shown;
[0100] Figure 2 A schematic diagram showing the overall training process of a polyphonetic character pronunciation prediction network provided by an embodiment of the present disclosure;
[0101] Figure 3 A flow chart of a speech generation method provided by an embodiment of the present disclosure is shown;
[0102] Figure 4 A schematic diagram showing a process of recognizing a text to be detected provided by an embodiment of the present disclosure;
[0103] Figure 5 A schematic diagram of the architecture of a training device for a polyphonetic character pronunciation prediction network provided by an embodiment of the present disclosure is shown;
[0104] Figure 6 A schematic diagram of the architecture of a speech generating device provided by an embodiment of the present disclosure is shown;
[0105] Figure 7 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown;
[0106] Figure 8 A schematic diagram of the structure of another computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0107] To make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part rather than all of the embodiments of the present disclosure. Components of the embodiments of the present disclosure generally described and illustrated in the drawings here can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the claimed present disclosure, but merely represents selected embodiments of the present disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without creative efforts fall within the scope of protection of the present disclosure.
[0108] When performing speech synthesis, it is necessary to first perform pronunciation recognition on the text of the speech to be generated to determine the pronunciation of the text, and then synthesize the audio corresponding to the text according to the determined pronunciation of the text. In the case where the text contains polyphonic characters, it is necessary to determine the pronunciation of the polyphonic character in the text context.
[0109] In the related art, generally, the pronunciations of polyphonic characters in different phrases are preset, and the pronunciation of the polyphonic character is determined by recognizing the phrases in the text. For example, it is preset that the character "泊" is pronounced as "po1" in the phrase "湖泊" ("po" represents a pinyin letter, and "1" represents that the tone of the pronunciation is the first tone), and is pronounced as "bo2" in the phrase "漂泊" ("bo" represents a pinyin letter, and "2" represents that the tone of the pronunciation is the second tone). When it is recognized that the phrase "湖泊" exists in the sentence, it is determined that the character "泊" is pronounced as "po1". However, this method requires a large amount of storage space to store the pronunciations of polyphonic characters in different phrases, and it is easy to occur that the polyphonic character and the adjacent Chinese characters are misrecognized as the preset phrases, resulting in a low accuracy. For example, the pronunciation of the character "还" in the preset phrase "还书" is pronounced as "huan2" ("huan" represents a pinyin letter, and "2" represents that the tone of the pronunciation is the second tone). However, in the sentence "XXX还书写了一篇新的文章", although "还" and "书" are adjacent, they are not a phrase, and obviously in this context, the character "还" is pronounced as "hai2" ("hai" represents a pinyin letter, and "2" represents that the tone of the pronunciation is the second tone).
[0110] Alternatively, the text features of the text are extracted through the BERT model (Bidirectional Encoder Representation from Transformers), and then the text features are input into the pronunciation classification model to determine the pronunciation of the polyphone. However, this method requires the BERT model and the pronunciation classification model to be trained separately, and each polyphone in the vocabulary of the BERT model is represented by the same identifier. When training the BERT model, the features of the polyphone extracted by the BERT model will not be distinguished according to different pronunciations, and in fact, the semantics represented by the different pronunciations of the polyphone are different. Therefore, the features of the polyphone extracted by the BERT model trained in this way are inaccurate, so that the accuracy of the pronunciation prediction result of the speech classification model based on the feature is low. Therefore, how to quickly and accurately determine the pronunciation of polyphones in text has become an urgent problem to be solved.
[0111] Based on the above research, the present disclosure provides a training method for a polyphone pronunciation prediction network, a speech generation method and a device. In a target vocabulary, polyphones with different pronunciations correspond to different text identifiers. Therefore, the different pronunciations of polyphones can be distinguished by text identifiers. When the sample text is converted into a corresponding sample text identifier, since the text identifiers corresponding to different pronunciations of the same Chinese character in the sample text identifier are the same, the polyphone pronunciation prediction network can predict different pronunciations of the same character. Furthermore, the polyphone pronunciation prediction network to be trained is trained according to the pronunciation prediction results and the labels corresponding to the sample texts (i.e., the text identifiers of the correct pronunciations corresponding to the polyphones), thereby improving the ability of the polyphone pronunciation prediction network to classify polyphones as correct pronunciations and improving the accuracy of the polyphone pronunciation prediction network in predicting the pronunciations of polyphones.
[0112] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0113] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.
[0114] For ease of understanding this embodiment, first, a detailed introduction is given to a method for training a polyphonic pronunciation prediction network disclosed in the embodiments of the present disclosure. The execution subject of the method for training the polyphonic pronunciation prediction network provided in the embodiments of the present disclosure is generally a computer device with certain computing capabilities, such as a client or a server. The client may be, for example, a tablet computer, a smart phone, a personal computer, etc. In some possible implementation manners, the method for training the polyphonic pronunciation prediction network can be implemented by a processor calling computer-readable instructions stored in a memory.
[0115] See Figure 1 As shown, it is a flowchart of the method for training the polyphonic pronunciation prediction network provided in the embodiments of the present disclosure. The method includes steps 101 to 104, where:
[0116] Step 101: Obtain a sample text containing polyphonic characters and the label corresponding to the sample text. The label corresponding to the sample text is used to represent the text identifier of the polyphonic characters in the pre-constructed target vocabulary. Different pronunciations of the same Chinese character in the target vocabulary correspond to different text identifiers;
[0117] Step 102: Convert the sample text into a corresponding sample text identifier. The text identifiers corresponding to different pronunciations of the same Chinese character in the sample text identifier are the same;
[0118] Step 103: Input the sample text identifier into the polyphonic pronunciation prediction network to be trained, and obtain a pronunciation prediction result corresponding to the sample text. The pronunciation prediction result is used to represent the probability that each character included in the sample text is the Chinese character corresponding to each text identifier in the target vocabulary;
[0119] Step 104: Train the polyphonic pronunciation prediction network to be trained based on the pronunciation prediction result and the label corresponding to the sample text.
[0120] The following is a detailed description of the above steps:
[0121] Regarding step 101,
[0122] Among them, the sample text may include sentences, paragraphs, articles, phrases, etc. The sample text is exemplified as "The small boat is floating on the lake", where "泊" is a polyphonic character. The text identifier is the unique identifier of the text in the target vocabulary. For example, it may be the number (Identity document, ID) of the polyphonic character in the target vocabulary in the sample text. For example, the text identifier of "泊" is 3788.
[0123] Each Chinese character in the target vocabulary corresponds to a different text identifier. For polyphonic characters in the target vocabulary, in addition to the text identifier corresponding to the Chinese character itself, there are also text identifiers corresponding to different pronunciations.
[0124] Exemplarily, "泊" has two pronunciations and can respectively correspond to three text identifiers: the text identifier 3788 of the character "泊" itself (i.e., the text identifier in the initial vocabulary), the text identifier 21569 of the pronunciation "bo2", and the text identifier 21474 of the pronunciation "po1".
[0125] In this embodiment, the pronunciation of a polyphonic character is represented in the way of "pinyin letter" + "tone". The tone can be represented by "1", "2", "3", "4", "5", where "1", "2", "3", "4" respectively represent "first tone", "second tone", "third tone", "fourth tone", and "5" represents "light tone". For example, the above pronunciation "bo2" means that the pronunciation of the pinyin letter "bo" is the second tone.
[0126] In a possible implementation manner, the target vocabulary can be constructed according to the following methods (A1 to A4):
[0127] A1. Obtain the initial vocabulary; wherein, the initial vocabulary includes the text identifiers corresponding to each Chinese character.
[0128] Exemplarily, the initial vocabulary can be the Chinese BERT vocabulary. There are 21,128 text identifiers in the Chinese BERT vocabulary. Each Chinese character in the initial vocabulary corresponds to a unique text identifier. For example, "泊" corresponds to 3788. In addition to Chinese characters, the initial vocabulary can also include text identifiers corresponding to special symbols. For example, "[CLS]" corresponds to 101, "[SEP]" corresponds to 102, and "。" corresponds to 511.
[0129] A2. Determine the polyphonic characters in the initial vocabulary and their respective pronunciations.
[0130] Exemplarily, there are 354 polyphonic characters in the Chinese BERT vocabulary. Any polyphonic character can correspond to at least two pronunciations. For example, if the 354 polyphonic characters are split according to their pronunciations, they can be split into 741 polyphonic characters corresponding to unique pronunciations. For example, "降" can be split into two polyphonic characters corresponding to unique pronunciations: the pronunciation "jiang4" and the pronunciation "xiang2".
[0131] A3. Add pronunciation marks to the polyphonic characters under different pronunciations and determine the text identifiers corresponding to the polyphonic characters under different pronunciations.
[0132] Specifically, when determining the text identifiers corresponding to the polyphonic characters under different pronunciations, the text identifiers in the initial vocabulary can be continued as the text identifiers of the polyphonic characters under different pronunciations. Exemplarily, the pronunciation marks can be Arabic numerals (such as 1, 2, 3...). The largest text identifier in the Chinese BERT vocabulary is 21128. Then, the numbers can be sequentially numbered starting from 21129 as the text identifiers corresponding to the polyphonic characters under different pronunciations. For example, the character "泊" has two pronunciations. After adding the pronunciation marks to "泊", it can be expressed as "泊1" and "泊2". The text identifier corresponding to "泊1" is 21474, the pronunciation corresponding to "泊1" is "po1", the text identifier corresponding to "泊2" is 21569, and the pronunciation corresponding to "泊2" is "bo2".
[0133] A4. Update the initial vocabulary to obtain the target vocabulary based on the polyphonic characters with added pronunciation marks and the text identifiers corresponding to the polyphonic characters under different pronunciations respectively.
[0134] Specifically, adding the polyphonic characters with added pronunciation marks and the text identifiers corresponding to the polyphonic characters under different pronunciations respectively to the initial vocabulary can obtain the target vocabulary. Exemplarily, after the Chinese BERT vocabulary is updated (after adding the text identifiers corresponding to the pronunciations of 741 polyphonic characters), the number of text identifiers in the vocabulary (or the length of the vocabulary) changes from 21128 to 21869.
[0135] Using this method, polyphonic characters can be split into multiple Chinese characters according to different pronunciations, so that subsequently, different text identifiers can be obtained for polyphonic characters with different pronunciations based on the target vocabulary, and thus the prediction ability of the polyphonic character pronunciation prediction network for different pronunciations can be trained.
[0136] After obtaining the target vocabulary, the label corresponding to the sample text can be determined according to the target vocabulary. In a possible implementation manner, the polyphonic characters in the sample text carry pronunciation marks corresponding to the pronunciations. When obtaining the label corresponding to the sample text, the text identifier of the polyphonic character in the sample text can be determined based on the target vocabulary and the pronunciation marks of the polyphonic character in the sample text.
[0137] Specifically, the polyphonic character with the added pronunciation mark can be searched in the target vocabulary to obtain the text identifier corresponding to the polyphonic character with the added pronunciation mark in the target vocabulary. Exemplarily, if the polyphonic character in the sample text is "泊" and the pronunciation mark is "1", then it can be searched based on "泊1" in the target vocabulary to obtain the text identifier 21474 corresponding to "泊1".
[0138] Since the polyphonic character pronunciation prediction network to be trained can be pre-trained, the method provided by the present disclosure can be a method for fine-tuning and training the polyphonic character pronunciation prediction network, and here the polyphonic character pronunciation prediction network to be trained can be trained only based on the labels of polyphonic characters.
[0139] For step 102,
[0140] Since the polyphonic character pronunciation prediction network cannot directly process Chinese characters, it is necessary to first convert the sample text into corresponding sample text identifiers. At the same time, since after the sample text identifier is input into the polyphonic character pronunciation prediction network, the polyphonic character pronunciation prediction network needs to extract features based on the sample text identifier, and the essence of polyphonic characters with different pronunciations is still the same Chinese character, so the text identifiers corresponding to different pronunciations of the same Chinese character in the sample text identifier can be the same.
[0141] In a possible implementation manner, when converting the sample text into the corresponding sample text identifier, the sample text can be converted into the corresponding sample text identifier based on the target vocabulary.
[0142] Specifically, the text identifier corresponding to the position of each character in the sample text can be determined from the target vocabulary. Exemplarily, if the sample text is "small boat", the converted sample text identifier is "101 22075670102".
[0143] Specifically, when converting the sample text into the corresponding sample text identifier, it can be specifically divided into the following steps B1 to B3:
[0144] B1. Determine the mask corresponding to the sample text, and replace the polyphonic characters with pronunciation marks in the sample text based on the mask;
[0145] Among them, the mask corresponding to the sample text can be the [MASK] character. Exemplarily, if the sample text is "The small boat drifts2 in the lake1.", the replaced sample text is "The small boat drifts[MASK] in the lake[MASK].".
[0146] B2.剔除读音标记的所述样本文本转换为对应的样本文本标识。
[0147] Specifically, the polyphonic characters with the pronunciation marks at the positions covered by the mask can be replaced with polyphonic characters without the pronunciation marks. Continuing the above example, the Chinese characters at the positions covered by the mask are "bo1" and "bo2", and "bo" can be replaced at that position again. The final replacement result is "The small boat drifts in the lake.".
[0148] B3. Convert the sample text with the pronunciation marks removed into the corresponding sample text identifier.
[0149] Specifically, the sample text identifier corresponding to the sample text may be determined based on the target word list as described above. For example, if the sample text is "a boat drifts in a lake.", the converted sample text identifier is "101 2207 5670 4023 3788 1762 3959 3788 7027 511 102".
[0150] Here, if the polyphone pronunciation prediction network is a pre-trained BERT network, when converting the sample text into the corresponding sample text identifier, it is necessary to add the corresponding text identifiers of [CLS] and [SEP] at the beginning of the sentence and the sentence interval of the sample text respectively, such as the text identifier corresponding to [CLS] is 101, and the text identifier corresponding to [SEP] is 102.
[0151] By adopting this method, the polyphones with different pronunciations in the sample text can be converted into text tags that do not distinguish pronunciations, so that the polyphone pronunciation prediction network can perform feature extraction on polyphones that do not carry pronunciation tags, avoid the influence of the pronunciation tags of polyphones on feature extraction, and improve the accuracy of polyphone pronunciation prediction.
[0152] For step 103,
[0153] The polyphone pronunciation prediction network can be a pre-trained model (natural language model), such as a pre-trained BERT network. The pronunciation prediction result, for example, if there are 21,869 text identifiers in the target vocabulary, then the pronunciation prediction result is used to represent the probability that each word contained in the sample text belongs to the 21,869 text identifiers, that is, each word in the sample text corresponds to 21,869 probabilities.
[0154] In a possible implementation, the polyphone pronunciation prediction network includes an initial feature extraction module, an encoding module and a probability prediction module, and the polyphone pronunciation prediction network is a pre-trained model. The initial feature extraction module of the polyphone pronunciation prediction network, such as an embedding layer, is used to convert the sample text identifier into an embedded feature vector and extract the initial feature information of the embedded feature vector. The encoding module can be composed of an encoder in a 12-layer transformer network, and the dimension of the encoding module is 768. The encoding module adopts a self-attention mechanism and includes 12 attention heads, which can extract deep feature information from the initial feature information. The probability prediction module is a multi-classification network, such as a masked language model (MLM), and the probability prediction module can determine the pronunciation prediction result according to the deep feature information.
[0155] By adopting this structure, training can be performed on the basis of an existing pre-trained natural language model, and the entire polyphone pronunciation prediction network can be directly trained without accessing other neural networks, thereby improving the training efficiency.
[0156] In one possible implementation, before the sample text identifier is input into the polyphonetic character pronunciation prediction network to be trained, the polyphonetic character pronunciation prediction network to be trained can be initialized according to the following method: determining the initial parameters corresponding to the target Chinese character in the polyphonetic character pronunciation prediction network; wherein the target Chinese character corresponds to multiple pronunciations (i.e., a polyphone); and then adjusting the parameters of the fully connected layer and the masked language layer of the polyphonetic character pronunciation prediction network based on the initial parameters corresponding to the target Chinese character.
[0157] In a possible implementation, the initial parameters corresponding to the target Chinese characters in the polyphone pronunciation prediction network may include: the weights in the fully connected layer in the initial feature extraction module, and the bias parameters (bias parameters) of the mask language layer in the probability prediction module.
[0158] When adjusting the parameters of the fully connected layer and the masked language layer of the polyphone pronunciation prediction network based on the initial parameters corresponding to the target Chinese characters, the initial parameters corresponding to the target Chinese characters can be used as the initial parameters corresponding to the updated Chinese characters; wherein the updated Chinese characters are target Chinese characters carrying different pronunciation marks, and then the initial parameters corresponding to the updated Chinese characters are added to the parameters of the fully connected layer and the masked language layer of the polyphone pronunciation prediction network.
[0159] Specifically, the fully connected layer includes weights corresponding to multiple Chinese characters. In order for the fully connected layer to process text identifiers of polyphonic characters with pronunciation marks, it is necessary to determine the weights corresponding to the updated Chinese characters in the fully connected layer to establish the links of the updated Chinese characters. Similarly, the masked language layer needs to output the probabilities that each character in the sample text is the Chinese character corresponding to each text identifier in the target vocabulary. Therefore, in order for the masked language layer to output the probabilities that each character in the sample text belongs to multiple polyphonic characters with pronunciation marks in the target dictionary, it is necessary to determine the bias parameters of the updated Chinese characters. Exemplarily, the weights and bias parameters of "bo" can be used as the weights and bias parameters of "bo1" and the weights and bias parameters of "bo2".
[0160] Since the target Chinese character and the updated Chinese character are the same Chinese character and their features are relatively similar, compared with randomly initializing the weights and bias parameters of the updated Chinese character, allowing the updated Chinese character to inherit the weights and bias parameters of the target Chinese character can improve the processing ability of the polyphonic character pronunciation prediction network for the updated Chinese character, thereby improving the accuracy of the polyphonic character pronunciation prediction network.
[0161] Regarding step 104
[0162] In a possible implementation manner, the polyphonic character pronunciation prediction network to be trained is a pre-trained network; when training the polyphonic character pronunciation prediction network to be trained based on the pronunciation prediction result and the label corresponding to the sample text, the target prediction result corresponding to the polyphonic character in the pronunciation prediction result can be determined first, and then the polyphonic character pronunciation prediction network to be trained can be trained based on the target prediction result and the label corresponding to the sample text.
[0163] Specifically, since Chinese characters with only one pronunciation do not need to perform pronunciation prediction, it is only necessary to train the polyphonic character pronunciation prediction network based on the target prediction result corresponding to the polyphonic character.
[0164] The target prediction result represents the probabilities that the polyphonic character is the Chinese character corresponding to each text identifier in the target vocabulary. When training the polyphonic character pronunciation prediction network to be trained based on the target prediction result and the label corresponding to the sample text, for any polyphonic character, the text identifier corresponding to the highest probability in the target prediction result can be used as the target prediction identifier first, and then the target loss value (such as cross-entropy loss) can be calculated based on the probability of the target prediction identifier and the text identifier corresponding to the label of the polyphonic character, and then the parameters of the polyphonic character pronunciation prediction network can be trained based on the target loss value.
[0165] By adopting this method, the prediction result of the polyphonic character pronunciation prediction network (i.e., the target prediction result) can be used to train the polyphonic character pronunciation prediction network, which improves the accuracy of the polyphonic character pronunciation prediction network in predicting polyphonic characters.
[0166] The following steps C1 to C2 are specific implementation steps of the above method:
[0167] C1. The pronunciation prediction result includes a probability matrix. When determining the target prediction result corresponding to the polyphonic character in the pronunciation prediction result, the probability vector corresponding to the polyphonic character in the probability matrix can be determined first.
[0168] Specifically, each row of numbers in the probability matrix represents the probability that the Chinese character corresponding to any text identifier in the sample text identifier is the Chinese character corresponding to each text identifier in the target vocabulary, and each row is arranged in the input order of the sample text identifier. For example, when the sample text identifier is "101 2207 5670 4023 3788 1762 39593788 7027 511 102", the first row of the output probability matrix represents the probability that the Chinese character in the sample text corresponding to the text identifier 101 is the Chinese character corresponding to each text identifier in the target vocabulary, the second row represents the probability that the Chinese character in the sample text corresponding to the text identifier 2207 is the Chinese character corresponding to each text identifier in the target vocabulary, the third row represents the probability that the Chinese character in the sample text corresponding to the text identifier 5670 is the Chinese character corresponding to each text identifier in the target vocabulary, and so on.
[0169] Therefore, the probability vector corresponding to the polyphonic character in the probability matrix can be determined according to the target index of the text identifier of the polyphonic character in the sample text. The target index is the position of the polyphonic character in the sample text. For example, Figure 2 as shown, the sample text is "The small boat drifts2 in the lake1", after the sample text is converted into a sample text identifier, the polyphonic characters "bo2" and "bo1" in it are in the 5th and 8th positions, then 5 and 8 are the target indexes, and the probabilities of the 5th and 8th rows of the probability matrix can be used as the probability vector (i.e., Figure 2 P4 and P7 in).
[0170] C2. When training the polyphonic character pronunciation prediction network to be trained based on the target prediction result and the label corresponding to the sample text, the polyphonic character pronunciation prediction network to be trained can be trained based on the probability vector corresponding to the polyphonic character and the label corresponding to the sample text.
[0171] Specifically, the text identifier corresponding to the highest probability in the probability vector corresponding to the polyphone can be used as the target prediction identifier, and then a target loss value (such as cross entropy loss) is calculated based on the probability of the target prediction identifier and the text identifier corresponding to the label of the polyphone, and then the parameters of the polyphone pronunciation prediction network are trained based on the target loss value.
[0172] Finally, if Figure 2 As shown, the overall training process of the polyphone pronunciation prediction network is introduced:
[0173] D1, obtaining an initial vocabulary, adding pronunciation marks to polyphones with different pronunciations in the initial vocabulary, and determining the text identifiers corresponding to the polyphones with different pronunciations, and then adding the polyphones with pronunciation marks and the corresponding text identifiers to the initial vocabulary to obtain a target vocabulary;
[0174] D2. Obtaining a sample text containing polyphones and a label corresponding to the sample text;
[0175] D3, based on the target vocabulary, converting the sample text into a corresponding sample text identifier;
[0176] D4, inputting the sample text identifier into the polyphone pronunciation prediction network to be trained, processing the sample text with the initial feature extraction module to obtain the initial feature information corresponding to the sample text, then processing the sample text with the encoding module to obtain the deep feature information corresponding to the sample text, and finally processing the sample text with the probability prediction module to obtain the pronunciation prediction result corresponding to the sample text;
[0177] D5. Calculate a target loss value based on the pronunciation prediction result and the label corresponding to the sample text, and train the polyphonetic character pronunciation prediction network to be trained based on the target loss value.
[0178] The training method of the polyphone pronunciation prediction network provided by the embodiment of the present disclosure is that in the target vocabulary, polyphones with different pronunciations correspond to different text identifiers, so the different pronunciations of polyphones can be distinguished by the text identifiers; when the sample text is converted into the corresponding sample text identifier, since the text identifiers corresponding to different pronunciations of the same Chinese character in the sample text identifier are the same, the polyphone pronunciation prediction network can predict the different pronunciations of the same character, and further, the polyphone pronunciation prediction network to be trained is trained according to the pronunciation prediction results and the labels corresponding to the sample texts (i.e., the text identifiers of the correct pronunciations corresponding to the polyphones), thereby improving the ability of the polyphone pronunciation prediction network to classify polyphones as correct pronunciations and improving the accuracy of the polyphone pronunciation prediction network in predicting the pronunciations of polyphones.
[0179] Based on the same inventive concept, the present disclosure also provides a speech generation method, which is applied to a user terminal or a server, and the user terminal can be, for example, a tablet computer, a smart phone, a personal computer, etc. Figure 3 FIG. 1 is a flowchart of a speech generation method provided by an embodiment of the present disclosure, wherein the method includes steps 301 to 304, wherein:
[0180] Step 301: Obtain the text to be detected;
[0181] Step 302: Based on the polyphone pronunciation prediction network trained by the polyphone pronunciation prediction network training method described in the above embodiment, the text to be detected is recognized to obtain a text detection result corresponding to the text to be detected, wherein the text detection result includes the text identifier corresponding to each Chinese character in the text to be detected in the target vocabulary;
[0182] Step 303: Determine the target pronunciation corresponding to the text detection result based on the predetermined pronunciation mapping relationship;
[0183] Step 304: synthesize a target speech based on the target pronunciation.
[0184] Here are detailed instructions for the above steps:
[0185] For step 301,
[0186] The text to be detected may include sentences, paragraphs, articles, phrases, etc., and the sample text may include polyphonic characters, such as "the boat is drifting in the center of the lake."
[0187] For step 302,
[0188] Specifically, when the text to be detected is recognized, Figure 4 As shown, you can do this by:
[0189] Step 1: Based on the target word list, convert the text to be detected into a corresponding text identifier to be detected;
[0190] Specifically, the target word list can be searched according to each Chinese character in the text to be detected to determine the text identifier corresponding to each Chinese character in the text to be detected, and the text identifier corresponding to each Chinese character in the text to be detected constitutes the text identifier to be detected. Exemplary, "The boat is drifting in the center of the lake." The corresponding text identifier to be detected is "1012207 5660 1762 3959 704 2552 4023 3788 511 102", where the character [CLS] corresponding to 101 is a fixed character at the beginning of the sentence, and the character [SEP] corresponding to 102 is a separator between sentences.
[0191] Step 2: Input the text identification to be detected into the polyphonic character pronunciation prediction network to obtain the target probability matrix corresponding to the text to be detected;
[0192] Specifically, after inputting the text identification to be detected into the polyphonic character pronunciation prediction network, the initial feature extraction module in the polyphonic character pronunciation prediction network can convert the text identification to be detected into an embedded feature vector corresponding to the text to be detected, and perform initial feature extraction on the embedded feature vector to obtain the initial feature information of the text to be detected. The encoding module can perform deep feature extraction based on the initial feature information to obtain the deep feature information of the text to be detected. The probability prediction module can determine, based on the deep feature information, the probability that each character included in the text to be detected is the Chinese character corresponding to each text identification in the target vocabulary, that is, the target probability matrix.
[0193] Step 3: Based on the target probability matrix, determine the probability vector corresponding to each Chinese character in the text to be detected, and based on the probability vector and the target vocabulary, determine the text identification corresponding to each Chinese character in the text to be detected.
[0194] Specifically, each row in the target probability matrix corresponds to the probability vector of each Chinese character in the text to be detected. According to the position of each Chinese character in the sample text identification, the probability vector corresponding to each Chinese character can be determined (for example, in the above example, "泊" is in the ninth position, and the corresponding probability vector is P8). Then, determine the highest probability in the probability vectors corresponding to each Chinese character in the text to be detected, and use the text identification in the target vocabulary corresponding to the highest probability as the text identification corresponding to each Chinese character in the text to be detected. For example, if the probability that "泊" belongs to "泊2" is 0.996, then the text detection result is "小舟在湖中心漂泊2。".
[0195] Regarding steps 303 - 304,
[0196] Specifically, the pronunciation mapping relationship can be the mapping relationship between Chinese characters (including polyphonic characters with pronunciation marks) and pinyin. Based on the pronunciation mapping relationship, the pronunciation corresponding to each Chinese character in the text detection result can be determined. For example, the pinyin corresponding to "泊2" is "bo2". Therefore, after determining the pronunciation corresponding to each Chinese character in the text detection result, the target pronunciation corresponding to the text to be detected can be determined. Based on the target pronunciation, the target speech can be synthesized.
[0197] Here, when synthesizing the target speech based on the target pronunciation, the text to be detected can also be subjected to sentiment analysis to determine the target tone (i.e., the tone of the speech), etc., and the target speech can be generated based on the target tone and the target pronunciation.
[0198] The speech generation method provided by the embodiment of the present disclosure can obtain the text to be detected, and input the text to be detected into the polyphone pronunciation prediction network, and accurately predict the pronunciation of the polyphone in the text to be detected based on the polyphone pronunciation prediction network, so that the target pronunciation can be determined based on the text detection result containing the accurate polyphone pronunciation, and the correct target speech can be generated. Using this method, the accuracy of the target speech is improved, and the pronunciation prediction is performed only by the polyphone pronunciation prediction network, without the need for other additional neural networks, saving storage space and improving the efficiency of pronunciation prediction.
[0199] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0200] Based on the same inventive concept, the embodiments of the present disclosure also provide a training device for a polyphonetic character pronunciation prediction network corresponding to the training method for the polyphonetic character pronunciation prediction network. Since the principle of solving the problem by the device in the embodiments of the present disclosure is similar to the training method for the polyphonetic character pronunciation prediction network in the embodiments of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0201] Reference Figure 5 As shown, it is a schematic diagram of the architecture of a training device for a polyphonetic character pronunciation prediction network provided by an embodiment of the present disclosure, the device comprises: a first acquisition module 501, a conversion module 502, a prediction module 503, and a training module 504; wherein,
[0202] The first acquisition module 501 is used to acquire a sample text containing polyphones and a label corresponding to the sample text, wherein the label corresponding to the sample text is used to represent the text identifier of the polyphone in the sample text in a pre-constructed target vocabulary, and different pronunciations of the same Chinese character in the target vocabulary correspond to different text identifiers;
[0203] A conversion module 502 is used to convert the sample text into a corresponding sample text identifier; wherein different pronunciations of the same Chinese character in the sample text identifier correspond to the same text identifier;
[0204] The prediction module 503 is used to input the sample text identifier into the polyphone pronunciation prediction network to be trained to obtain the pronunciation prediction result corresponding to the sample text; wherein the pronunciation prediction result is used to represent the probability that each character contained in the sample text is the Chinese character corresponding to each text identifier in the target vocabulary;
[0205] The training module 504 is used to train the polyphonetic character pronunciation prediction network to be trained based on the pronunciation prediction results and the labels corresponding to the sample texts.
[0206] In a possible implementation, the polyphonetic characters in the sample text carry pronunciation marks corresponding to the pronunciations;
[0207] The first acquisition module 501, when acquiring the label corresponding to the sample text, is used to:
[0208] Based on the target vocabulary and the pronunciation marks of the polyphonetic characters in the sample text, text identifiers of the polyphonetic characters in the sample text are determined.
[0209] In a possible implementation, the polyphonetic character pronunciation prediction network includes an initial feature extraction module, an encoding module and a probability prediction module 503, and the polyphonetic character pronunciation prediction network is a pre-trained model;
[0210] Before inputting the sample text identifier into the polyphonetic character pronunciation prediction network to be trained, the prediction module 503 is further used to initialize the polyphonetic character pronunciation prediction network to be trained:
[0211] Determining initial parameters corresponding to the target Chinese character in the polyphonetic character pronunciation prediction network; wherein the target Chinese character corresponds to multiple pronunciations;
[0212] The parameters of the fully connected layer and the masked language layer of the polyphone pronunciation prediction network are adjusted based on the initial parameters corresponding to the target Chinese character.
[0213] In a possible implementation, the prediction module 503, when adjusting the parameters of the fully connected layer and the masked language layer of the polyphonetic character pronunciation prediction network based on the initial parameters corresponding to the target Chinese character, is used to:
[0214] Using the initial parameters corresponding to the target Chinese characters as the initial parameters corresponding to the updated Chinese characters; wherein the updated Chinese characters are the target Chinese characters carrying different pronunciation marks;
[0215] The initial parameters corresponding to the updated Chinese characters are added to the parameters of the fully connected layer and the masked language layer of the polyphonetic character pronunciation prediction network.
[0216] In a possible implementation manner, the conversion module 502, when converting the sample text into a corresponding sample text identifier, is configured to:
[0217] Determine a mask corresponding to the sample text, and replace the polyphonetic characters with pronunciation marks in the sample text based on the mask;
[0218] Eliminate pronunciation marks in the sample text based on the mask;
[0219] The sample text with the pronunciation mark removed is converted into a corresponding sample text identifier.
[0220] In a possible implementation manner, the first acquisition module 501 is further configured to construct the target vocabulary according to the following method:
[0221] Obtaining an initial vocabulary; wherein the initial vocabulary includes text identifiers corresponding to each Chinese character;
[0222] Determining polyphonetic characters in the initial vocabulary and the respective pronunciations of the polyphonetic characters;
[0223] Adding pronunciation marks to the polyphones under different pronunciations, and determining the text identifiers corresponding to the polyphones under different pronunciations;
[0224] Based on the polyphones with pronunciation marks added, and the text identifiers corresponding to the polyphones under different pronunciations, the initial vocabulary is updated to obtain the target vocabulary.
[0225] In a possible implementation manner, the conversion module 502, when converting the sample text into a corresponding sample text identifier, is configured to:
[0226] Based on the target vocabulary, the sample text is converted into a corresponding sample text identifier.
[0227] In a possible implementation manner, the polyphonetic character pronunciation prediction network to be trained is a pre-trained network;
[0228] The training module 504, when training the polyphone pronunciation prediction network to be trained based on the pronunciation prediction result and the label corresponding to the sample text, is used to:
[0229] Determine a target prediction result corresponding to the polyphonetic character in the pronunciation prediction result;
[0230] The polyphone pronunciation prediction network to be trained is trained based on the target prediction result and the label corresponding to the sample text.
[0231] In a possible implementation manner, the pronunciation prediction result includes a probability matrix;
[0232] The training module 504, when determining the target prediction result corresponding to the polyphonetic character in the pronunciation prediction result, is used to:
[0233] Determining a probability vector corresponding to the polyphonetic character in the probability matrix;
[0234] The training module 504, when training the polyphone pronunciation prediction network to be trained based on the target prediction result and the label corresponding to the sample text, is used to:
[0235] The polyphone pronunciation prediction network to be trained is trained based on the probability vector corresponding to the polyphone and the label corresponding to the sample text.
[0236] Corresponds to Figure 3 The speech generation method described in the present disclosure also provides a schematic diagram of the architecture of a speech generation device, referring to Figure 6 As shown, the device includes: a second acquisition module 601, an identification module 602, a determination module 603, and a synthesis module 604; wherein,
[0237] The second acquisition module 601 is used to acquire the text to be detected;
[0238] The recognition module 602 is used to recognize the text to be detected based on the polyphonetic character pronunciation prediction network trained by the polyphonetic character pronunciation prediction network training method described in the above embodiment, and obtain the corresponding text detection result in the text to be detected, wherein the text detection result includes the text identifier corresponding to each Chinese character in the text to be detected in the target vocabulary;
[0239] A determination module 603 is used to determine the target pronunciation corresponding to the text detection result based on a predetermined pronunciation mapping relationship;
[0240] The synthesis module 604 is used to synthesize the target speech based on the target pronunciation.
[0241] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.
[0242] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 7 As shown, it is a schematic diagram of the structure of a computer device 700 provided in an embodiment of the present disclosure, including a processor 701, a memory 702, and a bus 703. Among them, the memory 702 is used to store execution instructions, including a memory 7021 and an external memory 7022; the memory 7021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 701, and the data exchanged with the external memory 7022 such as a hard disk. The processor 701 exchanges data with the external memory 7022 through the memory 7021. When the computer device 700 is running, the processor 701 communicates with the memory 702 through the bus 703, so that the processor 701 executes the following instructions:
[0243] Acquire a sample text containing polyphones and a label corresponding to the sample text, wherein the label corresponding to the sample text is used to represent a text identifier of the polyphone in the sample text in a pre-constructed target vocabulary, and different pronunciations of the same Chinese character in the target vocabulary correspond to different text identifiers;
[0244] Converting the sample text into a corresponding sample text identifier; wherein different pronunciations of the same Chinese character in the sample text identifier correspond to the same text identifier;
[0245] Inputting the sample text identifier into the polyphone pronunciation prediction network to be trained to obtain a pronunciation prediction result corresponding to the sample text; wherein the pronunciation prediction result is used to characterize the probability that each character contained in the sample text is the Chinese character corresponding to each text identifier in the target vocabulary;
[0246] The polyphone pronunciation prediction network to be trained is trained based on the pronunciation prediction result and the label corresponding to the sample text.
[0247] Based on the same technical concept, the embodiment of the present disclosure also provides another computer device. Figure 8 As shown, it is a schematic diagram of the structure of a computer device 800 provided in an embodiment of the present disclosure, including a processor 801, a memory 802, and a bus 803. Among them, the memory 802 is used to store execution instructions, including a memory 8021 and an external memory 8022; the memory 8021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 801, and the data exchanged with the external memory 8022 such as a hard disk. The processor 801 exchanges data with the external memory 8022 through the memory 8021. When the computer device 800 is running, the processor 801 communicates with the memory 802 through the bus 803, so that the processor 801 executes the following instructions:
[0248] Get the text to be detected;
[0249] Based on the polyphone pronunciation prediction network trained by the polyphone pronunciation prediction network training method described in the above embodiment, the text to be detected is recognized to obtain a corresponding text detection result in the text to be detected, wherein the text detection result includes the text identifiers corresponding to each Chinese character in the text to be detected in the target vocabulary;
[0250] Determining a target pronunciation corresponding to the text detection result based on a predetermined pronunciation mapping relationship;
[0251] A target speech is synthesized based on the target pronunciation.
[0252] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the polyphonetic character pronunciation prediction network training method and speech generation method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.
[0253] The present disclosure also provides a computer program product that carries a program code. The program code includes instructions that can be used to execute the steps of the training method of the polyphonetic character pronunciation prediction network and the speech generation method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.
[0254] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).
[0255] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.
[0256] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0257] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0258] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0259] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
[0260] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.
Claims
1. A method for training a polyphonetic character pronunciation prediction network, characterized in that: include: Acquire a sample text containing polyphones and a label corresponding to the sample text, wherein the label corresponding to the sample text is used to represent a text identifier of the polyphone in the sample text in a pre-constructed target vocabulary, and different pronunciations of the same Chinese character in the target vocabulary correspond to different text identifiers; Converting the sample text into a corresponding sample text identifier; wherein different pronunciations of the same Chinese character in the sample text identifier correspond to the same text identifier; Inputting the sample text identifier into the polyphone pronunciation prediction network to be trained to obtain a pronunciation prediction result corresponding to the sample text; wherein the pronunciation prediction result is used to characterize the probability that each character contained in the sample text is the Chinese character corresponding to each text identifier in the target vocabulary; Training the polyphone pronunciation prediction network to be trained based on the pronunciation prediction result and the label corresponding to the sample text; The converting the sample text into a corresponding sample text identifier includes: Determine a mask corresponding to the sample text, and replace the polyphonetic characters with pronunciation marks in the sample text based on the mask; Eliminate pronunciation marks in the sample text based on the mask; The sample text with the pronunciation mark removed is converted into a corresponding sample text identifier.
2. The method according to claim 1, characterized in that The polyphonetic characters in the sample text carry pronunciation marks corresponding to the pronunciations; Obtaining the label corresponding to the sample text includes: Based on the target vocabulary and the pronunciation marks of the polyphonetic characters in the sample text, text identifiers of the polyphonetic characters in the sample text are determined.
3. The method according to claim 1 or 2, characterized in that: The polyphonetic character pronunciation prediction network includes an initial feature extraction module, an encoding module and a probability prediction module, and the polyphonetic character pronunciation prediction network is a pre-trained model; Before inputting the sample text identifier into the polyphonetic character pronunciation prediction network to be trained, the method further includes initializing the polyphonetic character pronunciation prediction network to be trained according to the following method: Determining initial parameters corresponding to the target Chinese character in the polyphonetic character pronunciation prediction network; wherein the target Chinese character corresponds to multiple pronunciations; The parameters of the fully connected layer and the masked language layer of the polyphone pronunciation prediction network are adjusted based on the initial parameters corresponding to the target Chinese character.
4. The method according to claim 3, characterized in that The adjusting the parameters of the fully connected layer and the masked language layer of the polyphonetic character pronunciation prediction network based on the initial parameters corresponding to the target Chinese character includes: Using the initial parameters corresponding to the target Chinese characters as the initial parameters corresponding to the updated Chinese characters; wherein the updated Chinese characters are the target Chinese characters carrying different pronunciation marks; The initial parameters corresponding to the updated Chinese characters are added to the parameters of the fully connected layer and the masked language layer of the polyphonetic character pronunciation prediction network.
5. The method according to claim 1, characterized in that The method further comprises constructing the target vocabulary according to the following method: Obtaining an initial vocabulary; wherein the initial vocabulary includes text identifiers corresponding to each Chinese character; Determining polyphonetic characters in the initial vocabulary and the respective pronunciations of the polyphonetic characters; Adding pronunciation marks to the polyphones under different pronunciations, and determining the text identifiers corresponding to the polyphones under different pronunciations; Based on the polyphones with pronunciation marks added, and the text identifiers corresponding to the polyphones under different pronunciations, the initial vocabulary is updated to obtain the target vocabulary.
6. The method according to claim 1, characterized in that The converting the sample text into a corresponding sample text identifier includes: Based on the target vocabulary, the sample text is converted into a corresponding sample text identifier.
7. The method according to claim 1, characterized in that The polyphonetic character pronunciation prediction network to be trained is a pre-trained network; The training of the polyphonetic character pronunciation prediction network to be trained based on the pronunciation prediction result and the label corresponding to the sample text includes: Determine a target prediction result corresponding to the polyphonetic character in the pronunciation prediction result; The polyphone pronunciation prediction network to be trained is trained based on the target prediction result and the label corresponding to the sample text.
8. The method according to claim 7, characterized in that The pronunciation prediction result includes a probability matrix; The determining of the target prediction result corresponding to the polyphonetic character in the pronunciation prediction result includes: Determining a probability vector corresponding to the polyphonetic character in the probability matrix; The training of the polyphone pronunciation prediction network to be trained based on the target prediction result and the label corresponding to the sample text includes: The polyphone pronunciation prediction network to be trained is trained based on the probability vector corresponding to the polyphone and the label corresponding to the sample text.
9. A speech generation method, characterized in that: include: Get the text to be detected; Based on the polyphone pronunciation prediction network trained by the polyphone pronunciation prediction network training method according to any one of claims 1 to 8, the text to be detected is recognized to obtain a corresponding text detection result in the text to be detected, wherein the text detection result includes the text identifiers corresponding to each Chinese character in the text to be detected in the target vocabulary; Determining a target pronunciation corresponding to the text detection result based on a predetermined pronunciation mapping relationship; A target speech is synthesized based on the target pronunciation.
10. A training device for a polyphonetic character pronunciation prediction network, characterized in that: include: A first acquisition module is used to acquire a sample text containing polyphones and a label corresponding to the sample text, wherein the label corresponding to the sample text is used to represent a text identifier of the polyphone in the sample text in a pre-constructed target vocabulary, and different pronunciations of the same Chinese character in the target vocabulary correspond to different text identifiers; A conversion module, used for converting the sample text into a corresponding sample text identifier; wherein different pronunciations of the same Chinese character in the sample text identifier correspond to the same text identifier; A prediction module, used for inputting the sample text identifier into the polyphone pronunciation prediction network to be trained to obtain a pronunciation prediction result corresponding to the sample text; wherein the pronunciation prediction result is used to characterize the probability that each character contained in the sample text is the Chinese character corresponding to each text identifier in the target vocabulary; A training module, used for training the polyphone pronunciation prediction network to be trained based on the pronunciation prediction result and the label corresponding to the sample text; The conversion module, when converting the sample text into a corresponding sample text identifier, is used to: Determine a mask corresponding to the sample text, and replace the polyphonetic characters with pronunciation marks in the sample text based on the mask; Eliminate pronunciation marks in the sample text based on the mask; The sample text with the pronunciation mark removed is converted into a corresponding sample text identifier.
11. A speech generating device, characterized in that: include: The second acquisition module is used to acquire the text to be detected; A recognition module, for recognizing the text to be detected based on the polyphonetic character pronunciation prediction network trained by the polyphonetic character pronunciation prediction network training method according to any one of claims 1 to 8, and obtaining a corresponding text detection result in the text to be detected, wherein the text detection result includes the text identifiers corresponding to each Chinese character in the text to be detected in the target vocabulary; A determination module, used to determine the target pronunciation corresponding to the text detection result based on a predetermined pronunciation mapping relationship; A synthesis module is used to synthesize a target speech based on the target pronunciation.
12. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the training method for the polyphone pronunciation prediction network as described in any one of claims 1 to 8 are executed, or the steps of the speech generation method as described in claim 9 are executed.
13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method for training a polyphone pronunciation prediction network as described in any one of claims 1 to 8, or executes the steps of the method for generating speech as described in claim 9.
