Chinese polyphone disambiguation method based on small Chinese pre-training model

Abstract

The invention relates to the field of natural language processing, and discloses a Chinese polyphone disambiguation method based on a small Chinese pre-training model. The method is characterized in that a small Chinese pre-training model is adopted, and the problem that when a large-scale pre-training model carries out a polyphone disambiguation task, the large-scale pre-training model is difficult to deploy on a low-resource computing platform is solved. Then, Chinese word segmentation and part-of-speech tagging are carried out on the text data, and the text data are combined with a feature sequence output by the pre-training model, so that the understanding of the model on context semantics is improved, and polyphones are better classified. And finally, a neural network classifier composed of full connection layers is used to realize classification of the feature sequence, the pronunciation of the polyphone is accurately identified, and the accuracy of disambiguation of the polyphone is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a Chinese polyphone disambiguation method based on a small Chinese pre-training model. Background Art

[0002] Voice and text are one of the main ways for humans to receive and express information, and they play a vital role in people's lives. Text-to-Speech (TTS) technology is a technology that converts text information into voice signals. In recent years, with the improvement of computer computing power and the development of deep learning, speech synthesis technology has been applied in our daily lives. The most common ones are mobile phone voice assistants and voice navigation. With the development of speech synthesis technology, the application scenarios of speech synthesis are also expanding.

[0003] The conversion from graphemes to phonemes is an important part of text-to-speech systems, and in Chinese, a major challenge is polyphone disambiguation, which is determining the pronunciation of characters with multiple pronunciations. Chinese characters represent meaning rather than sound, so in order to pronounce Chinese characters in text-to-speech (TTS) systems, grapheme-to-phoneme conversion (G2P) must be used to convert Chinese characters to pinyin. The main challenge facing Chinese G2P conversion is how to disambiguate the pronunciation of polyphones. Polyphones refer to characters that have different pronunciations depending on their semantic and syntactic usage in context. Early polyphone disambiguation systems were all based on rule matching. These methods required collecting a large number of words, phrases, etc. containing Chinese polyphones, and manually annotating the corresponding pronunciations of the polyphones. However, there are about a thousand polyphones in Mandarin Chinese, and the collocation of words and words in the Chinese language is ever-changing. It is almost impossible to build a database that can cover all polyphone application scenarios. Therefore, most of the industry uses and studies treat the polyphone disambiguation problem as a classification problem, using data-driven machine learning methods to classify the pronunciation of polyphones. With the rise of large-scale pre-trained models, the BERT-based polyphone disambiguation method has further improved the accuracy of polyphone classification, but its model is large in scale and difficult to deploy on low-resource computing platforms. Summary of the invention

[0004] In order to solve the problem that large-scale pre-trained models are difficult to deploy on low-resource computing platforms and have too many parameters, the present invention uses a small Chinese pre-trained model MiniRBT. Text data is input into the pre-trained model for training to obtain the corresponding semantic feature sequence, and the classifier composed of fully connected layers is used to classify it to obtain the correct pronunciation of polyphones.

[0005] In order to achieve the above object, the present invention proposes a Chinese polyphone disambiguation method based on a small Chinese pre-training model, and the specific steps are as follows:

[0006] S1: Perform Chinese word segmentation and part-of-speech tagging on the text data in the dataset so that the model can better understand the context semantics;

[0007] S2: Input the Chinese sentences in the dataset into the pre-trained model MiniRBT to obtain the semantic feature sequence;

[0008] S3: Add the word segmentation embedding feature sequence and part-of-speech embedding feature sequence corresponding to the text to the semantic features output by MiniRBT, which can be regarded as the context feature vector corresponding to each Chinese character in the text;

[0009] S4: The obtained feature vector is classified through a classifier composed of two fully connected layers, and the final polyphonetic pronunciation is output.

[0010] In step S1, the text dataset uses the CVTE-poly dataset, which contains 845,254 sentences and 414 polyphones, and performs word segmentation and part-of-speech tagging on the Chinese sentences in the dataset. Chinese word segmentation refers to finding the boundaries between words in a sentence and dividing them, and marking each Chinese character in the sentence with a word segmentation label. The types of labels include B (Begin), E (End), M (Middle) and S (Single), which respectively indicate that the Chinese character is located at the beginning of a word, the end of a word, the non-beginning and end positions of a word, and the Chinese character alone constitutes a word. After the sentence is preprocessed for word segmentation, the label can be converted into a word segmentation embedding feature based on the word segmentation label. The Chinese part-of-speech refers to marking a correct word type for each word on the result of Chinese word segmentation, such as noun, verb, adjective, etc. After obtaining the Chinese part-of-speech tagging result through text preprocessing, we mark each Chinese character in the sentence with the part-of-speech label of the word it is in, and then convert it into a part-of-speech embedding feature.

[0011] Further, in step S2:

[0012] The pre-trained model MiniRBT accepts the original Chinese character sequence as input and outputs a series of semantic features. Before embedding the sequence, the character embedding layer and the position embedding layer process the input character sequence respectively. The following Transformer block converts the embedding sequence into a semantic feature sequence.

[0013] The MiniRBT model is pre-trained on a large amount of unlabeled data, divided into two tasks: predicting masked input characters and predicting the next sentence, and learning semantic representations from raw character sequences.

[0014] Further, in step S3:

[0015] The word segmentation embedding feature sequence and part-of-speech embedding feature sequence corresponding to the text are added to the semantic features output by MiniRBT, which can be regarded as the context feature vector corresponding to each Chinese character in the text.

[0016] The semantic feature output by MiniRBT is denoted as x=x mini . Chinese word segmentation tags can also be converted into vectors through embedding. For each character, we represent its tokenized information vector as x token We then concatenate it with the character vector before the final prediction, i.e. x = [x mini ;x token ].

[0017] After word segmentation, part-of-speech tagging assigns a tag to each Chinese word. For each character, we assume that its tag is inherited from the word it belongs to. Tags can also be converted into vectors through embedding. For each character, we represent the vector of its POS tag information as x pos We then concatenate the character vector x = [x mini ;x token ;x pos ].

[0018] Furthermore, in step S4, the feature vector obtained in step S3 is classified by a classifier composed of two fully connected layers, and the final polyphonetic pronunciation is output.

[0019] The specific steps are to use a two-layer fully connected network to predict the pronunciation of polyphonic characters based on the MiniRBT output sequence. The first fully connected layer is shared by all polyphonic characters. As for the second fully connected output layer, it is not shared. Each polyphonic character has a separate output layer with the number of units equal to the number of possible pronunciations. The classifier is trained using the Softmax cross entropy loss method. The model predicts the label distribution through p.

[0020] p = softmax(W x+b)

[0021] Where W,b are the weights and biases of the fully connected layer.

[0022] During training, we use cross entropy to measure the loss between the predicted label distribution p and the one-hot encoding of the label y:

[0023]

[0024] Where C represents the total number of all pronunciations collected from all polyphones.

[0025] During inference, to restrict the selection of polyphones from candidate pronunciations, we use a hard mask m = [m1, ···, m C], except for the candidate pronunciation which is 1, all other pronunciations are 0. Then the final pronunciation is determined according to the maximum value of p⊙m, where ⊙ represents element-by-element multiplication.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] The polyphone disambiguation method based on the pre-training model can better learn text data and understand the semantics of the context in the text, so that the classification of polyphones can be more accurate. At the same time, the present invention also adds Chinese word segmentation and part-of-speech tagging to the semantic features, and also strengthens the model's understanding of text semantics, which promotes the accuracy of polyphone disambiguation. Finally, in order to solve the problem that the current pre-training model data is too large and difficult to deploy on a low-resource computing platform, the present invention uses a small Chinese pre-training model, which has a smaller number of model parameters and can be deployed on a low-resource computing platform, and improves the accuracy of polyphone disambiguation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The present invention is a flowchart of a Chinese polyphone disambiguation method based on a small Chinese pre-training model.

[0029] Figure 2 This is a structural diagram of the pre-training model in the method of the present invention.

[0030] Figure 3 This is an overall structural diagram of a Chinese polyphone disambiguation method based on a small Chinese pre-training model described in the present invention. DETAILED DESCRIPTION

[0031] like Figure 1 As shown, an end-to-end speech synthesis method based on deep learning provided by an example of the present invention comprises the following steps:

[0032] Step S1, perform Chinese word segmentation and part-of-speech tagging on the text data in the data set so that the model can better understand the contextual semantics. The text data set uses the CVTE-poly data set, which contains 845,254 sentences and 414 polyphones, and performs word segmentation and part-of-speech tagging on the Chinese sentences in the data set. Chinese word segmentation refers to finding the boundaries between words in a sentence and dividing them, and marking each Chinese character in the sentence with a word segmentation label. The types of labels include B (Begin), E (End), M (Middle) and S (Single), which respectively indicate that the Chinese character is located at the beginning position of a word, the Chinese character is located at the end position of a word, the Chinese character is located at the non-beginning and end position of a word, and the Chinese character alone constitutes a word. After the sentence is preprocessed by word segmentation, the label can be converted into a word segmentation embedding feature based on the word segmentation label. The Chinese part of speech refers to marking a correct word type for each word on the result of Chinese word segmentation, such as noun, verb, adjective, etc. After obtaining the Chinese part-of-speech tagging results through text preprocessing, we tag each Chinese character in the sentence with the part-of-speech tag of the word it belongs to, and then convert it into a part-of-speech embedding feature.

[0033] Step S2 is as follows Figure 2 As shown in the figure, the Chinese sentences in the dataset are input into the pre-trained model MiniRBT to obtain the semantic feature sequence.

[0034] The pre-trained model MiniRBT accepts the original Chinese character sequence as input and outputs a series of semantic features. Before embedding the sequence, the character embedding layer and the position embedding layer process the input character sequence respectively. The following Transformer block converts the embedding sequence into a semantic feature sequence.

[0035] The MiniRBT model is pre-trained on a large amount of unlabeled data, divided into two tasks: predicting masked input characters and predicting the next sentence, and learning semantic representations from raw character sequences.

[0036] The MiniRBT network consists of two modules, an input representation module that converts Chinese characters into feature vectors and a main network module that performs feature operations.

[0037] The [CLS] tag is a special tag set by the MiniRBT network to identify a functional position and is always placed at the front of the sentence. Similarly, [SEP] is also a special tag set by MiniRBT to indicate the separation between two sentences. The text input to MiniRBT is first converted into the character embedding feature corresponding to a single Chinese character in units of characters. The network then adds the sentence information of the text, that is, the label 0 or 1 indicates that the current text belongs to the first sentence or the second sentence, and then converts it into a sentence embedding feature according to the label. Finally, the position embedding feature corresponding to each Chinese character is obtained. MiniRBT presets that the maximum number of text characters processed each time is 512, so the model pre-initializes 512 embedding vectors for representing the position. Each character only needs to select the vector of the corresponding position as its own position embedding feature according to its position in the sentence. Finally, each Chinese character in the sentence is represented as the sum of each character embedding feature, sentence embedding feature, and position embedding feature, thereby obtaining a feature sequence of the Chinese sentence.

[0038] The main network module is composed of multiple network models stacked similar to the encoder in Transformer, including multiple multi-head self-attention modules, layer normalization, fully connected layers, residual connections and other network structures. By setting different parameters such as the number of stacked Transformer encoders in the MiniRBT network, the number of heads of the multi-head attention mechanism in the Transformer encoder, and the dimension of the word vector in the network, MiniRBT networks with various parameter values ​​can be constructed. Generally, the larger the parameter value, the stronger the model capability.

[0039] In order to be deployed on a low-resource computing platform, the present invention uses a small Chinese pre-training model, sets the number of heads of the multi-head attention mechanism in the Transformer encoder to 8, and the dimension of the word vector in the network to 512.

[0040] Step S3 is as follows Figure 3 As shown in Figure 1, the word segmentation embedding feature sequence and part-of-speech embedding feature sequence corresponding to the text are added to the semantic features output by MiniRBT, which can be regarded as the context feature vector corresponding to each Chinese character in the text.

[0041] The semantic feature output by MiniRBT is denoted as x=x mini . Chinese word segmentation tags can also be converted into vectors through embedding. For each character, we represent its tokenized information vector as x token We then concatenate it with the character vector before the final prediction, i.e. x = [x mini ;x token ].

[0042] After word segmentation, part-of-speech tagging assigns a tag to each Chinese word. For each character, we assume that its tag is inherited from the word it belongs to. Tags can also be converted into vectors through embedding. For each character, we represent the vector of its POS tag information as x pos We then concatenate the character vector x = [x mini ;x token ;x pos ].

[0043] Step S4 is as follows Figure 3 As shown, the obtained feature vector is classified by a classifier composed of two fully connected layers, and the final polyphone pronunciation is output.

[0044] A two-layer fully connected network is used to predict the pronunciation of polyphonic characters based on the MiniRBT output sequence. The first fully connected layer is shared by all polyphonic characters. As for the second fully connected output layer, it is not shared. Each polyphonic character has a separate output layer with the number of units equal to the number of possible pronunciations. The classifier is trained using the Softmax cross entropy loss method. The model predicts the label distribution through p.

[0045] p = softmax(W x+b)

[0046] Where W,b are the weights and biases of the fully connected layer.

[0047] During training, we use cross entropy to measure the loss between the predicted label distribution p and the one-hot encoding of the label y:

[0048]

[0049] Where C represents the total number of all pronunciations collected from all polyphones.

[0050] For the training process, we first need to construct polyphone disambiguation training samples for the model. First, we process the polyphone text data corresponding to each polyphone in the training data set separately.

[0051] According to the pronunciation of polyphones, the polyphone text is divided into several sets S1,…,S m , where m is the number of pronunciations of the polyphone. Here, each set should contain at least two polyphone text data. If the data of a certain pronunciation of the polyphone cannot be collected into two data, all the data of the polyphone will be discarded and not included in the training data range.

[0052] A piece of data is randomly extracted from each text set of the polyphone without replacement, so that a total of m polyphone texts are obtained, and the pronunciations of the polyphones in these texts are different. Use symbols t1,…,t m express.

[0053] A text set of the polyphone is randomly selected, and a polyphone text t is extracted from it. t is used as the target polyphone text for the polyphone pronunciation to be confirmed, and the actual pronunciation of the polyphone is used as the output target p of the model.

[0054] Then t1,…,t m ,t and p form a training sample of the model in this chapter. The combination t1,…,t m ,t is the input of the model, and p is the target output of the model.

[0055] During inference, to restrict the selection of polyphones from candidate pronunciations, we use a hard mask m = [m1, ···, m C ], except for the candidate pronunciation which is 1, all other pronunciations are 0. Then the final pronunciation is determined according to the maximum value of p⊙m, where ⊙ represents element-by-element multiplication.

Claims

1. A Chinese polyphone disambiguation method based on a small Chinese pre-training model mainly includes the following steps: S1: Segment and tag the Chinese sentences in the dataset; S2: Input the Chinese sentences in the dataset into the pre-trained model MiniRBT to obtain the semantic feature sequence; S3: Add the word segmentation embedding feature sequence and part-of-speech embedding feature sequence corresponding to the text to the semantic features output by MiniRBT, which can be regarded as the context feature vector corresponding to each Chinese character in the text; S4: The obtained feature vector is classified through a classifier composed of two fully connected layers, and the final polyphonetic pronunciation is output.

2. A Chinese polyphone disambiguation method based on a small Chinese pre-training model according to claim 1, characterized in that : In the step S1, the text data set of the present invention uses the CVTE-poly data set, which contains 845254 sentences, 414 polyphones, and the Chinese sentences in the data set are segmented and POS tagged. Chinese word segmentation refers to finding the boundary between words in a sentence, and dividing it, and marking each Chinese character in the sentence with a segmentation label, and the types of labels include B (Begin), E (End), M (Middle) and S (Single), which respectively represent the four situations that the Chinese character is located at the starting position of a certain word, the Chinese character is located at the end position of a certain word, the Chinese character is located at the non-starting and ending position of a certain word, and the Chinese character is alone to form a word. After the sentence is preprocessed by word segmentation, the label can be converted into a word segmentation embedding feature only according to the word segmentation label. Chinese part of speech refers to labeling each word with a correct word type, such as noun, verb, adjective, etc., based on the result of Chinese word segmentation. After obtaining the Chinese part of speech tagging results through text preprocessing, we label each Chinese character in the sentence with the part of speech label of the word it belongs to, and then convert it into part of speech embedding features.

3. A Chinese polyphone disambiguation method based on a small Chinese pre-training model according to claim 1, characterized in that : In step S2, the Chinese sentences in the data set are input into the pre-trained model MiniRBT to obtain a semantic feature sequence, as follows: The pre-trained model MiniRBT accepts the original Chinese character sequence as input and outputs a series of semantic features. Before embedding the sequence, the character embedding layer and the position embedding layer process the input character sequence respectively. The following Transformer block converts the embedding sequence into a semantic feature sequence. The MiniRBT model is pre-trained on a large amount of unlabeled data, divided into two tasks: predicting masked input characters and predicting the next sentence, and learning semantic representations from raw character sequences.

4. A Chinese polyphone disambiguation method based on a small Chinese pre-training model according to claim 1, characterized in that : In step S3, the word segmentation embedding feature sequence and part-of-speech embedding feature sequence corresponding to the text are added to the semantic features output by MiniRBT, which can be regarded as the context feature vector corresponding to each Chinese character in the text. The semantic feature output by MiniRBT is denoted as x=x mini . Chinese word segmentation tags can also be converted into vectors through embedding. For each character, we represent its tokenized information vector as x token We then concatenate it with the character vector before the final prediction, i.e. x = [x mini ;x token ]. After word segmentation, part-of-speech tagging assigns a tag to each Chinese word. For each character, we assume that its tag is inherited from the word it belongs to. Tags can also be converted into vectors through embedding. For each character, we represent the vector of its POS tag information as x pos We then concatenate the character vector x = [x mini ;x token ;x pos ].

5. A Chinese polyphone disambiguation method based on a small Chinese pre-training model according to claim 1, characterized in that : In step S4, the feature vector obtained in step S3 is classified by a classifier composed of two fully connected layers, and the final polyphone pronunciation is output. A two-layer fully connected network is used to predict the pronunciation of polyphonic characters based on the MiniRBT output sequence. The first fully connected layer is shared by all polyphonic characters. As for the second fully connected output layer, it is not shared. Each polyphonic character has a separate output layer with the number of units equal to the number of possible pronunciations. The classifier is trained using the Softmax cross entropy loss method. The model predicts the label distribution through p. p=softmax(Wx+b) Where W, b are the weights and biases of the fully connected layer. During training, we use cross entropy to measure the loss between the predicted label distribution p and the one-hot encoding of the label y: Where C represents the total number of all pronunciations collected from all polyphones. During inference, to restrict the selection of polyphones from candidate pronunciations, we use a hard mask m = [m1, ···, m C ], except for the candidate pronunciation which is 1, all other pronunciations are 0. Then the final pronunciation is determined according to the maximum value of p⊙m, where ⊙ represents element-by-element multiplication.

Images