A neural speech transliteration method based on distinctive features

Abstract

The present invention relates to a neural speech transliteration method based on distinctive features, and belongs to the field of speech recognition technology. The method first recognizes audio using end-to-end speech recognition technology, and then performs regular conversion on the recognition sequence based on distinctive features. Among them, the end-to-end speech recognition part receives English speech and recognizes the corresponding International Phonetic Alphabet IPA sequence; the rule part converts the English IPA sequence into the closest Chinese Pinyin initial / final IPA sequence according to the distinctive features, and then converts the IPA sequence into a Pinyin sequence according to the mapping rules of IPA and Pinyin. The method combines distinctive features with end-to-end speech recognition technology, which is not only helpful for cross-border communication and English learning, but also provides fun for people in life, and has good practical applicability.

Description

Technical Field

[0001] The invention relates to a neural speech transliteration method based on distinctive features, belonging to the technical field of speech recognition processing. Technical Background

[0002] With the rapid development of computer technology, computer technology has been applied to various fields of society. At the same time, problems such as difficulty in processing massive voice data and difficulty in human-computer interaction have also arisen.

[0003] The goal of speech recognition is to automatically convert human speech into text through computers. With the further development of artificial intelligence and deep learning technology, speech recognition technology has made significant progress. Most of the existing speech recognition technologies use methods based on deep learning. Among them, distinctive features are various features that can distinguish language units based on the natural features of speech. Usually, for a phoneme, it is marked as "+" or "-" according to whether it has this feature. This feature makes it possible to construct a vector representation for the phoneme.

[0004] Transliteration is a concept that is well known to people and has a wide range of applications in life. At present, the research on transliteration mainly focuses on text, usually the transliteration of place names, personal names and some proper nouns in machine translation. However, the transliteration of the entire text, especially the transliteration of speech to text, is extremely rare. At present, the method of transliteration is mainly rule-based, which maps English sequences to Chinese character sequences by comparing English phonemes and Chinese pinyin, or directly maps English words to Chinese characters according to the written rules. However, this mapping method appears to be relatively rigid in some generated texts, and they also process text.

[0005] Although the application scenarios of voice transliteration are not as many as those of text transliteration, it is a technology that can provide convenience for people's daily lives. For example, in English learning, voice transliteration is often an effective mnemonic device. Voice transliteration can also provide convenience for people's temporary overseas communication. For example, voice transliteration of teaching audio can help memory. In addition, voice transliteration can also bring some fun to people in life. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies in the prior art, to solve the lack of transliteration methods in the field of speech, and to provide a new technical approach for the research and development of text transliteration, and to creatively propose a neural speech transliteration method based on distinctive features.

[0007] The innovation of the present invention is that it combines the neural speech recognition method and the distinctive features for the first time, and realizes the transliteration of English speech into Chinese Pinyin text. Among them, the bridge between neural speech recognition and the rule processing based on the distinctive features is IPA (International Phonetic Alphabet). In speech recognition, English speech is recognized as IPA, and in rule processing, similar IPAs are replaced by calculating the similarity between IPAs, and finally converted into Pinyin sequences according to the mapping rules between IPA and Pinyin.

[0008] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0009] A neural speech transliteration method based on distinctive features comprises the following steps:

[0010] First, the English text sequence corresponding to the training set is converted into an IPA sequence, and the converted data set is used to train a neural speech recognition model, wherein the neural speech recognition model can adopt a transformer model.

[0011] Then, the speech features that need to be transliterated are input into the trained neural speech recognition model to obtain the IPA sequence corresponding to the English speech.

[0012] Afterwards, the vector representation of each English IPA character in the output sequence is determined based on the distinctive features; at the same time, the initial consonant / final vowel of the Chinese pinyin is found to have its corresponding IPA, and the vector representation of the initial consonant / final vowel is determined based on the distinctive features.

[0013] Then, for each IPA vector in the output sequence, the Euclidean distance between it and each Chinese Pinyin initial consonant and final is calculated to measure the similarity between two IPA characters. The Chinese Pinyin initial consonant / final IPA character closest to the English IPA character in the output sequence is used to replace it to obtain the IPA sequence of Chinese Pinyin initial consonants / finals.

[0014] Finally, according to the initial / final consonants of Chinese Pinyin and their IPA mapping rules, IPA is replaced with the corresponding initial consonants or finals to obtain the initial / final consonant sequence of Pinyin. The initial / final consonant sequence is combined to obtain the final output Pinyin sequence.

[0015] Among them, if the initial consonant is followed by a final, the space between the two is removed and the two are combined together, such as: "hao" is combined into "hao"; if the initial consonant is followed by a final, the space between the two is removed and the two are combined together, such as "ks" is combined into "ks".

[0016] Beneficial Effects

[0017] The method of the present invention combines advanced speech recognition technology with distinctive features in linguistics, filling the gap in the field of speech transliteration. At the same time, the speech is transliterated into a pinyin sequence, which alleviates the problem of stiff pronunciation after transliteration. Some other improvements such as "ks" are not output as "ke si", but merged into "ks", which is also easy to understand for people whose native language is Chinese, and greatly shortens the length of the transliteration sequence. Using a pinyin sequence instead of an IPA sequence as the final output facilitates the understanding of the transliteration results by people whose native language is Chinese, and there is no need to learn the International Phonetic Alphabet. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the process of the present invention.

[0019] Figure 2 This is a partial screenshot of the IPA-distinctive feature annotation table used in the distinctive feature rules part of this method.

[0020] Figure 3 Screenshot of the initial consonants (right) / final consonants (left) used in the distinctive feature rules part of this method.

[0021] Figure 4 The results of data processing before training the neural speech recognition model for this method are shown. DETAILED DESCRIPTION

[0022] The present invention is further described in detail below with reference to the accompanying drawings.

[0023] like Figure 1 As shown, a neural speech transliteration method based on distinctive features comprises the following steps:

[0024] Step 1: Process the dataset and convert the English text sequence of text labels into IPA sequence. The sentence representations before and after the conversion are as follows Figure 4 As shown. Use this dataset to train a neural speech recognition model, which can use a transformer model.

[0025] The waveform file of the audio signal of the speech segment is subjected to feature extraction. The feature extraction method may adopt MFCC (Mel-frequency cepstral coefficients) or other audio feature extraction methods.

[0026] After the audio signal is feature extracted, the feature value is a two-dimensional array, where the first dimension represents time and the second dimension represents frequency. In the time dimension, the feature of each time step t is x t , x t is a vector.

[0027] Step 2: Input the extracted feature sequence into the neural speech recognition model to identify the corresponding IPA sequence.

[0028] Step 3: Determine the vector representation of each IPA character in the output sequence based on the distinctive features. Figure 2 As shown, each IPA character corresponds to a label for each distinctive feature. "+" is represented as the number 1, "-" is represented as the number -1, and "0" is represented as the number 0. In this way, each IPA character is represented as a vector containing only 0, -1, and 1.

[0029] At the same time, according to the distinctive features, the vector representation of the initial consonant / final vowel of each Chinese Pinyin is determined. The calculation of the vector representation of the initial consonant / final vowel of the Chinese Pinyin is the same as step 4. Figure 3 As shown, some initials and finals are represented by multiple IPA characters. When calculating the vector representation of the initial / final, the vector of each IPA character is averaged to obtain the vector of the initial / final.

[0030] Step 4: Calculate the distance between each IPA character and each initial / final in the output sequence, and replace the IPA character in the output sequence with the closest initial / final IPA.

[0031] Among them, since the contribution of each distinctive feature is different, the weighted Euclidean distance is used to calculate the distance between vectors, and the weights of syllables, consonants, sonorants, consonants, voiced, glottal extension, glottal contraction and harshness are reduced; the weights of the tip of the tongue or the tongue surface and the tongue body are increased. For each IPA character in the output sequence, it is replaced with the closest initial or final IPA according to the obtained distance, thereby obtaining the IPA sequence of Hanyu Pinyin.

[0032] Step 5: According to the Chinese Pinyin IPA sequence obtained in step 4, the output sequence is represented as an initial consonant / final vowel sequence.

[0033] After the conversion is completed, words can be separated by " / ", and each IPA character in a word can be separated by a space. The corresponding relationship between Chinese Pinyin and IPA is as follows Figure 3 shown.

[0034] Step 6: Post-process the obtained initial consonant / final vowel sequence.

[0035] Among them, within a word, the initial consonant and the following final vowel are merged, and the initial consonant and the following initial consonant are merged to obtain the final output pinyin sequence.

[0036] Verification

[0037] The method of the present invention is applied to English speech translation, the English data set is TEDLIUMv2, and the transliteration results are shown in Table 1.

[0038] Table 1: English phonetic transliteration results

[0039]

[0040]

[0041] It can be seen that the method of the present invention can complete the basic transliteration operation, the sentence length of the transliteration result is moderate, and the pinyin composed of initials and finals can be combined to form some non-standard pinyin, such as "rei", which is more flexible than the pronunciation that can be expressed by standard pinyin and specific Chinese characters. The output is pinyin, which is closer to Chinese and does not require learning international phonetic symbols.

Claims

1. A neural speech transliteration method based on distinctive features, characterized in that: The following steps are involved: Step 1: Convert the English text sequence corresponding to the training set into the International Phonetic Alphabet (IPA) sequence, and use the converted data set to train the neural speech recognition model; Step 2: Input the speech features to be transliterated into the trained neural speech recognition model to obtain the IPA sequence corresponding to the English speech; Step 3: Determine the vector representation of each English IPA character in the output sequence based on the distinctive features; at the same time, find the corresponding IPA for the initial consonant / final vowel of the Chinese pinyin, and determine the vector representation of the initial consonant / final vowel based on the distinctive features; Step 4: For each IPA vector in the output sequence, calculate the Euclidean distance between it and the initial consonant and final of each Chinese pinyin to measure the similarity between two IPA characters; Replace the Chinese Pinyin initial / final IPA characters that are closest to the English IPA characters in the output sequence to obtain the IPA sequence of Chinese Pinyin initial / final consonants; Step 5: According to the initial consonant / final vowel of Pinyin and its IPA mapping rule, IPA is replaced with the corresponding initial consonant or final vowel to obtain the initial consonant / final vowel sequence of Pinyin; the initial consonant and final vowel sequences are combined to obtain the final output Pinyin sequence; Among them, if the initial consonant is followed by a final, the space between the two is removed and they are combined together; if the initial consonant is followed by a final, the space between the two is removed and they are combined together.

2. A neural speech transliteration method based on distinctive features as claimed in claim 1, characterized in that: The neural speech recognition model uses the transformer model.

3. A neural speech transliteration method based on distinctive features as claimed in claim 1, characterized in that: In step 3, each IPA character is labeled in each distinctive feature, and "+" is represented as the number 1, "-" is represented as the number -1, and "0" is represented as the number 0. Each IPA character is represented as a vector containing only 0, -1, and 1.

4. The neural speech transliteration method based on distinctive features as claimed in claim 1, characterized in that: In step 3, when the initial consonant and the final are represented by multiple IPA characters, when calculating the vector representation of the initial consonant / final, the vector of each IPA character is averaged to obtain the vector of the initial consonant / final.

5. The neural speech transliteration method based on distinctive features as claimed in claim 1, characterized in that: In step 4, when calculating the distance between vectors, the weighted Euclidean distance is used, and the weights of syllables, consonants, sonorants, consonants, voiced sounds, glottal extension, glottal contraction, and harshness are reduced; and the weights of the tongue tip or tongue surface are increased.

Images