A multilingual phoneme recognition method based on iterative fusion of phoneme pairs

Abstract

The present invention relates to a multilingual phoneme recognition method based on iterative fusion of phoneme pairs, and belongs to the technical field of audio signal processing. Multilingual phoneme resources with different resource degrees are effectively utilized through multilingual international phonetic symbol conversion, and after uniformly mapping to the data set of the directly acquired language for non-mapped IPA phoneme cluster expansion, the reconstructed new data set replaces the traditional single universal data set for the fusion phoneme set. Taking the human vocal tract phonation as a constraint, the phoneme high-order linear prediction peak frequency band phonetic features constrained by human pronunciation resonance are constructed, and this phonetic feature is combined with the acoustic feature MFCC to be fused into a new type of phoneme distinguishing feature. Through the minimum co-occurrence phoneme pair iterative reduction algorithm or the phoneme feature cosine similarity pair iterative reduction algorithm. Finally, multilingual phoneme labels are constructed based on the fused complete phoneme set, and the new phoneme distinguishing features of the label language training set and the test set are extracted to realize the phoneme recognition of variable-length speech automatic alignment.

Description

Technical Field

[0001] The invention relates to a multilingual phoneme recognition method based on iterative fusion of phoneme pairs, and belongs to the technical field of audio signal processing. Background Art

[0002] Compared with traditional continuous time recognition models (such as hidden Markov models), deep learning neural network technology has been widely used due to its superior performance. The use of connected time classification CTC eliminates the need for mandatory alignment of speech sequences and speech labels, and achieves good results in phoneme recognition. Phoneme recognition is the automatic recognition between phoneme labels and phoneme feature vectors. Using phonetic features as input features for phoneme recognition has never achieved good recognition results. New discriminative features are needed to improve the discrimination of phoneme classification. Different from speech recognition, phoneme recognition does not focus on the semantics of speech; different from speaker recognition, phoneme recognition has little correlation with human pronunciation habits, sound quality and timbre. The accuracy and scalability of phoneme recognition systems play an important role in the development of automatic speech recognition, language recognition and other systems. However, phoneme recognition models face a series of problems in practical applications, especially the resource problem of phoneme recognition model training and learning, the scalability problem of cross-language application of phoneme recognition models, and the problem of phoneme recognition model accuracy. Clustering phonemes and building a fused multilingual phoneme recognition model can effectively utilize phoneme resources, enhance model scalability, and improve the performance of phoneme recognition in low-resource languages. Unlike the number of graphemes or words in a language, the total number of phonemes contained in a single language is very limited and considerable. In addition, due to the constraints of the human vocal tract and the different pronunciation habits of different languages, the phoneme sets of different languages ​​have a large degree of intersection and the special phoneme sets are small. This is reflected in the fact that the phoneme sets of different languages ​​in the same family of standard languages ​​are relatively small, and the phoneme sets of non-standard languages ​​(dialects) in different families of languages ​​are relatively different. This is a feature of the feasibility of merging multiple language phoneme sets into a complete multilingual recognition model, providing natural favorable conditions for flexible multi-level fusion of multiple languages. Combined with the alignment-free characteristics of the CTC network, the phoneme recognition model of variable-length speech came into being with more superior scalability and accuracy. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a multilingual phoneme recognition method based on iterative fusion of phoneme pairs to solve the above-mentioned problem.

[0004] The present invention constructs an extensible phoneme recognizer that can be trained with less training resources and has guaranteed accuracy. The main technical problem solved is that a method for obtaining and mapping multilingual phoneme sets that can be used for fusion is proposed, and on this basis, multiple language phoneme sets are flexibly and accurately fused in a multi-level manner, language phoneme labels are constructed under the complete fused phoneme set, and accurate fusion features that distinguish phonemes are constructed. The phoneme labels and phoneme distinguishing features are input into the connection time classification CTC network to construct an accurate and effective multilingual phoneme recognizer. When obtaining the phoneme set, the correlation between phonemes of different languages ​​is effectively used to intervene in the phoneme mapping conversion. In phoneme linguistics, different language phoneme sets are first mapped to a certain data set, and then this mapping set is expanded to construct a new data set to replace the traditional single universal data set, so as to enhance the adaptability of the data set to different languages. The unified mapping of the phoneme set is independent of the phoneme set fusion, and is a pre-processing of the phoneme set fusion before the phoneme obtains language phonetics knowledge based on a direct method. During fusion, with the guarantee of phoneme recognition accuracy as a prerequisite, the size of the phoneme set is reduced to avoid the fusion set being too large and the performance degradation of the model caused by competition among many highly similar phoneme classes in phoneme recognition. When reducing the phoneme set, the application of language phoneme arrangement statistics that allows a small number of phoneme errors to achieve the best overall global phoneme recognition accuracy is considered, and the language phoneme set reduction method of iterative small probability phoneme pairs is used to reduce the phoneme set. For the language phoneme grid grapheme encoding language characterization application, the cosine distance between phonemes based on the new phoneme distinguishing feature represents the phoneme similarity, and the phoneme pairs with the greatest similarity are merged and reduced; combined with the statistical differences of the phoneme set fusion model, after the phoneme set mapping, the problem of contradictory phoneme and phonetic knowledge attributes caused by direct phonetic knowledge fusion of multiple phoneme sets is avoided, and the number of iterations of phoneme set reduction and multi-level fusion of phoneme sets are determined in a way that maximizes the amount of interactive information. After fusion, CTC supervised learning of phoneme labels and phoneme features under the complete set is constructed based on the fused complete phoneme set to obtain a multilingual phoneme recognizer with high recognition accuracy.

[0005] The technical solution of the present invention is: a multilingual phoneme recognition method based on iterative fusion of phoneme pairs, the specific steps are:

[0006] Step 1: Obtain multiple foreign language phoneme corpora of different resource levels. The construction of the phoneme recognizer is limited by the corpus resources of the phoneme recognition model. The richness of the labeled corpus used for model supervised learning represents the resource level of the corpus. The corpus with higher resource level is directly obtained as the main language phoneme set for training the first language. The corpus indirectly obtained by non-IPA phoneme-IPA phoneme dictionary encoding is used as the extended language phoneme set for training the second or even more languages. The corpus indirectly obtained by grapheme-IPA phoneme dictionary encoding is used as the extended language phoneme set for training the third or even more languages.

[0007] Three phoneme corpus acquisition methods are used together to jointly acquire different types of corpus resources. For the phoneme corpus of a language with high precision and rich content, a direct acquisition method is used to obtain the language's speech set and phoneme labels as the language's phoneme corpus. For the language's phoneme corpus with sufficient precision and rich content, and the language has a language-specific phoneme standard representation, it is proposed to construct a dictionary encoding of the phoneme representation of the language-specific phoneme representation standard to the International Phonetic Alphabet IPA representation, and the proposed phoneme-phoneme dictionary encoding method is used to obtain the language's speech set and IPA phoneme labels as the language's phoneme corpus. For languages ​​that have no better phoneme data to replace and there is a clear mapping between the language's speech phonemes and graphemes, it is proposed to construct an IPA phoneme transcription dictionary that represents the language graphemes, and the proposed grapheme-phoneme dictionary encoding method is used to obtain the language's phoneme set and IPA phoneme labels as the language's phoneme corpus. It is particularly noteworthy that in the proposed method for jointly acquiring multilingual phoneme corpora, when acquiring isolated word species (most of the Sino-Tibetan languages), it is necessary to first decompose the single isolated word species phoneme into multiple continuous word class IPA phonemes, and then reconstruct the language phoneme set.

[0008] Step 2: Unified mapping of the obtained phoneme sets of corpora of different languages: Based on the phoneme set obtained in Step 1, the phoneme labels of the corpus resources obtained by non-IPA phoneme-IPA phoneme dictionary encoding and the phoneme labels of the corpus obtained by grapheme-IPA phoneme dictionary encoding are uniformly mapped to the phoneme-phoneme label representation of the main phoneme set of the first language based on linguistic knowledge.

[0009] In the Step 2, the modules for unified mapping of phoneme sets of corpora of different languages ​​are separated and independent: independent phoneme set mapping modules are constructed based on different pronunciation modes of phonemes including tone, vibrato, glissando and nasal consonant, so that the phoneme set mapping is independent of the acquisition of the phoneme set. It is proposed to separate the mapping of the phoneme set from the outside of the phoneme set fusion module, modularize the mapping of the phoneme set, and connect the unified phoneme mapping module to the phoneme set acquisition module in parallel to adapt to multiple different monolingual phoneme sets acquired in different ways.

[0010] A unified mapping scheme for phoneme sets of corpora of different languages: In the scheme mapping, the phoneme set of the main language obtained by direct acquisition is selected as the basic phoneme set, and the phoneme set of the extended language obtained by indirect acquisition is selected as the superimposed multi-level fusion phoneme set. The IPA format phoneme complete symbol set of the extended language obtained by phoneme-phoneme encoding is mapped to the main language phoneme complete symbol set obtained by direct acquisition. Similarly, the IPA format phoneme complete symbol set of the extended language obtained by grapheme-phoneme encoding is mapped to the main language phoneme complete symbol set obtained by direct acquisition.

[0011] The specific implementation method of unified mapping of phoneme sets of different language corpora is as follows: the number of phoneme symbols in a language phoneme set is very limited and observable. According to the principle of the uniformity of the human pronunciation organs and based on the knowledge of phoneme linguistics, the IPA phonemes of the language acquired indirectly are mapped to the highly similar phoneme sets of the language acquired directly. According to the principle of the mutual differences of the speaker's pronunciation organs and considering the language regionality, the phonemes of the language acquired indirectly that cannot be mapped are assigned language labels to form non-mapped IPA phoneme clusters. The Chinese-English linguistic knowledge mapping relationship ARPAbe is proposed and expressed as: 'a': 'ah', 'aa': 'ah', 'ei': 'eh', 'ai': 'ay', 'an': 'ah en', 'en': 'ax en', 'ang': 'ah nx', 'ao': 'aw', 'eng': 'ax nx', 'b': 'b', 'c': 'C_1', 'er': 'axr', 'ch': 'ch', 'd': 'd', 'f': 'f', 'e': 'ax', 'ee': 'ax', 'g': 'g', 'h': 'hh', 'i': 'iy', 'ia': 'C_2', 'ian': 'iy ah en', 'iang': 'iy ah nx', 'iao': 'iy aw', 'ie': 'y eh', 'ii': 'iy', 'in': 'iy nx', 'ing': 'iynx', 'iong': 'iy oy nx', 'iu': 'uw', 'ix': 'C_3', 'iy': 'C_4', 'iz': 'C_5', 'j': 'jh', 'k': 'k', 'l': 'l', 'm': 'm', 'n': 'n', 'o': 'oy', 'ong': 'oy nx', 'oo': 'oy', 'ou': 'ow', 'p': 'p', 'q': 'ch', 'r': 'r', 's': 's', 'sh': 'sh', 't': 't', 'u': 'uh', 'ua': 'uh ah', 'uai': 'wh aa ih', 'uan': 'uh ah en', 'uang': 'uh ah nx', 'ueng': 'uh ax nx', 'ui': 'wheh ih', 'un': 'wh eh en', 'uo': 'C_6', 'uu': 'uh', 'v': 'v', 'van': 'v ah en', 've': 'vax', 'vn': 'v en', 'vv': 'v', 'x': 'C_7', 'z': 'C_8', 'zh': 'zh'.The proposed English-Spanish linguistic knowledge mapping relationship ARPAbe is expressed as: 'J': 'iy', 'L': 'jh', 'RR': 'S_2', 'S': 's', 'S-1': 'S_1', 'S-2': 'S_3', 'T': 'th', 'T / ': 'ch', 'X': 'hh', 'a': 'aa', 'b': 'v', 'd': 'en 'jh', 'e': 'eh', 'f': 'f', 'g': 'g', 'gs': 's', 'h': 'h#', 'i': 'ih', 'j': 'ih', 'k': 'k', 'l': 'l', 'm': 'm', 'n': 'en', 'o': 'ow', 'p': 'p', 'q': 'k', 'r': 'S_4', 't': 't', 'u': 'uw', 'v': 'b', 'w': 'w', where 'S-2' is the phoneme corresponding to the special grapheme 'ü', and 'S-1' is the phoneme corresponding to the special grapheme '`'. Spanish has multiple pronunciations. Based on this phenomenon, a more complete phoneme-pronunciation multi-layer mapping method can be constructed to improve the completeness of the model phoneme dictionary.

[0012] Step 3: Construct new features to distinguish phonemes based on the constraints of human vocal tract sound production. Specifically, according to the mechanism of the speech sound production system, based on the multilingual phoneme set mapped to the first language IPA phoneme labels obtained in Step 2, all phonemes are refined into voiceless phonemes, nasal voiced phonemes, and non-nasal voiced phonemes. According to the speech sound production characteristics of different speech categories, new features with better phoneme-distinguishing characteristics are constructed.

[0013] Step 4: Based on the multilingual phoneme set obtained in Step 2, reduce the mapped IPA phoneme symbol set within the language, and reduce the size of the overall phoneme symbol set of the mapped main language, second language and third language phoneme sets respectively.

[0014] Step 5: Based on the reduced multilingual phoneme set obtained in Step 4, the main phoneme set directly obtained is used as the initial set, and combined with the indirectly obtained second language phoneme set for the first iterative first-level fusion. The new set formed after the fusion is then combined with the indirectly obtained third language phoneme set for the second iterative second-level fusion, and so on, to obtain more levels of multilingual phoneme set fusion in more languages.

[0015] Multi-level error-free fusion of phoneme sets in multiple languages: First, error-free fusion. Phoneme sets are fused without error to avoid vocabulary confusion between phoneme sets. The principle of reduction is: the phonemes within a single language are not merged, and the phonemes of non-mapped phoneme clusters with different language labels are not merged. Second, multi-level fusion with flexible expansion of the number of fused languages. The directly obtained reduced phoneme set is used as the initial set, and the first reduced phoneme set obtained indirectly is combined for the first iterative first-level fusion. The new set formed after the fusion is then iteratively fused with the remaining second reduced phoneme set for the second time. The new set of the second-level fusion is then iteratively fused with the remaining third phoneme set for the third time. The number of fused phoneme sets is the number of iterations and the number of fusion levels plus 1.

[0016] Step 6: Use the connection time classification network (CTC) to build a phoneme recognition network to realize the recognition of phoneme sequences with variable length and automatic alignment of multilingual speech.

[0017] The Step 3 is specifically as follows: constructing a discriminative feature quantity to characterize a single phoneme, dividing the phonemes into three categories: unvoiced, nasal voiced, and non-nasal voiced, combining the acoustic feature MFCC and the phoneme high-order linear prediction peak frequency band feature constrained by human pronunciation resonance to construct a new phoneme discriminative feature, T represents the number of speech frames, and the static characteristic discriminative feature of the phoneme in the nth frame Where n=1,2,…T,F n,0 Indicates the fundamental frequency of the nth frame, F n,1 ,F n,2 ,F n,3 represents the human voice constraint formant of the nth frame, W n,1 ,W n,2 ,W n,3 Indicates the bandwidth corresponding to the human voice constraint resonance peak of the nth frame, Represents the 13-dimensional MFCC of the nth frame. When the speech is nasal or unvoiced, the zero-pole frequency and its bandwidth are used to compensate for the first three formant frequencies and bandwidths of non-nasal voiced sounds. The static characteristics of the speech segment phonemes distinguish the features. Distinguishing features of dynamic characteristics of speech segment phonemes Speech segment phoneme distinguishing features

[0018] The Step 4 is specifically as follows: for the application target of the constructed phoneme recognition model, when it is applied to the language phoneme arrangement statistics that allows a small number of phoneme errors to achieve the best overall global phoneme recognition accuracy, a small number of phoneme speech recognition contributions in the multi-language phoneme symbol set are sacrificed, a similarity is defined, and the probability of two phonemes appearing in the phoneme set is calculated. When the probability of two phonemes in the same language phoneme driving set is small and similar at the same time, when the two phonemes appear at the same time in the phoneme driving set of different languages, the two phoneme probabilities are merged to reduce the phoneme with a smaller probability. For language phoneme lattice grapheme encoding, the phoneme language characterization accuracy is sacrificed to calculate the phoneme distinguishing feature F of phoneme o in language i, i = 0, 1, 2 (i,o) , based on the new phoneme distinguishing feature F (i,o) The cosine distance between the phonemes represents the phoneme similarity, and the phoneme pairs with the greatest similarity are merged and reduced. When reducing, the two phonemes are merged and represented by the phoneme symbol with a higher probability of appearing in the training set.

[0019] The Step 5 is specifically as follows:

[0020] In step S5, the fusion method of multiple language phoneme sets: the multi-level fusion method of phoneme sets is based on constructing a multi-level fusion algorithm of multiple complete phoneme sets with maximum mutual information. First, the language phoneme sets and the language phoneme symbol sets constituting the language phoneme sets are converted into random variables. When performing the fusion at a certain level, the expression of the mutual information of the two language phoneme sets fused at this level under different iteration reduction iteration numbers is calculated. The multi-level fusion of multiple phoneme sets is performed with the number of phoneme reduction iterations when this expression is the largest. The specific steps are:

[0021] Step 5.1: Find the random variables X of the phoneme set after iterative reduction i ,i=1,2 The amount of mutual information with X0

[0022] Step 5.2: Find the number of iterations E when the mutual information between the two language phoneme symbol sets is the largest i .

[0023] Step 5.3: Find the minimum E based on the maximum mutual information among all L languages i ,i=2,…,L value, this value is used as the optimal number of iteration reductions for all phoneme sets.

[0024] Step 5.4: The iteratively reduced phoneme symbol sets under the optimal number of iterative reductions are merged to determine the fused multilingual complete phoneme symbol set under the fusion random variable of the i-language phoneme symbol set.

[0025] The beneficial effects of the present invention are as follows: the present invention adopts three phoneme corpus acquisition methods to jointly acquire different types of corpus resources, and acquires low-resource corpus in an indirect way to reduce the independence of phoneme resources of different languages ​​and make up for the shortage of phoneme resources of different languages ​​acquired by existing acquisition methods; the mapping of phoneme sets is modularized, so that the unified mapping of phonemes is effectively applicable to multiple different monolingual phoneme sets acquired in different ways, and the unified mapping of phoneme sets of independent modules of phoneme set mapping eliminates the loss of phoneme interaction between phoneme sets caused by different pronunciation modes between phonemes of the same language phoneme set in the corresponding language of the module; the proposed mapping scheme makes full use of the precise characteristics of the phoneme set speech and phoneme labels of the main language acquired by the direct acquisition method, and reconciles the language phoneme set acquired by the indirect method. The fitting degree of phoneme supervised learning of phoneme labels is improved; phonemes are divided into three categories: unvoiced, nasal voiced, and non-nasal voiced to construct the discriminative feature quantity that characterizes a single phoneme, and a new type of phoneme discriminative feature is constructed by combining the acoustic feature MFCC and the phoneme high-order linear prediction peak frequency band feature constrained by human pronunciation resonance, which inherits the acoustic features of the speech segment level but is different from the speech acoustic features, making it more discriminative in phoneme classification; the overall multi-level fusion of phoneme recognition in this paper increases the adaptability of supervised learning of corpus resources in different languages, and constructs a single multilingual phoneme recognizer, which effectively replaces the traditional complicated multilingual application phoneme recognition of multiple monolingual phoneme recognizers in parallel, introduces new phoneme discriminative features, and achieves a significant improvement in recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is the overall flow chart of the present invention;

[0027] Figure 2 is a flow chart of the phoneme phonetics features of the present invention;

[0028] Figure 3 This is a flow chart of the multilingual phoneme collection fusion of the present invention;

[0029] Figure 4 It is a network diagram of the multilingual phoneme recognition supervised learning of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0031] Example 1: Figure 1 As shown, a multilingual phoneme recognition method based on iterative fusion of phoneme pairs, the specific steps are:

[0032] Step 1: Obtain a multilingual phoneme set;

[0033] The TIMIT English phoneme corpus with higher resource content includes phoneme annotations in ARPAbet format of audio 16kHz sampling rate speech sequence sampling-level manual segmentation labels, and directly obtains its highly accurate speech phoneme labels as the English phoneme set, and the corresponding phoneme symbol set is C0; the Thchs30 corpus with higher resource content includes phoneme labels annotated in the Chinese vowel and consonant format at the speech segment level, and constructs a complete dictionary from vowels and consonants to the International Phonetic Alphabet IPA. The phoneme IPA labels at the speech segment level are obtained as the Chinese phoneme set through dictionary encoding from vowels and consonants to the International Phonetic Alphabet, and the corresponding phoneme symbol set is C1; the TSC corpus with lower resource content includes speech grapheme labels at the language segment level, and the corresponding Spanish has its own clear mapping relationship between sound and grapheme. The IPA phoneme labels at the Spanish speech segment level are obtained by using the language's own grapheme-to-phoneme mapping relationship as the Spanish phoneme set, and the corresponding phoneme symbol set is C2.

[0034] Step 2: Language phoneme set linguistic knowledge mapping

[0035] The English language phoneme set TIMIT and the Spanish language phoneme set TSC are language phoneme sets of the continuous word phoneme recognition neighborhood, and the Chinese language phoneme set Thchs30 belongs to the isolated word phoneme recognition neighborhood language phoneme set, so it is necessary to decompose some phonemes in Thchs30 into multiple continuous word neighborhood phonemes.

[0036] The language phoneme symbol set consisting of the isolated word vowel phoneme set is transcribed into the international phonetic symbol format and then mapped to the TIMIT phoneme set. After the special pronunciation of Chinese language is expanded, the mapped Thchs30 language phoneme symbol set C′1 contains the mappable phoneme cluster C′ 1,IPA and unmappable unmapped labeled IPA clusters (unmapped IPA phoneme format with language tag "C")

[0037] The mapping relationship between the indirectly acquired Thchs30 vowels and finals transcribed by the International Phonetic Alphabet and mapped to the ARPAbe representation of TIMIT acquired directly is:'a':'ah','aa':'ah','ei':'eh','ai':'ay','an':'ahen','en':'ax en','ang':'ah nx','ao':'aw','eng':'ax nx','b':'b','c':'C_1','er':'axr','ch':'ch','d':'d','f':'f','e':'ax','ee':'ax','g':'g','h':'hh','i':'iy','ia':'C_2','ian':'iy ah en','iang':'iy ah nx', 'iao': 'iy aw', 'ie': 'y eh', 'ii': 'iy', 'in': 'iy nx', 'ing':

[0038] 'iy nx', 'iong': 'iy oy nx', 'iu': 'uw', 'ix': 'C_3', 'iy': 'C_4', 'iz': 'C_5', 'j': 'jh', 'k': 'k', 'l': 'l', 'm': 'm', 'n': 'n', 'o': 'oy', 'ong': 'oy nx', 'oo': 'oy', 'ou': 'ow', 'p': 'p', 'q': 'ch', 'r': 'r', 's': 's', 'sh': 'sh', 't': 't', 'u': 'uh', 'ua': 'uh ah', 'uai': 'wh aa ih', 'uan': 'uh ah en', 'uang': 'uh ah nx', 'ueng': 'uh axnx', 'ui': 'wh eh ih', 'un': 'wh eh en', 'uo': 'C_6', 'uu': 'uh', 'v': 'v', 'van': 'v ahen', 've': 'v ax', 'vn': 'v en', 'vv': 'v', 'x': 'C_7', 'z': 'C_8', 'zh': 'zh'.

[0039] Similarly, based on linguistic knowledge, the TSC phoneme symbol set obtained by transcribed grapheme-phoneme is mapped to the TIMIT phoneme set, and expanded by Spanish special pronunciation. The mapped TSC phoneme symbol set C′2 contains the mappable phoneme cluster C′ 2,IPA and unmappable unmapped labeled IPA clusters (unmapped IPA phoneme format with language tag "S")

[0040] The mapping relationship between the indirectly acquired TSC international phonetic symbols and the directly acquired TIMIT is as follows: 'J': 'iy', 'L': 'jh', 'RR': 'S_2', 'S': 's', 'S-1': 'S_1', 'S-2': 'S_3', 'T': 'th', 'T / ': 'ch', 'X': 'hh', 'a': 'aa', 'b': 'v', 'd': 'en 'jh', 'e': 'eh', 'f': 'f', 'g': 'g', 'gs': 's', 'h': 'h#', 'i': 'ih', 'j': 'ih', 'k': 'k', 'l': 'l', 'm': 'm', 'n': 'en', 'o': 'ow', 'p': 'p', 'q': 'k', 'r': 'S_4', 't': 't', 'u': 'uw', 'v': 'b', 'w': 'w', where 'S-2' is the phoneme corresponding to the special grapheme 'ü', and 'S-1' is the phoneme corresponding to the special grapheme '`'. Spanish has multiple pronunciations. Based on this phenomenon, a more complete phoneme-pronunciation multi-layer mapping method can be constructed to improve the completeness of the model phoneme dictionary.

[0041] Step 3: Construction of phoneme-discriminating features;

[0042] Constructing phoneme distinctive features includes constructing phoneme phonetic features and concatenating phoneme acoustic features.

[0043] Step 3.1: Construct phonetic features:

[0044] According to the mechanism of speech sound system and the peak frequency band characteristics of human voice constraints, a method combining speech short-time spectrum and vocal tract modeling is used to efficiently and accurately determine the fundamental frequency of voiced phonemes, the zero-pole frequency of voiceless and nasal voiced phonemes, the first three resonance peaks of non-nasal voiced sounds and their first-order differences, and other phonetic parameters to characterize different phonemes. When extracting parameters, a high-order optimal LPC peak estimation method based on human pronunciation digital resonance constraints is used to construct phonetic features such as Figure 2 shown.

[0045] Step 3.1: Fusion of phoneme acoustic features and phoneme phonetic features

[0046] A new type of phoneme discriminative feature is constructed by combining the acoustic feature MFCC and the phoneme high-order linear prediction peak frequency band characteristics constrained by human pronunciation resonance.

[0047] T represents the number of speech frames, and the static characteristics of the phonemes in the nth frame are distinguished features. Where n=1,2,…T,F n,0 Indicates the fundamental frequency of the nth frame, Fn,1 ,F n,2 ,F n,3 represents the human voice constraint formant of the nth frame, W n,1 ,W n,2 ,W n,3 Indicates the bandwidth corresponding to the human voice constraint resonance peak of the nth frame, Represents the 13-dimensional MFCC of the nth frame. When the speech is nasal or unvoiced, the zero-pole frequency and its bandwidth are used to compensate for the first three formant frequencies and bandwidths of non-nasal voiced sounds. Static characteristics of speech segment phonemes distinguish features Distinguishing features of dynamic characteristics of speech segment phonemes Speech segment phoneme distinguishing features

[0048] Step 4: Combine multiple language complete phoneme sets

[0049] Step 4.1: Prime set reduction

[0050] The phoneme set is reduced for the mappable IPA phoneme clusters to reduce the size of the phoneme set. Before reduction, the language phoneme symbol set C i , i=0,1,2 The phoneme symbol set C′ obtained by mapping i , and the IPA cluster C′ can be mapped i,IPA , i=0,1,2 constitutes a random variable X i,IPA , i=1,2, and the probability space is as follows:

[0051]

[0052] p(c′ i,IPA (m i,IPA )) represents the random variable X i,IPA The phoneme symbol c′ i,IPA (m i,IPA ) occurs, m i,IPA Denotes a random variable X i,IPA ,i=0,1,2 belongs to the symbol set C′ i,IPA ,i=0,1,2 the number of phonemes.

[0053] The phoneme iterative reduction is done in a data-driven way. After e times of iterative reduction, the random variable X i,IPA , i=0,1,2, the number of phonemes in the symbol set is reduced by e. After the phoneme set is reduced, the random variables and their probabilities of the probability space expression are updated.

[0054] Find the phoneme distinguishing feature F of phoneme o in language i, i=0,1,2 (i,o) .

[0055] At this time, if the low error rate of phoneme recognition is sought from the perspective of phoneme characterization, cosine similar phoneme pairs are iterated to find the cosine distance of different phonemes. When the cosine distance between two phoneme features is the smallest, it indicates that the similarity between the two phonemes is high. The two phoneme probabilities are merged and the phoneme with a smaller probability is reduced.

[0056] At this time, if the maximum global recognition accuracy is sought in the application of sequence feature statistics in phoneme recognition, a small probability phoneme pair iteration is used to define a similarity and calculate the probability of two phonemes appearing in the driving data set. When the probabilities of two phonemes in the same language phoneme driving set are both small and similar, and when the two phonemes appear in different language phoneme driving sets at the same time, the probabilities of the two phonemes are merged and the phoneme with the smaller probability is reduced.

[0057] Repeat the above iterative process e times, and update the reduced random variable probability space each time.

[0058] Step 4.2: Phoneme set fusion

[0059] From the perspective of different learners, from phonemes to phrases to sentences to paragraphs, the richness of phonemes and the correlation between phonemes increase, the speaker's attention to phonemes weakens, and a universal transfer characteristic is produced between phonemes of different languages, which is reflected in daily life as foreigners speaking native languages; comparing and analyzing the types of differences between languages, the greater the difference, the higher the degree of first language transfer, and the smaller the mutual information of the speech phoneme set. The vocal tract model parameters of the speech sounding principle can effectively classify phonemes, and different language phoneme sets are responses to similar vocal tract model parameters, with high overlap and compatibility. This paper fuses the Chinese, English, and Spanish phoneme sets. To avoid vocabulary confusion, multi-level error-free fusion of phoneme sets based on the criterion of maximum mutual information is achieved, and the fusion model, such as Figure 3 shown.

[0060] Based on the updated phoneme set obtained by phoneme set reduction, during fusion, phonemes in a single language phoneme set are not merged, and phonemes in non-IPA phoneme clusters with different language labels are not merged with each other.

[0061] The fusion steps are as follows:

[0062] Step 4.2.1: Find the random variables X of the phoneme set after iterative reduction i ,i=1,2 The amount of mutual information with X0

[0063] Step 4.2.2: Find the number of iterations E when the mutual information between the two language phoneme symbol sets is the largest i ;

[0064] Step 4.2.3: Find the minimum E based on the maximum mutual information among all L languages i,i=2,…,L value, this value is used as the optimal number of iteration reductions for all phoneme sets.

[0065] Step 4.2.4: The iteratively reduced phoneme symbol sets under the optimal number of iterative reductions are merged to determine the fused multilingual complete phoneme symbol set under the fusion random variable of the i-language phoneme symbol set.

[0066] Step 5: Construct CTC supervised learning:

[0067] Step 5.1: Construct CTC supervised learning phoneme feature labels;

[0068] All the acquired TIMIT, TSC, and Thchs30 original corpora are encoded based on the fused multilingual complete phoneme symbol set, and unified language training labels for all languages ​​are constructed under the fused multilingual complete phoneme symbol set.

[0069] The speech training label is a label sequence at the language segment level, and the phoneme distinguishing feature F of the speech label sequence corresponding to the speech segment is further calculated.

[0070] Step 5.2: Build a CTC supervised learning network

[0071] Connectionist temporal classification (CTC) relies on dynamic programming to define the maximum word-missing rate supervised learning, realizes the alignment between speech expression and phonemes at the speech frame level, and maximizes the posterior probability of the speech output phoneme sequence obtained by RNN relative to the input label through a greedy algorithm.

[0072] The constructed connection time classification network, such as Figure 4 shown.

[0073] The supervised learning of the CTC network first obtains the phoneme-distinguishing features at the speech segment level corresponding to the speech time series, and then inputs the features into the CTC training network.

[0074] The phoneme distinguishing feature F is a feature vector with time step information. The long short-term memory time network composed of RNN calculates the feature vector F n The phoneme label posterior probability vector h of the context feature n ,h n Contains F n The posterior probabilities of all observed values ​​with respect to the labels.

[0075] When training the model, the loss function is calculated by h n Maximize the probability of correctly outputting phonemes and update to obtain an optimal coefficient matrix a n,t , represents the phoneme posterior probability vector h at time t n The (connected context) phoneme classification score coefficient is used to predict the optimal solution based on the coefficient matrix.

[0076] Phoneme label observation vector h n After greedy decoding of the phoneme labels, the most likely label sequence for F is g n , when aligning the phoneme sequence of the speech segment, the phoneme vector g corresponding to the speech feature is n and its context (…g n-1 ,g n-2 ,g n ,g n+1 ,g n+2 ...) is assigned to state S n , the space character distinguishes the pause blank label in the speech and the continuous repeated phonemes in the speech, and combines the current state and the previous state S n-1 Corresponding phoneme symbol c n-1 Determine the current state S n Corresponding phoneme symbol c n .

[0077] CTC learns from speech segments to obtain the optimal distribution of all possible phoneme sequence outputs, obtains an accurate mapping between speech segments and phoneme sequences, inputs frame-level phoneme-distinguishing features into the model, and outputs the phoneme sequence corresponding to the entire speech segment, thereby constructing a phoneme recognition model that can recognize multiple languages.

[0078] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A multilingual phoneme recognition method based on iterative fusion of phoneme pairs, characterized by: Step 1: Obtain multiple foreign language phoneme corpora of different resource levels, directly obtain the corpora with higher resource levels as the main language phoneme set for training the first language, indirectly obtain the corpora by non-IPA phoneme-IPA phoneme dictionary encoding as the expanded language phoneme set for training the second or even more languages, and indirectly obtain the corpora by grapheme-IPA phoneme dictionary encoding as the expanded language phoneme set for training the third or even more languages; Step 2: Based on the phoneme set obtained in Step 1, the phoneme labels of the corpus resources obtained by the non-IPA phoneme-IPA phoneme dictionary encoding method and the phoneme labels of the corpus obtained by the grapheme-IPA phoneme dictionary encoding method are uniformly mapped to the phoneme-phoneme label representation of the main phoneme set of the first language based on linguistic knowledge; Step 3: Construct new features to distinguish phonemes based on the human vocal tract sounding constraint. Specifically, according to the mechanism of speech sounding system, based on the multilingual phoneme set mapped with the first language IPA phoneme label obtained in Step 2, all phonemes are refined into voiceless phonemes, nasal voiced phonemes, and non-nasal voiced phonemes. According to the speech sounding characteristics of different speech categories, new features with better phoneme distinguishing characteristics are constructed. Step 4: Based on the multilingual phoneme set obtained in Step 2, reduce the mapped IPA phoneme symbol set in the language, and reduce the size of the overall phoneme symbol set of the mapped main language, second language and third language phoneme sets respectively; Step 5: Based on the reduced multilingual phoneme set obtained in Step 4, the main phoneme set directly obtained is used as the initial set, and combined with the indirectly obtained second language phoneme set for the first iterative first-level fusion. The new set formed after the fusion is then combined with the indirectly obtained third language phoneme set for the second iterative second-level fusion, and so on, to obtain the fusion of more levels of multilingual phoneme sets in more languages; Step 6: Use the connection time classification network (CTC) to build a phoneme recognition network to realize the recognition of phoneme sequences with variable length and automatic alignment of multilingual speech.

2. The multilingual phoneme recognition method based on iterative fusion of phoneme pairs according to claim 1, characterized in that: The Step 3 is specifically as follows: constructing a discriminative feature quantity to characterize a single phoneme, dividing the phonemes into three categories: unvoiced, nasal voiced, and non-nasal voiced, combining the acoustic feature MFCC and the phoneme high-order linear prediction peak frequency band feature constrained by human pronunciation resonance to construct a new phoneme discriminative feature, T represents the number of speech frames, and the static characteristic discriminative feature of the phoneme in the nth frame Where n=1,2,…T,F n,0 Indicates the fundamental frequency of the nth frame, F n,1 ,F n,2 ,F n,3 represents the human voice constraint formant of the nth frame, W n,1 ,W n,2 ,W n,3 Indicates the bandwidth corresponding to the human voice constraint resonance peak of the nth frame, Represents the 13-dimensional MFCC of the nth frame. When the speech is nasal or unvoiced, the zero-pole frequency and its bandwidth are used to compensate for the first three formant frequencies and bandwidths of non-nasal voiced sounds. The static characteristics of the speech segment phonemes distinguish the features. Distinguishing features of dynamic characteristics of speech segment phonemes Speech segment phoneme distinguishing features 3. The multilingual phoneme recognition method based on iterative fusion of phoneme pairs according to claim 1, characterized in that: The Step 4 is specifically as follows: define a similarity, find the probability of two phonemes appearing in the phoneme set, when the probability of two phonemes appearing in the same language phoneme driving set is small and similar, when the two phonemes appear in the phoneme driving sets of different languages ​​at the same time, merge the two phoneme probabilities and reduce the phoneme with smaller probability; for language phoneme lattice grapheme encoding, sacrifice the accuracy of phoneme language characterization, find the phoneme distinguishing feature F of phoneme o in language i, i=0,1,2 (i,o) , based on the new phoneme distinguishing feature F (i,o) The cosine distance between the phonemes represents the phoneme similarity, and the phoneme pairs with the greatest similarity are merged and reduced. When reducing, the two phonemes are merged and represented by the phoneme symbol with a higher probability of appearing in the training set.

4. The multilingual phoneme recognition method based on iterative fusion of phoneme pairs according to claim 1, characterized in that: The specific step 5 is: first convert the language phoneme set and the language phoneme symbol set constituting the language phoneme set into random variables, and when performing the fusion at the first level, find the expression of the mutual information amount of the two language phoneme sets fused at the same level under different iteration reduction times, and perform the multi-level fusion of multiple phoneme sets with the phoneme reduction iteration number when the expression is the largest. The specific steps are: Step 5.1: Find the random variables X of the phoneme set after iterative reduction i ,i=1,2 The amount of mutual information with X0 Step 5.2: Find the number of iterations E when the mutual information between the two language phoneme symbol sets is the largest i ; Step 5.3: Find the minimum E based on the maximum mutual information among all L languages i ,i=2,…,L value, take this value as the optimal number of iteration reductions for all phoneme sets; Step 5.4: The iteratively reduced phoneme symbol sets under the optimal number of iterative reductions are merged to determine the fused multilingual complete phoneme symbol set under the fusion random variable of the i-language phoneme symbol set.

Images