A multilingual training method and device and a storage medium

Abstract

The application discloses a kind of multilingual training method, device and storage medium, comprising: after obtaining phonological features by neural network, it is converted into with vector coding phoneme embedding vector, wherein, phonological features are coded as phonological vector;After obtaining acoustic spectrum, extract acoustic features by acoustic model DNN;After the inner product of phoneme embedding vector and acoustic features, the posterior probability of phoneme is calculated out.The present application avoids the trouble of training phonological feature extractor in the prior art, so that cross-language zero-shot learning becomes possible, and also has good transfer effect for never seen phoneme.

Description

Technical Field

[0001] This invention relates to the field of language recognition technology, and in particular to a multilingual training method, apparatus, and storage medium. Background Technology

[0002] Training a speech recognition model based on DNN (Deep Neural Network) requires a large amount of data. However, for the more than 7,100 languages ​​in the world, only a small number of languages ​​have enough labeled training data available, making the recognition of other low-resource languages ​​a challenge.

[0003] To improve the recognition performance of low-resource languages ​​when training data is insufficient, recent works have introduced multilingual and crosslingual methods. By using other languages ​​for training and leveraging the similar features between languages ​​to aid in the learning of the target language, the problem of insufficient target language data can be effectively alleviated.

[0004] Multilingual training generally refers to the acoustic model (AM) being trained jointly by multiple languages. This is because different languages ​​may share similar pronunciation features, while the language model (LM) is more language-independent, as different languages ​​have significantly different grammatical rules, making it unsuitable for multilingual training. Some end-to-end implementations combine AM and LM to train multilingual models, but this requires a much larger amount of training data. Bottleneck features were an early implementation of multilingual methods, using the output of the bottleneck layer of a multilingual DNN as input to the AM to train the acoustic model. The method of sharing hidden layers is also widely used in multilingual implementations. Different languages ​​share a single hidden layer while having different output layers, allowing the hidden layer to learn common features between languages. These studies have all confirmed that multilingual models improve target language recognition performance.

[0005] In crosslingual speech recognition, the target language is not included in the training set. A multilingual model is trained using other languages ​​and then fine-tuned to transfer the model to the target language. In zero-shot scenarios, the multilingual model is tested directly on the target language without any fine-tuning. Few-shot scenarios involve fine-tuning the multilingual model using a small amount of target language data. When target language data is extremely scarce, crosslingual methods can be considered, utilizing information obtained from other languages ​​to aid in target language recognition.

[0006] To achieve multilingual training, it's necessary for different languages ​​to share information. The most common approach is to use a unified phoneme set, encompassing phonemes from all languages. The International Phonetic Alphabet (IPA), based on pronunciation methods, is the optimal choice for this unified phoneme set. Phonemes with similar pronunciations in different languages ​​are labeled with the same IPA, thus achieving information sharing at the phoneme level. The more information shared between languages, the greater the improvement in multilingual recognition. Experiments by Feng et al. show that in multilingual training, the more languages ​​a phoneme is shared, the greater its benefit to multilinguality, likely because more data about that phoneme is available for training the multilingual model. Conversely, phonemes unique to certain languages ​​show almost no improvement in multilinguality. To allow these language-specific phonemes to also benefit from multilingual models, some works have broken down phonemes into smaller units, namely phonological features. Phonological features encompass all the phonological characteristics of a phoneme, such as place of articulation and manner of articulation. Each feature of a phoneme is marked with three symbols: "+", "-", and "0". "+" indicates that the feature is present, "-" indicates that the feature is not present, and "0" indicates that the phoneme is irrelevant to the feature. For example, a feature related to a vowel is meaningless to a consonant phoneme. Figure 1 The diagram illustrates the overlap of phoneme sets between Spanish and Italian, as shown in the figure. Figure 1 This demonstrates the benefits of using phonological features to represent phonemes. Spanish and Italian share some overlapping phonemes, which can obtain more training data in multilingual training, thereby improving the model's ability to recognize them. For their individual phonemes, such as those in Spanish... The Italian ε, however, cannot benefit from phoneme-based multilingual training. But if phonemes are represented as phonological features, we can see... They share many features with ε, and at the level of phonological features they share information, which helps the model improve its ability to recognize independent phonemes.

[0007] Some studies have utilized phonological features to aid in multilingual and cross-lingual speech recognition. However, a shortcoming of current techniques is that the methods for extracting phonological features from acoustic features make training the extractor very difficult.

[0008] Furthermore, there is another drawback: it cannot handle unseen phonemes in cross-language recognition. Summary of the Invention

[0009] This invention provides a multilingual training method, apparatus, and storage medium to solve the problem that training the extractor is very difficult when extracting phonological features from acoustic features.

[0010] This invention provides the following technical solutions:

[0011] A multilingual training method, comprising:

[0012] After obtaining phonological features through a neural network, they are converted into phoneme embedding vectors encoded by vectors, where the phonological features are encoded as phonological vectors;

[0013] After obtaining the acoustic spectrum, acoustic features are extracted using an acoustic model DNN;

[0014] The posterior probability of a phoneme is calculated by performing an inner product between the phoneme embedding vector and the acoustic feature.

[0015] In implementation, phonological features are encoded into phonological vectors, including:

[0016] Each IPA phoneme in each language is represented as a 24-dimensional phonological feature, with each dimension representing a specific phonological feature. Each phonological feature's +, -, and 0 are represented by a 2-bit 1 / 0 vector, and the sign of each phonological feature is represented by a 3-bit vector. <blk> 、 <spn> 、 <nsn>coding.

[0017] In implementation, the posterior probability of the phoneme is calculated by performing an inner product between the phoneme embedding vector and the acoustic feature, including:

[0018] The phonological vector p of the i-th phoneme i Convert to phoneme embedding vector e i The dimension is equal to the dimension H of the DNN output. In frame t, the output of the acoustic model DNN is...

[0019] e i Acoustic characteristics h t Calculate the inner product logits, z t,i This represents the logit corresponding to the i-th phoneme in frame t:

[0020]

[0021] In practice, the phonological vector p of the i-th phoneme is converted as follows: i Convert to phoneme embedding vector e i :

[0022] Use a linear matrix A to act on p i ,

[0023] e i =Ap i #

[0024] ;or,

[0025] Phonometry using a multi-layer neural network:

[0026] e i =A2σ(A1p) i )#

[0027] Where A1 and A2 are linear matrices, and σ(·) represents a nonlinear activation function.

[0028] During implementation, it further includes:

[0029] The Phonetisaurus toolkit is used to implement G2P (graph-to-phoneme conversion) to generate pronunciation dictionaries of IPA symbols for various languages.

[0030] During implementation, it further includes:

[0031] The training and development sets of multiple languages ​​are mixed and shuffled to form a training and development set that includes multiple languages;

[0032] The pronunciation dictionaries generated by G2P in multiple languages ​​are merged and used as the dictionary during multilingual training.

[0033] In implementation, the posterior probability of the phoneme is calculated by performing an inner product between the phoneme embedding vector and the acoustic feature, including:

[0034] Statistically analyze the IPA symbols appearing in the training set and establish corresponding phonological vectors for each IPA symbol.

[0035] The phoneme phonological vectors are concatenated into a phonological vector matrix and then input together. The phone embedding vector of each phoneme is then calculated.

[0036] During training, the output of the DNN is multiplied by the phone embedding vector converted from the phonology vector to calculate the inner product logit of each phoneme.

[0037] During implementation, it further includes:

[0038] Error backpropagation during training is used to update the parameters of the DNN and neural network.

[0039] In practice, this further includes setting the parameters during spectrogram acquisition in one or a combination of the following ways:

[0040] The training process used a speech recognition toolkit based on CTC-CRF.

[0041] For each audio data point, extract 40-dimensional FBank features and perform second-order differencing, which are then used as input to AM;

[0042] The acoustic model uses a 3-layer VGGBLSTM, with 1024 hidden units per layer of BLSTM;

[0043] During training, the random dropout factor was set to 0.5, and the Adam optimizer was used for optimization with an initial learning rate of 1e-3.

[0044] When the model’s performance on the development set declines, the learning rate is reduced to 1 / 10 of its previous value, and training is stopped when the learning rate is less than 1e-5.

[0045] In practice, after training many language models, further steps include:

[0046] The target language is identified.

[0047] A multilingual training device, comprising:

[0048] The processor is used to read programs from memory and execute the following procedures:

[0049] After obtaining phonological features through a neural network, they are converted into phoneme embedding vectors encoded by vectors, where the phonological features are encoded as phonological vectors;

[0050] After obtaining the acoustic spectrum, acoustic features are extracted using an acoustic model DNN;

[0051] The posterior probability of a phoneme is calculated by taking the inner product of the phoneme embedding vector and the acoustic feature.

[0052] A transceiver is used to receive and send data under the control of a processor.

[0053] In implementation, phonological features are encoded into phonological vectors, including:

[0054] Each IPA phoneme in each language is represented as a 24-dimensional phonological feature, with each dimension representing a specific phonological feature. Each phonological feature's +, -, and 0 are represented by a 2-bit 1 / 0 vector, and the sign of each phonological feature is represented by a 3-bit vector. <blk> 、 <spn> 、 <nsn>coding.

[0055] In implementation, the posterior probability of the phoneme is calculated by performing an inner product between the phoneme embedding vector and the acoustic feature, including:

[0056] The phonological vector p of the i-th phoneme i Convert to phoneme embedding vector e i The dimension is equal to the dimension H of the DNN output. In frame t, the output of the acoustic model DNN is...

[0057] e i Acoustic characteristics h t Calculate the inner product logits, z t,i This represents the logit corresponding to the i-th phoneme in frame t:

[0058]

[0059] In practice, the phonological vector p of the i-th phoneme is converted as follows: i Convert to phoneme embedding vector e i :

[0060] Use a linear matrix A to act on p i ,

[0061] e i =Ap i #

[0062] ;or,

[0063] Phonometry using a multi-layer neural network:

[0064] e i =A2σ(A1p) i )#

[0065] Where A1 and A2 are linear matrices, and σ(·) represents a nonlinear activation function.

[0066] During implementation, it further includes:

[0067] The Phonetisaurus toolkit is used to implement G2P (graph-to-phoneme conversion) to generate pronunciation dictionaries of IPA symbols for various languages.

[0068] During implementation, it further includes:

[0069] The training and development sets of multiple languages ​​are mixed and shuffled to form a training and development set that includes multiple languages;

[0070] The pronunciation dictionaries generated by G2P in multiple languages ​​are merged and used as the dictionary during multilingual training.

[0071] In implementation, the posterior probability of the phoneme is calculated by performing an inner product between the phoneme embedding vector and the acoustic feature, including:

[0072] Statistically analyze the IPA symbols appearing in the training set and establish corresponding phonological vectors for each IPA symbol.

[0073] The phoneme phonological vectors are concatenated into a phonological vector matrix and then input together. The phone embedding vector of each phoneme is then calculated.

[0074] During training, the output of the DNN is multiplied by the phone embedding vector converted from the phonology vector to calculate the inner product logit of each phoneme.

[0075] During implementation, it further includes:

[0076] Error backpropagation during training is used to update the parameters of the DNN and neural network.

[0077] In practice, this further includes setting the parameters during spectrogram acquisition in one or a combination of the following ways:

[0078] The training process used a speech recognition toolkit based on CTC-CRF.

[0079] For each audio data point, extract 40-dimensional FBank features and perform second-order differencing, which are then used as input to AM;

[0080] The acoustic model uses a 3-layer VGGBLSTM, with 1024 hidden units per layer of BLSTM;

[0081] During training, the random dropout factor was set to 0.5, and the Adam optimizer was used for optimization with an initial learning rate of 1e-3.

[0082] When the model’s performance on the development set declines, the learning rate is reduced to 1 / 10 of its previous value, and training is stopped when the learning rate is less than 1e-5.

[0083] In practice, after training many language models, further steps include:

[0084] The target language is identified.

[0085] A multilingual training device, comprising:

[0086] The phonology module is used to obtain phonological features through a neural network and then convert them into phoneme embedding vectors encoded by vectors, where the phonological features are encoded into phonological vectors.

[0087] The acoustic module is used to extract acoustic features through an acoustic model DNN after acquiring the acoustic spectrum.

[0088] The inner product module is used to calculate the posterior probability of a phoneme by performing an inner product between the phoneme embedding vector and the acoustic feature.

[0089] In implementation, the phonology module is further used to encode phonological features into phonological vectors, including:

[0090] Each IPA phoneme in each language is represented as a 24-dimensional phonological feature, with each dimension representing a specific phonological feature. Each phonological feature's +, -, and 0 are represented by a 2-bit 1 / 0 vector, and the sign of each phonological feature is represented by a 3-bit vector. <blk> 、 <spn> 、 <nsn>coding.

[0091] In implementation, the inner product module is further used to calculate the posterior probability of a phoneme after performing an inner product between the phoneme embedding vector and the acoustic features, including:

[0092] The phonological vector p of the i-th phoneme i Convert to phoneme embedding vector e i The dimension is equal to the dimension H of the DNN output. In frame t, the output of the acoustic model DNN is...

[0093] e i Acoustic characteristics h t Calculate the inner product logits, z t,i This represents the logit corresponding to the i-th phoneme in frame t:

[0094]

[0095] In implementation, the phonology module is further used to convert the phonology vector p of the i-th phoneme in the following manner. i Convert to phoneme embedding vector e i :

[0096] Use a linear matrix A to act on p i ,

[0097] e i =Ap i #

[0098] ;or,

[0099] Phonometry using a multi-layer neural network:

[0100] e i =A2σ(A1p) i )#

[0101] Where A1 and A2 are linear matrices, and σ(·) represents a nonlinear activation function.

[0102] In practice, the phonology module is further used to implement glyph-phoneme conversion (G2P) using the Phonetisaurus toolkit, generating pronunciation dictionaries of IPA symbols for various languages.

[0103] In practice, the phonology module is further used to mix and shuffle training and development sets of multiple languages ​​to form training and development sets that include multiple languages.

[0104] The pronunciation dictionaries generated by G2P in multiple languages ​​are merged and used as the dictionary during multilingual training.

[0105] In implementation, the inner product module is further used to calculate the posterior probability of a phoneme after performing an inner product between the phoneme embedding vector and the acoustic features, including:

[0106] Statistically analyze the IPA symbols appearing in the training set and establish corresponding phonological vectors for each IPA symbol.

[0107] The phoneme phonological vectors are concatenated into a phonological vector matrix and then input together. The phone embedding vector of each phoneme is then calculated.

[0108] During training, the output of the DNN is multiplied by the phone embedding vector converted from the phonology vector to calculate the inner product logit of each phoneme.

[0109] During implementation, it further includes:

[0110] The update module is used for backpropagation of errors during training to update the parameters of the DNN and neural network.

[0111] During implementation, it further includes:

[0112] The settings module is used to configure the acquisition of the sound spectrum in one or a combination of the following ways:

[0113] The training process used a speech recognition toolkit based on CTC-CRF.

[0114] For each audio data point, extract 40-dimensional FBank features and perform second-order differencing, which are then used as input to AM;

[0115] The acoustic model uses a 3-layer VGGBLSTM, with 1024 hidden units per layer of BLSTM;

[0116] During training, the random dropout factor was set to 0.5, and the Adam optimizer was used for optimization with an initial learning rate of 1e-3.

[0117] When the model’s performance on the development set declines, the learning rate is reduced to 1 / 10 of its previous value, and training is stopped when the learning rate is less than 1e-5.

[0118] During implementation, it further includes:

[0119] The recognition module is used to recognize the target language after training many language models.

[0120] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multilingual training method.

[0121] The beneficial effects of this invention are as follows:

[0122] The technical solution provided in this invention introduces phonological features. A top-down phonological transformation is used to obtain phoneme embedding vectors, and a bottom-up DNN is used to obtain acoustic features output by the acoustic model. The two are combined to calculate the posterior probability of the phoneme. By introducing phonological knowledge, different phonemes can be more closely linked at the levels of articulation place and manner, avoiding the difficulties of training phonological feature extractors in existing technologies.

[0123] Furthermore, since phonological features are not extracted from acoustic features, but are calculated and extracted separately using two-part models, and the neural network for phoneme embedding vectors learns the processing and transformation of phonological features, even for unseen phones, as long as their phonological features are written out, the network can work normally. This makes cross-language zero-shot learning possible, and it can also achieve good transfer effects for phonemes that have never been seen before. Attached Figure Description

[0124] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0125] Figure 1 This is a schematic diagram illustrating the overlap of phoneme sets between Spanish and Italian in the background technology.

[0126] Figure 2 This is a schematic diagram of the traditional "bottom-up" phonological feature method in an embodiment of the present invention;

[0127] Figure 3 This is a schematic diagram illustrating the implementation process of the multilingual training method in this embodiment of the invention;

[0128] Figure 4 This is a schematic diagram illustrating the implementation process of the JoinAP method in this embodiment of the invention;

[0129] Figure 5 This is a schematic diagram of the multilingual training device in an embodiment of the present invention. Detailed Implementation

[0130] The inventor noticed the following during the invention process:

[0131] Several studies have utilized phonological features to aid in multilingual and cross-lingual speech recognition, and these studies often employ a bottom-up approach. Figure 2 This is a schematic diagram of the traditional "bottom-up" phonological feature method, such as... Figure 2 As shown, it includes: Acoustic spectra acquisition, Phonological feature extractor processing, Voicing and high input, Phonological feature posteriors, Feature concatenation or Model combination, Standard acoustic model processing, and Phone probabilities.

[0132] In this model, a phonological feature extractor is trained, which outputs the posterior probability of each phonological feature. These probabilities are then concatenated and fed into the subsequent acoustic model to obtain the posterior probability of each phoneme. This method has some drawbacks. First, this bottom-up "acoustics to phonology" extraction is complex; training a phonological feature extractor requires subdividing the sentence into phoneme levels and labeling each phoneme individually. Second, this model cannot handle unseen phonemes in crosslingual zero-shot scenarios, as the parameters related to these unseen phonemes in the output layer are directly initialized.

[0133] Many current multilingual speech recognition studies are based on a common phoneme set. Only when a certain phoneme appears in different languages ​​can it be helpful in multilingual training. Phonemes that are unique to a particular language are unlikely to benefit from more training data in multilingual training.

[0134] A few works have also used phonological features to further break down phonemes and share information at the phonological feature level. However, the methods used are all "bottom-up" approaches to extract phonological features from acoustic features. Such extractors are difficult to train and have a significant drawback: they cannot handle unseen phonemes in cross-language recognition.

[0135] Based on this, this embodiment of the invention provides a multilingual and cross-lingual speech recognition scheme that combines acoustics and phonology. This is a multilingual model training scheme that integrates phonological features and acoustic features to improve the performance of multilingual and cross-lingual speech recognition.

[0136] The specific embodiments of the present invention will now be described with reference to the accompanying drawings.

[0137] Figure 3 The flowchart for implementing a multilingual training method is shown in the figure, and may include:

[0138] Step 301: After obtaining the phonological features through the neural network, they are converted into phoneme embedding vectors encoded by vectors, wherein the phonological features are encoded into phonological vectors;

[0139] Step 302: After obtaining the acoustic spectrum, extract acoustic features using the acoustic model DNN;

[0140] Step 303: Calculate the posterior probability of the phoneme by performing an inner product between the phoneme embedding vector and the acoustic feature.

[0141] The solution mainly combines bottom-up acoustic feature extraction with top-down phonological feature transformation to train multilingual models and help achieve better results in multilingual and cross-lingual recognition.

[0142] Specifically, this is a multilingual training scheme that transforms phonological features into phoneme embeddings from a top-down perspective, and combines them with acoustic features extracted from a bottom-up perspective to jointly calculate the posterior probabilities of phonemes. In this embodiment, this scheme will be referred to as JoinAP (Joining of Acoustics and Phonology).

[0143] Phonological features (or distinguishing features) are an important concept in phonetics, used to differentiate different phonemes. A phoneme can be viewed as a series of phonological features. Using phonological features to train multilingual models can, for example... Figure 1 As shown, this allows independent phonemes in different languages ​​to maintain characteristic connections, thereby facilitating information sharing.

[0144] In implementation, phonological features are encoded into phonological vectors, including:

[0145] Each IPA phoneme in each language is represented as a 24-dimensional phonological feature, with each dimension representing a specific phonological feature. Each phonological feature's +, -, and 0 are represented by a 2-bit 1 / 0 vector, and the sign of each phonological feature is represented by a 3-bit vector. <blk> 、 <spn> 、 <nsn>coding.

[0146] Specifically, using the phonological feature representation of IPA phonemes in the Panphon toolkit, each IPA phoneme of each language is represented as a 24-dimensional phonological feature, with each dimension representing a specific phonological feature. Each phonological feature has three representations: "+", "-", and "0". To enable the neural network to better represent and process phonological features, the phonological features are encoded as phonological vectors. Specifically, each phonological feature can be represented by a 2-bit vector of 1s and 0s. Taking the "round" feature as an example, the first bit indicates "round+", and the second indicates "round-". Thus, "+" is encoded as "10" in the phonological vector, "-" is "01", and "0" is "00".

[0147] Additionally, when training an acoustic model using CTC (Connectionist Temporal Classification)-CRF (Conditional Random Fields), there are three extra special output notations: <blk>(null), <spn>(noise from talking) <nsn>(Natural noise) was considered, so an additional 3 bits were added, and these three symbols were encoded using one-hot encoding. Ultimately, a 51-bit phonological vector was obtained, representing each IPA phoneme as a 1 / 0 vector. Wherein:

[0148] CRF: Typically used for sequence labeling tasks, such as: Bi LSTM In +CRF and IDCNN+CRF scenarios, the characteristic is that the input and output are in one-to-one correspondence. The semantic model first generates a "score" (posterior probability minus log) for each character based on the input, which is used as the backward observation probability during decoding.

[0149] CTC decoding: a speech recognition method that takes speech as input and outputs text. Its characteristic is that one output may correspond to multiple correct paths.

[0150] In implementation, the posterior probability of the phoneme is calculated by performing an inner product between the phoneme embedding vector and the acoustic feature, including:

[0151] The phonological vector p of the i-th phoneme i Convert to phoneme embedding vector e i The dimension is equal to the dimension H of the DNN output. In frame t, the output of the acoustic model DNN is...

[0152] e i Acoustic characteristics h t Calculate the inner product logits, z t,i This represents the logit corresponding to the i-th phoneme in frame t:

[0153]

[0154] Specifically, without introducing phonological vectors, the method for calculating the posterior probability of phonemes is as follows: In frame t, the output of the acoustic model DNN (Dynamic Neural Network) is... If the DNN is an LSTM (Long Short-Term Memory) with a hidden layer size of 1024, then H equals 1024. Then h... t By using a fully connected linear layer, we obtain the logits (inner product) with a dimension equal to the number of phoneme tokens. After passing through a softmax layer, we obtain the posterior probability of each phoneme.

[0155] The goal is to combine the phonological vectors and acoustic features extracted by the DNN during training to jointly calculate the posterior probability of a phoneme. First, the phonological vector p of the i-th phoneme... i Convert to embedded vector e i This is called phone embedding, and its dimension is equal to the dimension H of the DNN output. Then, e i with h t Calculate the inner product logits, z t,i This represents the logit corresponding to the i-th phoneme in frame t:

[0156]

[0157] In practice, the phonological vector p of the i-th phoneme is converted as follows: i Convert to phoneme embedding vector e i :

[0158] Use a linear matrix A to act on p i ,

[0159] e i =Ap i #

[0160] ;or,

[0161] Phonometry using a multi-layer neural network:

[0162] e i =A2σ(A1p) i )#

[0163] Where A 1、 A2 is a linear matrix, and σ(·) represents a nonlinear activation function.

[0164] Specifically, there are two ways to convert phonological vectors into phone embeddings:

[0165] (1) Linear JoinAP, directly using the linear matrix A to act on p i ,

[0166] e i =Ap i #(2)

[0167] (2) Nonlinear JoinAP, similar to linear JoinAP, except that nonlinearity is inserted into the calculation, and a multi-layer neural network is used for phonological transformation:

[0168] e i =A2σ(A1p) i )#(3)

[0169] Where A1A2 is a linear matrix, and σ(·) represents a nonlinear activation function (such as sigmoid). Introducing nonlinearity can improve the model's ability to represent deep features. e is transformed using two methods. i Then, logits are calculated according to formula (1).

[0170] Figure 4 The following is a schematic diagram of the implementation process of the JoinAP method, as shown below. Figure 4 As shown, it includes:

[0171] Phone input, Phoneological transformation, Phone embedding;

[0172] Acoustic spectra input, DNN-based feature extractor extraction, DNN output;

[0173] Phoneme embedding and acoustic feature Logits (inner product).

[0174] Specifically, the phoneme embedding vector e is obtained through top-down phonological transformation. i The acoustic model output h is obtained through a bottom-up DNN. t The two are combined to calculate phoneme logits. Compared to the traditional "bottom-up" approach, JoinAP does not extract phonological features from acoustic features, but instead uses two separate models to calculate and extract them. Moreover, the neural network of Phonological Transformation learns the processing and transformation of phonological features. Even with unseen phones, as long as their phonological features are written down, the network can work normally. This makes cross-language zero-shot learning possible, and it can also have good transfer effects for phonemes that have never been seen before.

[0175] The following describes the specific implementation methods for data preparation and model training.

[0176] (I) Data Preprocessing

[0177] 1. G2P generates IPA dictionaries.

[0178] In practice, it may further include:

[0179] The Phonetisaurus toolkit is used to implement G2P (graph-to-phoneme conversion) to generate pronunciation dictionaries of IPA symbols for various languages.

[0180] Specifically, to unify phoneme representation across multiple languages, a pronunciation dictionary of IPA symbols can be generated for each language. This is achieved using the Phonetisaurus toolkit to implement G2P (Grapheme-to-Phoneme), which automatically generates the phoneme sequence based on the word's word combination, resulting in the commonly known pronunciation dictionary. Phonetisaurus is based on WFST (Weighted Finite-State Transducer), using N-grams to model phonemes and construct a WFST graph. During decoding, it uses RNNLM (Recurrent Neural Network) for rescoping, employs lattice minimization of Bayesian risk (LMBR) decoding, and utilizes the EM (Expectation-Maximization algorithm) to align words and phonemes. When using Phonetisaurus, a pre-trained G2P model by Mark Hasegawa-Johnson et al. can be stored in WFST format. In addition, using this toolkit to perform G2P conversion on all words that have appeared in the training set also effectively reduces the problems caused by out-of-set words (OOV).

[0181] 2. Preparation of multilingual datasets.

[0182] In practice, it may further include:

[0183] The training and development sets of multiple languages ​​are mixed and shuffled to form a training and development set that includes multiple languages;

[0184] The pronunciation dictionaries generated by G2P in multiple languages ​​are merged and used as the dictionary during multilingual training.

[0185] Specifically, to train a multilingual acoustic model, the training and development sets for all languages ​​can be mixed and shuffled to form multilingual training and development sets. Simultaneously, the pronunciation dictionaries generated by G2P for all languages ​​are merged and used as the dictionary for multilingual training. Before multilingual training, other data preparation operations are the same as for monolingual training, except that multilingual language models are not trained.

[0186] (ii) Multilingual model training.

[0187] 1. Training methods.

[0188] In implementation, the posterior probability of the phoneme is calculated by performing an inner product between the phoneme embedding vector and the acoustic feature, including:

[0189] Statistically analyze the IPA symbols appearing in the training set and establish corresponding phonological vectors for each IPA symbol.

[0190] The phoneme phonological vectors are concatenated into a phonological vector matrix and then input together. The phone embedding vector of each phoneme is then calculated.

[0191] During training, the output of the DNN is multiplied by the phone embedding vector converted from the phonology vector to calculate the inner product logit of each phoneme.

[0192] Specifically, a multilingual model is trained using a training set that incorporates multiple languages. All IPA symbols appearing in the training set are pre-collected, and corresponding phonological vectors are created for them. The phonological vectors are then... Figure 4 The top-down input concatenates the phonological vectors of all phonemes into a phonological vector matrix, which is then input together, while simultaneously calculating the phoneme embedding. The network structure during training is as follows: Figure 4 As shown, the output of the DNN does not pass through a fully connected layer, but is multiplied by the phone embedding converted from the phoneme vector, thereby calculating the logit of each phoneme.

[0193] In specific implementation, it may further include:

[0194] Error backpropagation during training is used to update the parameters of the DNN and neural network.

[0195] Specifically, during training, error backpropagation can simultaneously update the parameters of both the DNN and the phonology conversion network.

[0196] 2. Training parameter settings.

[0197] In practice, it may further include setting the parameters in one or a combination of the following ways when acquiring the sound spectrum:

[0198] The training process used a speech recognition toolkit based on CTC-CRF.

[0199] For each audio data point, extract 40-dimensional FBank features and perform second-order differencing, which are then used as input to AM;

[0200] The acoustic model uses a 3-layer VGGBLSTM, with 1024 hidden units per layer of BLSTM;

[0201] During training, the random dropout factor was set to 0.5, and the Adam optimizer was used for optimization with an initial learning rate of 1e-3.

[0202] When the model’s performance on the development set declines, the learning rate is reduced to 1 / 10 of its previous value, and training is stopped when the learning rate is less than 1e-5.

[0203] Specifically, during training, the Computer-Aided Translation (CAT) speech recognition toolkit based on CTC-CRF was used. 40-dimensional FBank features were extracted for each audio data point and second-order differencing was performed, serving as input to AM. The acoustic model employed a 3-layer VGGBLSTM (VGG: Visual Geometry Group) and BLSTM (Bidirectional LSTM) replication method, with 1024 hidden units per layer. To prevent overfitting during training, a random dropout factor of 0.5 was set, and the Adam optimizer was used for optimization, with an initial learning rate of 1e-3. When the model's performance on the development set deteriorated, the learning rate was reduced to 1 / 10 of its previous value, and training was stopped when the learning rate fell below 1e-5.

[0204] (III) Multilingual and cross-linguistic recognition.

[0205] In practice, after training many language models, further steps can be taken, including:

[0206] The target language is identified.

[0207] Specifically, after training the multilingual models, target language recognition tests can begin. Target language data is prepared using traditional methods, and the language models are trained to generate decoding maps. For the acoustic model, the trained multilingual models are fine-tuned using the target language training data, with an initial learning rate set to 1e-4. In cross-language recognition, zero-shot learning can be performed directly without target language fine-tuning, or fine-tuning can be done using a small amount of target language data; this is the few-shot scenario for cross-language recognition. During fine-tuning, the phonological vector matrix used in the multilingual training is simply replaced with the phonological vector matrix of the target language.

[0208] Based on the same inventive concept, this invention also provides a multilingual training device and a computer-readable storage medium. Since the principles by which these devices solve problems are similar to those of the multilingual training method, the implementation of these devices can refer to the implementation of the method, and repeated details will not be repeated.

[0209] When implementing the technical solutions provided in the embodiments of the present invention, they can be implemented in the following manner.

[0210] Figure 5 The diagram shows the structure of a multilingual training device. The device includes:

[0211] Processor 500 is used to read the program from memory 520 and execute the following procedures:

[0212] After obtaining phonological features through a neural network, they are converted into phoneme embedding vectors encoded by vectors, where the phonological features are encoded as phonological vectors;

[0213] After obtaining the acoustic spectrum, acoustic features are extracted using an acoustic model DNN;

[0214] The posterior probability of a phoneme is calculated by taking the inner product of the phoneme embedding vector and the acoustic feature.

[0215] Transceiver 510 is used to receive and send data under the control of processor 500.

[0216] In implementation, phonological features are encoded into phonological vectors, including:

[0217] Each IPA phoneme in each language is represented as a 24-dimensional phonological feature, with each dimension representing a specific phonological feature. Each phonological feature's +, -, and 0 are represented by a 2-bit 1 / 0 vector, and the sign of each phonological feature is represented by a 3-bit vector. <blk> 、 <spn> 、 <nsn>coding.

[0218] In implementation, the posterior probability of the phoneme is calculated by performing an inner product between the phoneme embedding vector and the acoustic feature, including:

[0219] The phonological vector p of the i-th phoneme i Convert to phoneme embedding vector e i The dimension is equal to the dimension H of the DNN output. In frame t, the output of the acoustic model DNN is...

[0220] e i Acoustic characteristics h t Calculate the inner product logits, z t,i This represents the logit corresponding to the i-th phoneme in frame t:

[0221]

[0222] In practice, the phonological vector p of the i-th phoneme is converted as follows: i Convert to phoneme embedding vector e i :

[0223] Use a linear matrix A to act on p i ,

[0224] e i =Ap i #

[0225] ;or,

[0226] Phonometry using a multi-layer neural network:

[0227] e i =A2σ(A1p) i )#

[0228] Where A1 and A2 are linear matrices, and σ(·) represents a nonlinear activation function.

[0229] During implementation, it further includes:

[0230] The Phonetisaurus toolkit is used to implement G2P (graph-to-phoneme conversion) to generate pronunciation dictionaries of IPA symbols for various languages.

[0231] During implementation, it further includes:

[0232] The training and development sets of multiple languages ​​are mixed and shuffled to form a training and development set that includes multiple languages;

[0233] The pronunciation dictionaries generated by G2P in multiple languages ​​are merged and used as the dictionary during multilingual training.

[0234] In implementation, the posterior probability of the phoneme is calculated by performing an inner product between the phoneme embedding vector and the acoustic feature, including:

[0235] Statistically analyze the IPA symbols appearing in the training set and establish corresponding phonological vectors for each IPA symbol.

[0236] The phoneme phonological vectors are concatenated into a phonological vector matrix and then input together. The phone embedding vector of each phoneme is then calculated.

[0237] During training, the output of the DNN is multiplied by the phone embedding vector converted from the phonology vector to calculate the inner product logit of each phoneme.

[0238] During implementation, it further includes:

[0239] Error backpropagation during training is used to update the parameters of the DNN and neural network.

[0240] In practice, this further includes setting the parameters during spectrogram acquisition in one or a combination of the following ways:

[0241] The training process used a speech recognition toolkit based on CTC-CRF.

[0242] For each audio data point, extract 40-dimensional FBank features and perform second-order differencing, which are then used as input to AM;

[0243] The acoustic model uses a 3-layer VGGBLSTM, with 1024 hidden units per layer of BLSTM;

[0244] During training, the random dropout factor was set to 0.5, and the Adam optimizer was used for optimization with an initial learning rate of 1e-3.

[0245] When the model’s performance on the development set declines, the learning rate is reduced to 1 / 10 of its previous value, and training is stopped when the learning rate is less than 1e-5.

[0246] In practice, after training many language models, further steps include:

[0247] The target language is identified.

[0248] Among them, Figure 5 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 500) and memory (memory 520). The bus architecture may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 510 may be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 500 is responsible for managing the bus architecture and general processing, and the memory 520 may store data used by the processor 500 during operation.

[0249] This invention also provides a multilingual training device, comprising:

[0250] The phonology module is used to obtain phonological features through a neural network and then convert them into phoneme embedding vectors encoded by vectors, where the phonological features are encoded into phonological vectors.

[0251] The acoustic module is used to extract acoustic features through an acoustic model DNN after acquiring the acoustic spectrum.

[0252] The inner product module is used to calculate the posterior probability of a phoneme by performing an inner product between the phoneme embedding vector and the acoustic feature.

[0253] In implementation, the phonology module is further used to encode phonological features into phonological vectors, including:

[0254] Each IPA phoneme in each language is represented as a 24-dimensional phonological feature, with each dimension representing a specific phonological feature. Each phonological feature's +, -, and 0 are represented by a 2-bit 1 / 0 vector, and the sign of each phonological feature is represented by a 3-bit vector. <blk> 、 <spn> 、 <nsn>coding.

[0255] In implementation, the inner product module is further used to calculate the posterior probability of a phoneme after performing an inner product between the phoneme embedding vector and the acoustic features, including:

[0256] The phonological vector p of the i-th phoneme i Convert to phoneme embedding vector e i The dimension is equal to the dimension H of the DNN output. In frame t, the output of the acoustic model DNN is...

[0257] e i Acoustic characteristics h t Calculate the inner product logits, z t,i This represents the logit corresponding to the i-th phoneme in frame t:

[0258]

[0259] In implementation, the phonology module is further used to convert the phonology vector p of the i-th phoneme in the following manner. i Convert to phoneme embedding vector e i :

[0260] Use a linear matrix A to act on p i ,

[0261] e i =Ap i #

[0262] ;or,

[0263] Phonometry using a multi-layer neural network:

[0264] e i =A2σ(A1p) i )#

[0265] Where A1 and A2 are linear matrices, and σ(·) represents a nonlinear activation function.

[0266] In practice, the phonology module is further used to implement glyph-phoneme conversion (G2P) using the Phonetisaurus toolkit, generating pronunciation dictionaries of IPA symbols for various languages.

[0267] In practice, the phonology module is further used to mix and shuffle training and development sets of multiple languages ​​to form training and development sets that include multiple languages.

[0268] The pronunciation dictionaries generated by G2P in multiple languages ​​are merged and used as the dictionary during multilingual training.

[0269] In implementation, the inner product module is further used to calculate the posterior probability of a phoneme after performing an inner product between the phoneme embedding vector and the acoustic features, including:

[0270] Statistically analyze the IPA symbols appearing in the training set and establish corresponding phonological vectors for each IPA symbol.

[0271] The phoneme phonological vectors are concatenated into a phonological vector matrix and then input together. The phone embedding vector of each phoneme is then calculated.

[0272] During training, the output of the DNN is multiplied by the phone embedding vector converted from the phonology vector to calculate the inner product logit of each phoneme.

[0273] During implementation, it further includes:

[0274] The update module is used for backpropagation of errors during training to update the parameters of the DNN and neural network.

[0275] During implementation, it further includes:

[0276] The settings module is used to configure the acquisition of the sound spectrum in one or a combination of the following ways:

[0277] The training process used a speech recognition toolkit based on CTC-CRF.

[0278] For each audio data point, extract 40-dimensional FBank features and perform second-order differencing, which are then used as input to AM;

[0279] The acoustic model uses a 3-layer VGGBLSTM, with 1024 hidden units per layer of BLSTM;

[0280] During training, the random dropout factor was set to 0.5, and the Adam optimizer was used for optimization with an initial learning rate of 1e-3.

[0281] When the model’s performance on the development set declines, the learning rate is reduced to 1 / 10 of its previous value, and training is stopped when the learning rate is less than 1e-5.

[0282] During implementation, it further includes:

[0283] The recognition module is used to recognize the target language after training many language models.

[0284] For ease of description, the various parts of the device described above are divided into modules or units according to their functions. Of course, in implementing this invention, the functions of each module or unit can be implemented in one or more software or hardware components.

[0285] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multilingual training method.

[0286] For specific implementation details, please refer to the implementation of multilingual training methods.

[0287] In summary, the technical solution provided by this invention introduces phonological features, converts them into phoneme embedding vectors using linear or nonlinear networks, and combines them with bottom-up extracted acoustic features to jointly calculate the posterior probability of the phonemes. The introduced phonological knowledge can more closely link different phonemes at the levels of articulation place and manner of articulation, avoiding the cumbersome process of training phonological feature extractors in existing technologies.

[0288] Furthermore, it is also helpful for unseen phonemes (i.e., phonemes that do not appear in the training set) in cross-language recognition.

[0289] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0290] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0291] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0292] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0293] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.< / nsn> < / spn> < / blk> < / nsn> < / spn> < / blk> < / nsn> < / spn> < / blk> < / nsn> < / spn> < / blk> < / nsn> < / spn> < / blk> < / nsn> < / spn> < / blk> < / nsn> < / spn> < / blk>

Claims

1. A multilingual training method, characterized in that, include: After obtaining phonological features through a neural network, they are converted into phoneme embedding vectors encoded by vectors, where the phonological features are encoded as phonological vectors; After obtaining the acoustic spectrum, acoustic features are extracted using a deep neural network (DNN) acoustic model. The posterior probability of a phoneme is calculated by taking the inner product of the phoneme embedding vector and the acoustic feature. Encoding phonological features into phonological vectors includes: Each IPA phoneme in each language is represented as a 24-dimensional phonological feature, with each dimension representing a specific phonological feature. Each phonological feature's +, -, and 0 are represented by a 2-bit 1 / 0 vector, and the symbol for each phonological feature is represented by a 3-bit vector. <blk> 、 <spn> 、 <nsn> coding;< / nsn> < / spn> < / blk> The posterior probability of a phoneme is calculated by taking the inner product of the phoneme embedding vector and the acoustic feature, including: The phonological vector of the i-th phoneme Convert to phoneme embedding vector The dimension is equal to the dimension H of the DNN output. In frame t, the output of the acoustic model DNN is... ; Will Acoustic characteristics Calculate logits by performing the inner product. This represents the logit corresponding to the i-th phoneme in frame t: ； The phonological vector of the i-th phoneme is calculated as follows: Convert to phoneme embedding vector : Using linear matrices To play a role A : ;or, Phonometry using a multi-layer neural network: ； in It is a linear matrix. This represents a non-linear activation function.

2. The method as described in claim 1, characterized in that, Further includes: The Phonetisaurus toolkit is used to implement glyph-to-phoneme conversion (G2P) and generate pronunciation dictionaries of IPA symbols for various languages.

3. The method as described in claim 1, characterized in that, Further includes: The training and development sets of multiple languages ​​are mixed and shuffled to form a training and development set that includes multiple languages; The pronunciation dictionaries generated by G2P in multiple languages ​​are merged and used as the dictionary during multilingual training.

4. The method as described in claim 3, characterized in that, The posterior probability of a phoneme is calculated by taking the inner product of the phoneme embedding vector and the acoustic feature, including: Statistically analyze the IPA symbols appearing in the training set and establish corresponding phonological vectors for each IPA symbol. The phoneme phonological vectors are concatenated into a phonological vector matrix and then input together. The phoneme embedding vector of each phoneme is then calculated. During training, the output of the DNN is multiplied by the phone embedding vector converted from the phonology vector to calculate the inner product logit of each phoneme.

5. The method as described in claim 4, characterized in that, Further includes: Error backpropagation during training is used to update the parameters of the DNN and neural network.

6. The method as described in claim 1, characterized in that, This further includes setting the parameters in one or a combination of the following ways when acquiring the spectrogram: During training, a speech recognition toolkit based on connectionist temporal classification-conditional random field (CTC-CRF) was used. For each audio data point, 40-dimensional FBank features are extracted and second-order differences are performed, which are then used as input to the acoustic model AM. The acoustic model uses a 3-layer visual geometry group bidirectional long short-term memory VGGBLSTM, with each layer of the bidirectional long short-term memory BLSTM containing 1024 hidden units. During training, the random dropout factor was set to 0.5, and the Adam optimizer was used for optimization with an initial learning rate of 1e-3. When the model’s performance on the development set declines, the learning rate is reduced to 1 / 10 of its previous value, and training is stopped when the learning rate is less than 1e-5.

7. The method as described in any one of claims 1 to 6, characterized in that, After training many language models, further steps include: The target language is identified.

8. A multilingual training device, characterized in that, include: The processor is used to read programs from memory and execute the following procedures: After obtaining phonological features through a neural network, they are converted into phoneme embedding vectors encoded by vectors, where the phonological features are encoded as phonological vectors; After obtaining the acoustic spectrum, acoustic features are extracted using an acoustic model DNN; The posterior probability of a phoneme is calculated by taking the inner product of the phoneme embedding vector and the acoustic feature. A transceiver is used to receive and send data under the control of a processor; Encoding phonological features into phonological vectors includes: Each IPA phoneme in each language is represented as a 24-dimensional phonological feature, with each dimension representing a specific phonological feature. Each phonological feature's +, -, and 0 are represented by a 2-bit 1 / 0 vector, and the sign of each phonological feature is represented by a 3-bit vector. <blk> 、 <spn> 、 <nsn> coding;< / nsn> < / spn> < / blk> The posterior probability of a phoneme is calculated by taking the inner product of the phoneme embedding vector and the acoustic feature, including: The phonological vector of the i-th phoneme Convert to phoneme embedding vector The dimension is equal to the dimension H of the DNN output. In frame t, the output of the acoustic model DNN is... ; Will Acoustic characteristics Calculate logits by performing the inner product. This represents the logit corresponding to the i-th phoneme in frame t: ； The phonological vector of the i-th phoneme is calculated as follows: Convert to phoneme embedding vector : Using linear matrices To play a role A : ;or, Phonometry using a multi-layer neural network: ； in It is a linear matrix. This represents a non-linear activation function.

9. A multilingual training device, characterized in that, include: The phonology module is used to obtain phonological features through a neural network and then convert them into phoneme embedding vectors encoded by vectors, where the phonological features are encoded into phonological vectors. The acoustic module is used to extract acoustic features through an acoustic model DNN after acquiring the acoustic spectrum. The inner product module is used to calculate the posterior probability of a phoneme by performing an inner product between the phoneme embedding vector and the acoustic feature. The phonology module is further used in encoding phonological features into phonological vectors, including: Each IPA phoneme in each language is represented as a 24-dimensional phonological feature, with each dimension representing a specific phonological feature. Each phonological feature's +, -, and 0 are represented by a 2-bit 1 / 0 vector, and the sign of each phonological feature is represented by a 3-bit vector. <blk> 、 <spn> 、 <nsn> coding;< / nsn> < / spn> < / blk> The inner product module is further used to calculate the posterior probability of a phoneme after performing an inner product between the phoneme embedding vector and the acoustic features, including: The phonological vector of the i-th phoneme Convert to phoneme embedding vector The dimension is equal to the dimension H of the DNN output. In frame t, the output of the acoustic model DNN is... ; Will Acoustic characteristics Calculate logits by performing the inner product. This represents the logit corresponding to the i-th phoneme in frame t: ； The phonology module is further used to convert the phonology vector of the i-th phoneme in the following manner. Convert to phoneme embedding vector : Using linear matrices To play a role A : ;or, Phonometry using a multi-layer neural network: ； in It is a linear matrix. This represents a non-linear activation function.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images