Systems and methods for a multilingual speech recognition framework

Abstract

The embodiments described herein provide an adaptation and adjustment (A2) mechanism for multilingual speech recognition models that combines both adaptation and adjustment methods as an integrated end-to-end training to improve the model’s generalization and mitigate the long-tail problem. Specifically, the multilingual language model mBERT is utilized and converted into a self-attention decoder. Furthermore, to explore the acoustic space beyond the text space, a cross-attention module is added on top of the self-attention layers of mBERT in the encoder. The joint training of the encoder and mBERT decoder can bridge the semantic gap between speech and text.

Description

[0001] Inventors: G. Wong, C.H. Hoy, and G.I. Vinata

[0002] Cross-references

[0003] The published text claims priority to U.S. Patent Application No. 17 / 162,624, filed January 29, 2021, which is a non-provisional application filed October 2, 2020, under 35U.SC119, and claims priority to that provisional application. Both are hereby explicitly incorporated herein by reference. Technical Field

[0004] The publicly available texts generally relate to machine learning models and neural networks, and more specifically, to a multi-task learning framework based on end-to-end transformation for multilingual speech recognition. Background Technology

[0005] End-to-end (E2E) speech recognition systems have been applied to handle multilingual speech recognition tasks. E2E systems eliminate the need for language-dependent vocabulary, phoneme units, and language models. For E2E-based multilingual systems, a single network can be trained from a pooled dataset containing all target languages. Compared to monolingual systems, multilingual models allow for parameter sharing and knowledge transfer across languages, which has proven beneficial for all languages, especially low-resource languages. However, a challenge in training E2E-based multilingual speech recognition is the data imbalance problem, also known as the long-tail data distribution problem. Specifically, for real-world multilingual data, there are significantly more training samples for the major dominant languages, such as English, than for low-resource languages, causing the multilingual model to be biased towards the dominant languages.

[0006] Therefore, it is necessary to improve multilingual models with imbalanced training data from the real world. Attached Figure Description

[0007] Figure 1 A simplified illustration of the architecture of an adaptive and adaptive framework for end-to-end multilingual speech recognition according to the implementation described herein is shown.

[0008] Figure 2A Some embodiments described herein are illustrated. Figure 1 The diagram shows a simplified representation of the encoder's structure.

[0009] Figure 2B Some embodiments described herein are illustrated. Figure 1 The diagram shows a simplified representation of the decoder's structure.

[0010] Figure 3This is a simplified diagram illustrating the parameter transfer from the pre-trained multilingual language model mBERT to the transformation decoder according to the implementation described herein.

[0011] Figure 4A An exemplary diagram illustrating the workflow within the illustrative language adapter according to the implementation described herein, and Figure 4B It shows Figure 1 and Figures 2A to 2B An exemplary structure of a language adapter is shown.

[0012] Figure 5 An illustration of an implementation scheme described herein is provided. Figure 1 The example diagram shows an example operation of the adjustment module.

[0013] Figure 6 This is a simplified diagram of a computing device for an end-to-end conversion framework for multilingual speech recognition, based on some implementation schemes.

[0014] Figure 7A An illustration of an implementation scheme according to the description herein is provided. Figure 1 An exemplary pseudocode snippet illustrating the processing flow of the multilingual framework implementation, and Figures 7B to 7C An exemplary logic flowchart corresponding to the exemplary pseudocode segment is provided.

[0015] Figure 8 A table of exemplary test results based on the character error rate (CER) on the training dataset is provided according to one implementation described herein.

[0016] Figure 9 A table showing the results of an ablation study of mBERT based on one implementation described herein (demonstrating the effectiveness of mBERT) is provided.

[0017] Figure 10 A table of example ablation study results on a language adapter based on one implementation described herein is provided.

[0018] Figure 11 A table of example ablation study results for a language adapter based on one implementation described herein is provided (showing the effectiveness of the public language adapter).

[0019] Figure 12 An exemplary performance table is provided according to one embodiment described herein, which shows logistic regression tuning for the training and inference phases with different models.

[0020] In the accompanying drawings and appendices, elements with the same name have the same or similar functions. Detailed Implementation

[0021] Multilingual automatic speech recognition (ASR) models are typically used to recognize multiple languages. In practical applications, multilingual ASR models often encounter the problem of imbalanced training data, commonly known as the long-tailed distribution challenge. For example, some resource-rich languages ​​(such as English) have abundant training data, while most low-resource languages ​​have varying amounts of training data. This imbalanced data setting presents many open challenges for multi-task training because the distribution of training data is often quite skewed. Therefore, this imbalanced training dataset can be caused by at least two real-world scenarios: first, very limited audio samples are available for low-resource languages ​​(such as Kyrgyz, Swedish, and Turkish), while simultaneously, there is a large amount of data from high-resource languages ​​(such as English, French, and Spanish); and second, character or sub-word labels follow a long-tailed distribution in multilingual ASR because some labels appear significantly more frequently, even in monolingual settings. Furthermore, multilingual systems may include languages ​​with writing systems other than Latin, such as Chinese or Cyrillic, which further exacerbates the skewness.

[0022] To address the problem of skewed training data distribution, a major challenge is class imbalance, such as multilingual models favoring the dominant language. Another challenge is how to robustly model languages ​​with limited training data. Specifically, the "long-tail problem" can be twofold: 1) long-tailed class distribution caused by skewed multilingual data and sentence chunk distribution; and 2) robust modeling of languages ​​with limited training data (i.e., tail languages).

[0023] To address the long-tail problem in multilingual speech recognition, the implementation described in this paper employs speech conversion combined with a language adapter in both the encoder and decoder, along with logistic regression tuning that follows the decoder. The conversion framework (referred to as the Adaptation and Tuning (A2) framework) combines adaptation and tuning methods as a complete end-to-end training process to improve model generalization and mitigate the long-tail problem.

[0024] Specifically, the extracted mBERT model is converted into an autoregressive transform decoder to jointly explore multilingual acoustic and textual spaces, thereby improving performance for low-resource languages. For example, the transform decoder uses parameters from a pre-trained mBERT. A cross-attention module is added to the encoder above the self-attention layer of mBERT to explore acoustic spaces beyond the textual space. Joint training of the encoder and mBERT decoder can bridge the semantic gap between speech and text. In this way, the language generation capability for low-resource languages ​​with insufficient data for training can be significantly improved.

[0025] Furthermore, to enable multilingual networks to adapt to language specificity with minimal additional parameters, language-specific and language-agnostic adapters are employed in parallel to augment each encoder and decoder layer. While the language-specific adapter focuses on adapting shared network weights to a specific language, a language-agnostic (public) adapter is proposed to learn some shared and language-agnostic knowledge for better knowledge transfer across languages.

[0026] In one implementation, to increase the relative margin between the logistic regressions for rare languages ​​and the dominant language, class imbalance adjustment is performed during the training or inference of the multilingual model adjusted based on logistic regression. Specifically, class imbalance adjustment is applied by adjusting the logistic regression of the softmax input with class priors. This approach balances the relative gap between the dominant and rare languages ​​and can be applied in conjunction with end-to-end learning or as a plug-and-play method during post-training.

[0027] As used herein, the term “network” can include any hardware- or software-based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or with it.

[0028] As used herein, the term "module" can include a hardware- or software-based framework that performs one or more functions. In some implementations, the module may be implemented on one or more neural networks.

[0029] A2 Framework Overview

[0030] Figure 1 A simplified illustration of the architecture of the A2 framework 100 for end-to-end multilingual ASR according to the implementation described herein is shown. Specifically, the A2 framework 100 is built on a transformation-based sequence-to-sequence model as the base model. For example, a sequence-to-sequence speech transformation model based on a hybrid connectionist temporal classification (CTC)-attention network is used for acoustic modeling. Such transformation models can be similar to those described below: Dong et al., Non-recursive sequence-to-sequence model for speech recognition, Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5884-5888, 2018; Kim et al., End-to-end speech recognition based on joint CTC attention using multi-task learning, CoRR, abs / 1609.067732016; Karita et al., Improvement of transformation-based end-to-end speech recognition with integration of connectionist temporal classification and language models, Proceedings of Interspeech 2019, pp. 1408-141222019, the entire contents of which are explicitly incorporated herein by reference.

[0031] The basic conversion model includes an encoder 110 and a decoder 115. The encoder 110 receives an audio feature sequence. The input is T, and the output is the audio latent representation f = fnc(x), where T and F represent the sequence length and feature dimension, respectively.

[0032] Encoder 110 includes multiple encoder layers. Each encoder layer includes a two-dimensional convolutional layer 102 followed by a self-attention layer 106. The convolutional layers 102 are used to extract more robust features before they are sent to the self-attention layer 106. For example, Figure 2A Simplified diagrams illustrating the structure of encoder 110 according to some embodiments described herein are provided. Figure 2A As shown, in the two layers of normalized 104a-104b ( Figure 1 (Not shown in the image) and an adapter layer 108 are added after the self-focused layer 106. The self-focused layer 106 focuses on the characteristics of the input. (As combined with...) Figure 3 Further described, adapter layer 108 includes a language-specific adapter that adapts the self-focused output to an adapter output, capturing language-intrinsic knowledge from the self-focused features, and a public adapter that adapts the self-focused output to an adapter output, capturing language-independent knowledge from the self-focused features. Feedforward sublayer 109 generates an encoded representation of the input (encoder hidden state) based on a combination of the first adapter output and the second adapter output.

[0033] Two residual connections 202 and 204 are applied after the self-focus layer 106 and the adapter layer 108. The feedforward layer 109 outputs the encoder layer output of the l-th encoder layer.

[0034]

[0035]

[0036] in This refers to the hidden state of the encoder in the previous layer (1-1), and It is the output of the encoder layer.

[0037] Return to reference Figure 1 The decoder 115 has two attention mechanisms: a self-attention layer 112 and a cross-attention layer 114 for the encoder output. For example, Figure 2B A simplified diagram illustrating the structure of decoder 115 according to some embodiments described herein is provided. Figure 2BAs shown, adapter layer 118 is placed after layer normalization 117c, cross-concern layer 114, layer normalization 117b, self-concern layer 112, and layer normalization 117a. Self-concern layer 112 focuses on the embeddings corresponding to the previously labeled data. Cross-concern layer 114 focuses on the encoder hidden state and the normalized self-concern output. Adapter 118 includes a language-specific adapter and a common adapter that adapt the cross-concern outputs within the decoder layer. Feedforward sublayer 119 generates a logit output based on a combination of the outputs from the second language-specific adapter and the second common adapter.

[0038] Therefore, the hidden state of the decoder in layer l is calculated as follows:

[0039]

[0040] o2=CrossAttn(henc, LayerNorm(o1))+o1,

[0041]

[0042] in It is the hidden state of the encoder in the previous layer, and This is the output of the current layer.

[0043] Decoder 115 employs beam search to predict sentence segments. The decoding score is calculated as a weighted sum of the CTC and the attention network probability, using β as a decoding parameter to balance them.

[0044]

[0045] Where y' is the sequence that has been decoded so far.

[0046] Return to reference Figure 1 In one implementation, a CTC layer can be added to the encoder output to serve as a regularizer for the focus model. Further details of the CTC layer can be found in Graves et al., “Connectionist Temporal Classification: Labeling Unsegmented Sequence Data with Recurrent Neural Networks,” Proceedings of the 23rd International Conference on Machine Learning, 2006, pp. 369–376, the entire contents of which are explicitly incorporated herein by reference. Therefore, the encoder f enc The output of (x) can be passed to the softmax module 125 to produce the CTC output p. ctc .

[0047] At the decoder branch, the decoder outputs y. t =f(y t-1 h encThe data is passed to the tuning module 120. The tuning module 120 uses a normalization method to calibrate the logistic regression of the model trained on long-tailed data. For example, in an imbalanced data setting, the logistic regression generated by the model is highly skewed, giving very low scores to the rare class. Therefore, when the resource-poor class is incorrectly predicted, these scores must be adjusted according to the distribution of label occurrences to give a higher penalty. Instead of applying the adjustment to the highest estimated class probability, the tuning module 120 tunes the logistic regression for all classes, as some existing systems may choose to do. Therefore, the tuning process can be applied to generative models, such as sequence-to-sequence models, and this approach is task-independent. The following section combines... Figure 5 Further details describe the logic adjustment module 120.

[0048] The output p of the language adjustment module 120 attn With p ctc Together they can be used to calculate the loss target to jointly train encoder 110 and decoder 115.

[0049] The framework 100 is trained in an autoregressive manner by predicting the next label given the current output. The encoder 110 and decoder 115 can be trained using a multi-task loss function. (Combined output p from CTC) ctc The calculated CTC loss and the output p from the decoder attn Calculated loss of attention (To conduct joint training. Multi-task loss) It can be calculated as an interpolation of the CTC loss and the attention loss using the hyperparameter λ (0≤λ≤1):

[0050]

[0051]

[0052] Where p Y This is the label distribution after label smoothing, to prevent the model from making overconfident predictions. The Kulbak-Leibler divergence loss (KL) is used for attention loss.

[0053] In one implementation, the transformation decoder 115 can be replaced by a pre-trained mBERT 105 for better language modeling, especially for low-resource languages. For example, the parameters of the pre-trained mBERT can be fed into the transformation decoder 115, as shown below. Figure 3 Further description.

[0054] Figure 3 This is a simplified diagram illustrating the parameter transfer from a pre-trained multilingual language model mBERT 105 to a speech recognition decoder 115 according to the implementation described herein. Figure 3As shown, mBERT 105 can be adapted to autoregressive transform decoder 115 in the following manner: the embedding from mBERT embedding layer 303 and the self-attention parameters from mBERT self-attention layer 304 are transferred to the embedding layer 313 and self-attention layer 112 of the decoder layer of decoder 115, respectively.

[0055] Since mBERT105 was initially trained only on text data, a cross-interest layer 114 was added to the decoder 115 to explore the acoustic and text spaces, and applied to the encoder output of each mBERT layer except for the self-interest layer 112. In this way, the cross-interest layer 114 and the self-interest layer 112 are learned to "align" the acoustic and text spaces for speech recognition. This is because the text space may deviate significantly from the acoustic space of the encoder output.

[0056] Specifically, in the current decoding step t, the autoregressive decoder 115 uses the current input label y. t To predict the next label y t+l Embedding layer 313 (operating under the embedding from mBERT embedding layer 303) converts the input tokens into a vector representation. Subsequently, cross-interest layer 114 outputs the encoder output h. enc Use the self-following output as the key and value, and use the self-following output as the query, and compute the follow-up output.

[0057] It is noteworthy that the original mBERT vocabulary size may be too large (119,547 tags) for training an end-to-end speech recognition system. Therefore, vocabulary mapping is performed to reduce the number of targets for speech conversion. For example, sentence segments (SPs) are used as target tags. The SP model is trained on transcriptions with a preset vocabulary size. As an example implementation, a shared set of 5,237 tags can be used as the vocabulary for a multilingual system. For all monolingual systems, the minimum number of tags in the sentence chunk model is 150, except for Chinese, which has 2,265 tags. The generated sentence chunk tags are then matched against the mBERT tag set. During training, the embeddings of all tags in the mBERT vocabulary are initialized with mBERT embeddings.

[0058] Figure 4A An exemplary diagram 400 illustrates the workflow within the illustrative language adapter 108 or 118 according to the embodiments described herein, and Figure 4B It shows Figure 1 and Figures 2A to 2BThe exemplary structure of language adapter 108 or 118 shown is illustrated. For example, language adapter 108 or 118 can employ a dual-adaptor structure: a lightweight residual language adapter 402 for better acoustic modeling with minimal language-specific parameters to increase model robustness for languages ​​with limited resources; and a shared common adapter 404, in addition to the language-specific adapter 402 for capturing intrinsic language knowledge, is trained to learn language-independent information in multilingual data. The language-specific and common adapters 402 and 404 are respectively denoted as A lang and A com .

[0059] In one implementation, the language-specific adapter is identified by a language mask. For example, such as Figure 4A As shown, the language-specific adapter 402 is designed for Spanish.

[0060] like Figure 4B As shown, each adapter 108 or 118 in layer 1 includes layer normalization 411 and a downward projection layer 412W. l d Following this is the ReLU activation function 413 and the upward projection layer 414W. l u Specifically, the language-specific adapter 402 and the common adapter 404 receive h l As input, where h l This is the self-focused output of layer l (e.g., from self-focused layer 106 in encoder 110 or self-focused layer 112 in decoder 115, depending on whether the language adapter resides in encoder 110 or decoder 115). Then, for language-specific adapter 402 and common adapter 404, adapter Adapter(h l The output of ) can be calculated as follows, configuring W for the language-specific adapter 402 and the common adapter 404 respectively. l d W l u ReLU parameters:

[0061]

[0062] For example, Adapter(h l These can be calculated as the outputs of the language-specific adapter 402. l lang and the output of common adapter 404 l com Then, by outputting o from the adapter l lang and o l com and o lThe sum is used as input to the feedforward layer 109 in encoder 110 or the feedforward layer 119 in decoder 115 to calculate the final adapter output. l During training and inference, the outputs of the language-specific and common adapters are merged.

[0063] Figure 5 Provided are embodiments illustrating the implementation described herein. Figure 1 The diagram illustrates an example operation of the adjustment module 120. As previously mentioned, the base models of encoder 110 and decoder 115 may be affected by sentence block class imbalance due to differences in the availability of training data for different languages. The logistic regression adjustment module 120 can address the imbalance problem by incorporating class priors during training or inference via logistic regression adjustment.

[0064] The adjustment module can use the calculated category prior 406 to adjust the values ​​derived from... Figure 1 The decoder output of decoder 115 in the code is 407, for example, y. t =f(y t-1 h enc For example, category prior π y These are the natural frequencies of sentence block tags estimated from multilingual training data. To form an effective prior distribution, the category prior calculation module 404 applies smoothing to the raw counts of zero-occurrence tags to calculate the category prior:

[0065]

[0066] Where C is the total count of all tags, n o It is the number of tags that appear zero times, N is the number of categories, and C is the number of tags that appear zero times. i It is the original count of category i.

[0067] The category prior 406 is then output to the logistic regression adjuster 408, which also receives the decoder output 407. The logistic regression adjuster 408 can output to the softmax classifier 410, where the adjusted logistic regression 406 serves as input that minimizes the balance error across all categories. At the logistic regression adjuster 408, the natural adjustment is to adjust the logistic regression f... y (x) scaled to the corresponding class prior π y The reciprocal of r. In the log domain, it can be adjusted as follows: where r > 0 is a hyperparameter. The adjustment can be viewed as applying class-dependent offsets to reweight each logistic regression based on its class prior.

[0068] To incorporate priors during training, including logistic regression from the last decoder layer. The decoder output 407 is adjusted before softmax 410 according to the following:

[0069]

[0070]

[0071]

[0072] Adjusted softmax output vector from a sequence of softmax classifier 410 Used as p attn To calculate as about Figure 1 The loss objective is described, and backpropagation is performed to update the base model. t-l These are the labels that were previously only available during training. To reduce the discrepancy between training and inference, scheduled sampling can be used for sequential classification tasks like speech recognition. In later training iterations, instead of using the ground truth labels y... t-l To calculate logistic regression, approximate labels y' can be used. t-l It selects from the maximum predicted output of the current model to simulate inference:

[0073]

[0074] In one implementation, if scheduled sampling is used, the adjusted logistic regression 406 at step t will affect all subsequent labels in the current sequence. In some traditional label smoothing methods, the prior π y Typically, it is a uniform distribution independent of the data. Logistic Regression Adjusted 408 applies class-specific "smoothing" based on class priors and has been shown to outperform baselines with standard label smoothing.

[0075] Alternatively, the category prior 406 can be incorporated during inference through logistic regression adjustment. The decoding score is calculated as follows:

[0076]

[0077] During beam search, focus on the decoding score. It is calculated in the same manner as planned sampling based on adjusted logistic regression 406.

[0078] Computer environment

[0079] Figure 6 This is a simplified diagram of a computing device for implementing an end-to-end conversion framework 600 for multilingual speech recognition, according to some implementation schemes. (See diagram for example.) Figure 6As shown, computing device 600 includes a processor 610 coupled to memory 620. The operation of computing device 600 is controlled by processor 610. And although computing device 600 is shown to have only one processor 610, it should be understood that processor 610 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), and / or the like in computing device 600. Computing device 600 may be implemented as a standalone subsystem, such as a board added to the computing device and / or a virtual machine.

[0080] Memory 620 may be used to store software executed by computing device 600 and / or one or more data structures used during operation of computing device 600. Memory 620 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, floppy hard disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tape, any other physical media with a perforated pattern, RAM, PROM, EPROM, flash memory EPROM, any other memory chip or cassette tape, and / or any other media adapted for reading by a processor or computer.

[0081] Processor 610 and / or memory 620 can be arranged in any suitable physical configuration. In some embodiments, processor 610 and / or memory 620 may be implemented on the same board, in the same package (e.g., system in package), on the same chip (e.g., system on chip), and / or the like. In some embodiments, processor 610 and / or memory 620 may include distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, processor 610 and / or memory 620 may reside in one or more data centers and / or cloud computing facilities.

[0082] In some embodiments, memory 620 may include a non-transitory, tangible, machine-readable medium comprising executable code that, when run by one or more processors (e.g., processor 610), enables one or more processors to perform the methods described further in detail herein. For example, as shown, memory 620 includes instructions for a multilingual speech recognition module 660 (which can be used to implement and / or simulate systems and models, and / or implement any methods further described herein). In some embodiments, multilingual speech recognition module 660 may receive input 640, such as audio speech samples in different languages, via data interface 615. Data interface 615 may be either a user interface for receiving articulated audio input or a communication interface for receiving or retrieving previously stored audio samples from a database. Multilingual speech recognition module 660 may generate output 650, such as a transcript corresponding to audio speech input 640.

[0083] In some implementations, the multilingual speech recognition module 660 can achieve the following: Figure 1 The encoder 110 and decoder 115 shown are based on a transformation model. (Refer to reference...) Figure 1 Similar to those described, encoder 110 and decoder 115 each include language adapters 108 and 118, respectively. The multilingual speech recognition module 660 may also include... Figure 1 The logistic regression adjuster 120 is shown.

[0084] In some embodiments, the multilingual speech recognition module 660 and submodules 110, 115 and 120 may be implemented using hardware, software and / or a combination of hardware and software.

[0085] Adapting and adjusting workflows

[0086] Figure 7A An illustration of an implementation scheme according to the description herein is provided. Figure 1 An exemplary pseudocode snippet illustrating the processing flow of the multilingual framework 100 shown, and Figure 7B child Figure 7C An exemplary logic flowchart 700 corresponding to an exemplary pseudocode segment is provided. One or more of the processes 702-728 of method 700 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when run by one or more processors, enables one or more processors to execute one or more of processes 702-728. In some embodiments, method 700 may correspond to the method used by module 660.

[0087] In step 702, for example, a training dataset of multilingual data can be received at data interface 615, and a conversion model including encoder 110 and decoder 115 can be initialized with randomly generated parameters. For example, the conversion model can be stored in memory 620.

[0088] In step 704, decoder parameters can be transferred from a pre-trained language model (e.g., mBERT) to the transformation decoder, for example, as per [reference to...]. Figure 3 As described.

[0089] In step 706, a category prior can be calculated based on the multilingual training data, for example, as described in category prior module 404 in Figure 4.

[0090] In step 708, a batch of multilingual discourses can be sampled from the training dataset.

[0091] In step 710, language tags from the batch of discourses can be used to generate a language adapter mask. For example, the language adapter mask can indicate a specific type of language in a multilingual discourse, such as Spanish, English, etc.

[0092] In step 712, encoder 110 can calculate encoder hidden state based on input utterance x and language mask, for example h enc .

[0093] In step 714, decoder 115 can compute logistic regression based on encoder hidden state and language mask.

[0094] In step 716, the calculated logistic regression can be adjusted using language tuning, for example, regarding... Figure 5 As described.

[0095] In step 718, the encoder can generate the CTC posterior, for example... Figure 1 p shown CTC (y|h enc ).

[0096] In step 720, the attention loss can be calculated based on the adjusted logistic regression and attention output.

[0097] In step 724, the multi-task loss can be calculated based on the attention loss, CTC output, and interpolation parameter λ.

[0098] In step 726, the transformed models, such as encoder 110 and decoder 115, can then be jointly updated based on the multi-task loss via backpropagation.

[0099] Steps 708 to 726 can be repeated until there are no more training steps (at decision 728). When there are no more training steps, process 700 can end after decision 728; otherwise, steps 708 to 726 can be repeated starting from decision 728.

[0100] Example Performance

[0101] The multilingual training dataset can include a general speech dataset (see Aldira et al., "General Speech: A Large-Scale Multilingual Speech Corpus," Proceedings of the 12th Conference on Language Resources and Evaluation, 2020, pp. 4218-4222). Eleven languages ​​are used: English (en), Spanish (es), French (fr), Italian (it), Kyrgyz (ky), Dutch (nl), Russian (ru), Swedish (sv), Turkish (tr), Tatar (tt), and Chinese (zh). Following the ESPNET recipe, the dataset is split into training, development, and evaluation sets. Transcriptions are tokenized using a sentence fragment model with a unigram model algorithm. Speech transcriptions are used to train the sentence fragment model. Special tags are added for CTC targets, such as... <unk> 、 <sos> 、 <eos>And blank markers.

[0102] The model employs six transformation encoder layers with a hidden size of 2048 units and eight attention heads, each with an attention dimension of 256. For the decoder, distil-mBERT is used (see Sanh et al., A Distillation of BERT: Smaller, Faster, Cheaper, and Lighter, arXiv preprint arXiv:11910.011082019). The mBERT decoder consists of six transformation decoder layers and four attention heads with a hidden size of 3072 and an attention dimension of 756. The model is trained with a batch size of 32 and gradients are accumulated in two steps using a single GPU NVIDIA V100 16GB to achieve a larger batch size. The model is trained using the Adam optimizer with a warm-up step size of 25000. Specifically, for balanced sampling, six samples are taken for each language, and balanced batches are constructed by accumulating gradients 11 times.

[0103] The model can be implemented using beam search with a beamwidth of 10, λ = 0.3, and β = 0.5. For class imbalance adjustment during both training and inference phases, the hyperparameter τ is set to 0.3. The multilingual model is trained over 150K iterations. The average of the last ten checkpoints is used to calculate the decoding model. For the monolingual setting, training stops after 100 epochs. The character error rate (CER) is used to evaluate the model to simplify evaluation and provide a universal metric for all languages.

[0104] For baseline comparisons, baseline models include: monolingual (monlingual model); SMT (Standard Multilingual Training), which randomly samples batches from the data distribution; BS (Balanced Sampling), which samples an equal number of utterances for each language in a batch so that they contribute roughly equally to the training; large-scale multilingual speech recognition with streaming end-to-end models by Kannan et al., Proc, Interspeech, pp. 2130-2134, 2019; and LID (Language ID) LAN-specific adapters including the language-specific adapter proposed by Kannan et al.; and single-sequence-to-sequence model for multi-dialect speech recognition by Li et al., IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2018, pp. 4749-4753, and single-hot language vector conditionalization proposed in 2018.

[0105] Figure 8 A table showing exemplary test results of CER on the training dataset according to one implementation described herein is provided. Figure 8 As shown in Table 1, even the SMT model significantly improved the performance of low-resource languages ​​compared to monolingual models. In other words, SMT is a good multilingual baseline. This is likely because multilingual models can capture sub-speech features shared by different languages, which is beneficial for the recognition of low-resource languages.

[0106] Furthermore, the performance of tail language is significantly improved compared to SMT. However, the performance of head language is affected due to its lower frequency of occurrence during training. The model clearly overfits to tail language due to upsampling; for example, the CERs for "ky" and "sv" on the training set are significantly lower than on the evaluation data (3.4% and 4.2% for training, compared to 13.4% and 22.8% for evaluation). Therefore, the overall performance is the same as SMT. In fact, even after balanced sampling, sentence block tokens still exhibit a long-tailed distribution.

[0107] Language adaptation techniques are compared to LAN-specific adapters, one-hot language vectors, and dual adapters. Note that all adapters are based on BS+mBERT, which outperforms the pure BS model. Adding language-specific adapters without a common adapter significantly outperforms the BS baseline, with an absolute performance gain of 0.9%. Another approach to injecting language information is to add one-hot language vectors. Interestingly, applying sentence block class imbalance adjustment (LID+adjustment-training) to the language vectors significantly improves CER.

[0108] Both the training and inference phases of A2 and A2 show significant performance gains compared to the LAN-specific adapter, with an absolute CER reduction of 1%. These gains are primarily due to improved head language performance, although tail languages ​​also benefit from logistic regression tuning. More importantly, the gap between monolingual and multilingual performance for head languages ​​is significantly narrowed, resulting in better "balanced error" performance. This strongly demonstrates the importance of class imbalance tuning. Compared to BS, A2 also avoids overfitting to tail languages, achieving CERs of 8.2% and 23.6% on "ky" and "sv" respectively, closer to the evaluation CER. Compared to the randomly sampled SMT, A2 has a significantly better average CER and moderate cost for both head languages ​​"fr" and "en".

[0109] Figure 9 A table of ablation study results for mBERT based on one implementation described in this paper is provided (showing the effectiveness of mBERT). The performance of mBERT is highly dependent on the quality of the acoustic model. Without adapters or logistic regression tuning, the improvement relative to BS is negligible, and for SMT, mBERT performs even worse. This may suggest that using better acoustic models, such as A2, the text space of ordinary mBERT is better aligned with the acoustic space, leading to performance improvements across all languages, especially for low-resource languages. Interestingly, even without adapters, "SMT+mBERT+tuning-training" produces the same overall CER as the best adapter system (BS+mBERT+dual adapter).

[0110] In some implementations, to study the impact of pre-trained language models, a more advanced XLM-R pre-trained model is used instead of distilled-mBERT. Although XLM-R has better multilingual language generation capabilities than mBERT, this does not translate into a final performance gain for multilingual ASR tasks.

[0111] Figure 10 A table of example ablation study results on a language adapter based on one implementation described herein is provided. Results and parameter sizes for different adapters are shown. Figure 10 The results are given in Table 3. For example, the decoder layer adapter may be less effective than the encoder layer, which suggests that adaptation in the acoustic space is much more effective than adaptation in the text space. Therefore, the decoder adapter (e.g., 118) may be optional.

[0112] Figure 11 A table of example ablation study results for a language adapter based on one implementation described herein is provided (showing the effectiveness of the common language adapter). The dual adapter significantly outperforms the language-specific adapter, with an absolute CER reduction of 0.5%, indicating that using the common adapter for knowledge transfer is effective.

[0113] In addition to individual language adapters, languages ​​are grouped to allow adapter sharing within the same group. Based on their written script, the 11 languages ​​can be grouped into language groups, such as Latin, Chinese, and Cyrillic. They can also be grouped into language families, such as Romance, Chinese, Turkic, and Germanic. This grouping focuses more on similarities in vocabulary, grammar, and pronunciation, which are typically included in end-to-end multilingual architectures. Languages ​​belonging to the same group do not necessarily belong to the same family in another group. For example, Tatar and Turkish are both Turkic languages. However, Tatar uses Cyrillic script, and Turkish uses the Latin alphabet. All languages ​​within the same group share the same dual adapter, which is trained with all language members. Generally, grouping by language family is better than grouping by written script because it is more consistent with the encoder adapter used to adapt the acoustic space, which is more efficient than the decoder adapters in Table 3.

[0114] Compared to a single language adapter, sharing a language adapter by language family helps the performance of low-resource languages, such as "sv" for Germanic languages ​​and "ky" and "tr" for Turkic languages, because more data is used to train the group adapter. However, this also incurs a cost for resource-rich languages ​​compared to a "separate dual adapter". Therefore, given that the adapter's parameter size is much smaller than the attention weights of the encoder and decoder, using a separate language adapter is recommended.

[0115] Figure 12 An exemplary performance table is provided for one implementation according to the description herein (showing logistic regression tuning for the training and inference phases with different models). Figure 12 As shown in Table 5, for the SMT system, training phase adjustment exhibits a significant advantage over inference phase adjustment. Under the convex assumption, the solutions of both adjustment methods are identical. However, deep neural network optimization is a non-convex problem, so they may converge to different local minima. Under SMT, the model is heavily biased towards the head class due to random sampling. Class imbalance adjustment during the training phase can help the training focus more on the tail classes, resulting in better balance and lower error. Using a better acoustic model, such as a language adapter, inference phase adjustment can better calibrate the original classification scores and produce similar performance to training phase adjustment.

[0116] Some embodiments of a computing device, such as computing device 100, may include a non-transitory, tangible, machine-readable medium comprising executable code that, when run by one or more processors (e.g., processor 110), causes one or more processors to perform the processing of method 600. Some common forms of machine-readable media that may include the processes of method 600 are, for example, floppy disks, floppy hard disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tape, any other physical media with a perforated pattern, RAM, PROMs, EPROMs, flash memory EPROMs, any other memory chips or cassette tapes, and / or any other media from which a processor or computer is adapted to read.

[0117] The description and accompanying drawings illustrating aspects, embodiments, implementations, or applications of the invention should not be considered limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this specification and claims. In some cases, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure the embodiments disclosed. Similar numerals in two or more numbers represent the same or similar elements.

[0118] This description sets forth specific details of some embodiments consistent with the disclosed text. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not limiting. Those skilled in the art will recognize that other elements are within the scope and spirit of the disclosed text, although not specifically described herein. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless otherwise specifically described or if one or more features would render the embodiment ineffective.

[0119] Although illustrative embodiments have been shown and described, extensive modifications, alterations, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without correspondingly using others. Those skilled in the art will recognize many variations, substitutions, and modifications. Therefore, the scope of the invention should be defined only by the following claims, and it is appropriate to interpret the claims broadly in a manner consistent with the scope of the embodiments disclosed herein.< / eos> < / sos> < / unk>

Claims

1. A system for multilingual speech recognition, the system comprising: an encoder comprising a plurality of encoder layers, wherein an encoder layer comprises: a first self-attention layer that attends to features of an input; a first language-specific adapter that adapts a first self-attention output into a first adapter output, capturing language-intrinsic knowledge from the self-attended features, a first common adapter that adapts the first self-attention output into a second adapter output, capturing language-agnostic knowledge from the self-attended features, and a first feed-forward sublayer that generates an encoded representation of the input based on a combination of the first adapter output and the second adapter output; a decoder that generates a logistic regression output for a current token based on an encoder hidden state from the encoder and a previous token, wherein the decoder comprises a plurality of decoder layers, wherein a decoder layer from the plurality of decoder layers comprises: a second self-attention layer that attends to an embedding corresponding to the previous token; a cross-attention layer that attends to the encoder hidden state and a second self-attention output from the second self-attention layer; a second language-specific adapter and a second common adapter that adapt a cross-attention output within the decoder layer, and a second feed-forward sublayer that generates the logistic regression output based on a combination of outputs from the second language-specific adapter and the second common adapter; and a language adjustment module that adjusts the logistic output from the decoder by applying a category-dependent offset to each logistic regression according to a corresponding category prior.

2. The system of claim 1, wherein the decoder is a self-recursive transducer decoder, and wherein the embedding and parameters of the second self-attention layer are copied from counterparts in a pre-trained language model.

3. The system of claim 1, wherein the cross-attention layer takes the encoder hidden state as a key and value and the second self-attention output as a query to compute the cross-attention output.

4. The system of claim 1, wherein each of the first language-specific adapter and the common adapter comprises a down-projection layer, a ReLU activation module, and an up-projection layer.

5. The system of claim 1, wherein the first language-specific adapter is specified by a language mask, and wherein the first adapter output is generated based at least in part on the language mask.

6. The system of claim 1, wherein the category prior is computed by estimating frequencies of sentence chunk tokens from a multilingual training dataset.

7. The system of claim 1, wherein the adjusted logistic regression output is used to compute a predicted probability distribution for the current token.

8. The system of claim 1, wherein the encoder and the decoder are jointly trained by a loss objective that is computed based on the adjusted logistic regression output from the encoder, a label distribution, and a connectionist temporal classification output.

9. A method for multilingual speech recognition, the method comprising: receiving a multilingual utterance; computing, by an encoder based on input of the multilingual utterance, an encoder hidden state by: attending, by a first self-attention module, to features of the multilingual utterance, adapting, by a first language-specific adapter, first self-attention output to a first adapter output, capturing language intrinsic knowledge from the self-attended features, adapting, by a first common adapter, the first self-attention output to a second adapter output, capturing language agnostic knowledge from the self-attended features, and generating, by a first feed-forward sub-layer, an encoded representation of the input of the multilingual utterance based on a combination of the first adapter output and the second adapter output; computing, by a decoder based on an encoder hidden state from the encoder and a previous token, a logistic regression output for a current token by: attending, by a second self-attention layer, to an embedding corresponding to the previous token, attending, by a cross-attention layer, to the encoder hidden state and a second self-attention output from the second self-attention layer, adapting, by a second language-specific adapter and a second common adapter within the decoder, cross-attention output, and generating the logistic regression output based on a combination of outputs from the second language-specific adapter and the second common adapter; and adjusting logistic output from the decoder by applying class-dependent bias to each logistic regression by corresponding class priors.

10. The method of claim 9, further comprising: computing, by the cross-attention layer, the encoder hidden state as keys and values and the second self-attention output as a query to compute the cross-attention output.

11. The method of claim 9, wherein the class priors are computed by frequencies of sentence chunk tokens estimated from a multilingual training dataset.

12. The method of claim 11, further comprising: computing, based on the encoder hidden state, a connectionist temporal classification output from the encoder; computing an attention loss based on a decoder output probability distribution computed from adjusted logistic output; computing a multi-task loss based on the connectionist temporal classification output and the attention loss; and jointly training the encoder and the decoder based on the multi-task loss via backpropagation.

13. A non-transitory processor-readable storage medium storing processor-executable instructions for multilingual speech recognition, the instructions executed by a processor to perform: receiving a multilingual utterance; computing, by an encoder based on input of the multilingual utterance, an encoder hidden state by: attending, by a first self-attention module, to features of the multilingual utterance, adapting, by a first language-specific adapter, first self-attention output to a first adapter output, capturing language intrinsic knowledge from the self-attended features, adapting, by a first common adapter, the first self-attention output to a second adapter output, capturing language agnostic knowledge from the self-attended features, and generating, by a first feed-forward sub-layer, an encoded representation of the input of the multilingual utterance based on a combination of the first adapter output and the second adapter output; computing, by a decoder based on an encoder hidden state from the encoder and a previous token, a logistic regression output for a current token by: attending, by a second self-attention layer, to an embedding corresponding to the previous token, attending, by a cross-attention layer, to the encoder hidden state and a second self-attention output from the second self-attention layer, adapting, by a second language-specific adapter and a second common adapter within the decoder, cross-attention output, and generating the logistic regression output based on a combination of outputs from the second language-specific adapter and the second common adapter; and adjusting logistic output from the decoder by applying class-dependent bias to each logistic regression by corresponding class priors. computing, by a decoder, a logistic regression output for a current token based on an encoder hidden state from an encoder and a previous token, by: attending, by a second self-attention layer, to an embedding corresponding to the previous token, attending, by a cross-attention layer, to the encoder hidden state and a second self-attention output from the second self-attention layer, adapting, by a second language-specific adapter and a second public adapter within the decoder, the cross-attention output, and generating the logistic regression output based on a combination of outputs from the second language-specific adapter and the second public adapter; and adjusting the logistic output from the decoder by applying category-dependent bias to each logistic regression, according to corresponding category priors.

Images