Transliteration-based data augmentation for training multilingual asr acoustic models in low-resource settings

Abstract

A computer-implemented method of building a multilingual acoustic model for automatic speech recognition in a low resource setting includes training a multilingual network on a training language set with originally transcribed training data to create a baseline multilingual acoustic model. Transliteration of the transcribed training data is performed by processing a plurality of multilingual data types from the language set through the multilingual network and outputting a pool of transliterated data. A filtering metric is applied to the pool of transliterated data output to select one or more portions of the transliterated data for retraining of the acoustic model. Data augmentation is performed by adding the one or more selected portions of the output transliterated data back to the originally transcribed training data to update the training data. Training of a new multilingual acoustic model through the multilingual network is performed using the updated training data.

Description

Background Technology Technical Field

[0002] This disclosure relates generally to acoustic speech recognition (ASR), and more specifically to training multilingual ASR acoustic models.

[0003] Description of related technologies

[0004] Acoustic models used in speech recognition systems typically involve hundreds of hours of task-specific training data. However, challenges arise when the amount of available resources for task-specific training is low—for example, far less than the typical hundreds or thousands of hours (e.g., tens of hours). Low resource settings make it difficult to adequately train the acoustic model. Consequently, the error rate of such acoustic models is often much higher than when sufficient training resources are available.

[0005] In such low-resource settings, transcribed data from other languages ​​can be used to construct multilingual acoustic models. These models can then be used to extract multilingual bottleneck features for subsequent processing, or they can be used directly as acoustic models after fine-tuning steps on the low-resource language. Previous attempts to address the low-resource setting problem have included applying data augmentation to the training data, which did not lead to significant improvements in training the acoustic model, and manual intervention.

[0006] However, even with additional transcribed data, acoustic models trained with low-resource settings exhibit relatively high word error rates (WERs). Previous attempts to improve performance include: (1) data augmentation to cleanse speech; (2) tract length perturbation (VTLP); (3) audio velocity and temperature perturbation, and various combinations of these methods.

[0007] Therefore, it is necessary to address the problem of training multilingual acoustic models with reduced WER in low-resource settings. Summary of the Invention

[0008] According to one embodiment, a computer-implemented method for building a multilingual acoustic model for automatic speech recognition in a low-resource setting includes: training a multilingual network on a training language set with training data from the original transcription to create a baseline multilingual acoustic model; performing transliteration of the transcribed training data by processing multiple multilingual data types from the language set via the multilingual network and outputting a transliterated data pool; applying a filtering metric to the transliterated data output pool to select one or more portions of the transliterated data for retraining the acoustic model; performing data augmentation to update the training data by adding one or more selected portions of the output transliterated data back to the original transcribed training data; and training a new multilingual acoustic model with the updated training data. This computer-implemented method provides more accurate training of the multilingual acoustic model, at least due to the transliteration and data augmentation operations and the data augmentation performed on the output transliterated data. Applying data augmentation to the output transliterated data, particularly in a low-resource setting, also provides more diverse and accurate training of the acoustic model because the training data is improved and augmented.

[0009] In one embodiment, the computer-implemented method further includes retraining the benchmark multilingual acoustic model with updated training data. This operation results in a more accurate and robust benchmark acoustic model.

[0010] In one embodiment, the original training data comes from a low-resource language, and the multilingual network includes a neural network comprising multiple language-specific output layers configured to model the symbol set of each language, wherein the neural network outputs language-specific portions of the transliterated data to at least one corresponding language-specific output layer. The use of neural networks provides faster and more efficient training of acoustic models.

[0011] In one embodiment, adding one or more selected portions of the output transliterated data back to the original transcription during training involves relabeling the data from the new data copy using symbols in other languages. Relabeling the data from the new data copy using symbols in other languages ​​assists in training the multilingual network for the multilingual acoustic model.

[0012] In one embodiment, training of a multilingual network on a training language set is performed using low-resource settings with training data consisting of raw transcripts comprising tens of hours of raw transcript data. This method provides more accurate training of the acoustic model, as training would otherwise be difficult and inefficient with low-resource settings.

[0013] In one embodiment, the computer-implemented method includes generating semi-supervised labels in response to processing untranscribed data by a multilingual neural network.

[0014] In one embodiment, filtering of the transliterated data pool is performed by selecting one or more portions of the pool that have a relatively higher symbol count compared to the remainder of the transliterated data. This operation provides for training a more accurate acoustic model. The application of filtering metrics provides for more robust and accurate training of multilingual acoustic models.

[0015] In one embodiment, a metric filtering of the transliterated data pool is performed by comparing the ratio of symbols in the transliterated data to the ratio of symbols in the utterance of the training data, which includes the original transcriptions, and selecting one or more portions of the transliterated data pool that have a higher ratio of symbols. The application of this filtering metric provides for more robust and accurate training of the multilingual acoustic model.

[0016] In one embodiment, processing of multiple multilingual data types includes processing transcribed training data, untranscribed data from the same training language set, and untranscribed data from different languages.

[0017] In one embodiment, a new language is added to the input of the multilingual network, and the output is transliterated data of the new language. The new language increases the diversity of the robust model, thereby improving accuracy and training.

[0018] According to an embodiment, an automatic speech recognition system configured for multilingual acoustic models with transliteration-based data augmentation includes a processor and a memory coupled to the processor. The memory stores instructions to cause the processor to perform actions including: training a multilingual network on a training language set using training data from the original transcription to construct a baseline multilingual acoustic model; performing transliteration by processing multiple multilingual data types from the language set via the multilingual network; and outputting a transliterated data pool. A filtering metric is applied to the transliterated data pool output from the multilingual network to select one or more portions of the transliterated data for retraining the acoustic model. Data augmentation is performed by adding one or more selected portions of the output transliterated data back to the original transcribed training data to obtain updated training data. The updated training data is processed by the multilingual network, and the multilingual acoustic model is retrained with the updated training data. This configuration provides more accurate training of the acoustic model, at least due to the transliteration and data augmentation operations and the data augmentation performed on the output transliterated data.

[0019] In one embodiment, the multilingual network includes a neural network comprising multiple language-specific output layers configured to model the symbol set of each language, and wherein the neural network is configured to output language-specific portions of transliterated data to at least one corresponding language-specific output layer. The use of neural networks provides faster and more efficient training of acoustic models.

[0020] According to an embodiment, a non-transitory computer-readable storage medium tangibly embodies computer-readable program code having computer-readable instructions, which, when executed, cause a computer device to perform a method for constructing a multilingual acoustic model for automatic speech recognition in a low-resource setting. The method includes training a multilingual model with end-to-end recurrent layers on a pooled dataset extracted from multiple languages.

[0021] Training data transcribed from a utterance is transliterated by forwarding the acoustic features of each utterance through a trained network. Untranscribed training data belonging to the language used to train the network is also transliterated by forwarding it through the network. A filtering metric is applied, which includes at least one of the following: a count of the number of symbols in the transliterated output of the utterance, or a ratio of the number of symbols in the transliterated language to the number of symbols in the reference transcript. Data augmentation is performed by adding the transliterated data to the language's training pool and training a new multilingual model.

[0022] These and other features will become apparent from the following detailed description of illustrative embodiments of the invention, which is read in conjunction with the accompanying drawings. Attached Figure Description

[0023] The accompanying drawings are illustrative embodiments. They do not show all embodiments. Other embodiments may be used in addition to or as alternatives. Details that may be obvious or unnecessary may be omitted to save space or for more efficient illustration. Some embodiments may be practiced with additional components or steps and / or without all components or steps shown. When the same numbers appear in different drawings, they refer to the same or similar components or steps.

[0024] Figure 1 An overview of the architecture of a multilingual network, consistent with the illustrative embodiments, is provided.

[0025] Figure 2 The examples shown are consistent with the illustrative embodiments and include various types of training data and filtering metrics. Figure 1 A multilingual network.

[0026] Figure 3 The acoustic model performance is shown to be consistent with the illustrative embodiment.

[0027] Figure 4 The acoustic model performance is shown to be consistent with the illustrative embodiment.

[0028] Figure 5 The model performance, consistent with the exemplary embodiments, is shown in terms of word error rate (WER%) after porting the model to include the new language.

[0029] Figure 6This is a flowchart illustrating a computer-implemented method for creating a multilingual acoustic model for ASR, consistent with the illustrated embodiment.

[0030] Figure 7 This is a functional block diagram illustration of a computer hardware platform that can communicate with various networked components of a system used for seismic imaging, consistent with the illustrative embodiments.

[0031] Figure 8 An illustrative cloud computing environment consistent with the illustrative embodiments is described.

[0032] Figure 9 A set of functional abstraction layers provided by a cloud computing environment, consistent with the illustrative embodiments, is described. Detailed Implementation

[0033] Overview

[0034] In the following detailed description, numerous specific details are illustrated with examples to provide a thorough understanding of the relevant teachings. However, it should be understood that these teachings can be practiced without these details. In other instances, well-known methods, procedures, components, and / or circuits have been described at a relatively high level without detail to avoid unnecessarily obscuring aspects of these teachings.

[0035] While the illustrative embodiments of this disclosure relate to building multilingual acoustic models for raw training data in low-resource settings, computer-implemented methods and systems are not limited to such conditions. A low-resource setting is, for example, a setting where only a few tens of hours of training data are available. It should be understood that a “low-resource language” is a language in which there are not sufficient language resources to train an automatic speech recognition system to operate at an acceptable word error rate. In situations where insufficient training data is available, there are multiple dialects of some languages. Although this disclosure addresses how to overcome the problems of training automatic speech recognition systems, these teachings are also applicable and offer benefits to higher-resource settings that may have thousands of hours of available training data, compared to the tens of hours available for low-resource settings.

[0036] The computer-implemented methods and apparatus of this disclosure provide improvements in the accuracy of training and implementation of acoustic models used in devices such as Automatic Speech Recognition (ASR) systems, especially in low-resource settings where training data may be insufficient to train the acoustic model to perform with satisfactory accuracy. Additionally, the computer-implemented methods and apparatus of this disclosure provide improvements in the efficiency of computer operation. For example, based on the teachings herein, the technological improvements result in a reduction in the amount of processing power required to train and operate acoustic models in ASR systems, as fewer epochs, iterations, and less time are used to obtain a model that provides accurate results. The need for reduced and / or eliminated human interaction to correct errors in the operation of the acoustic model is reduced, making computer operation more efficient. The reduction in epochs and iterations also provides for a reduction in computer memory usage. Using the teachings of this disclosure, time savings are achieved in generating more accurate acoustic models.

[0037] Example Architecture

[0038] Figure 1 An overview of an example architecture 100 for a multilingual network consistent with the illustrative embodiment is provided. In this illustrative embodiment, the multilingual network 105 includes an input feature layer 110, multiple language-independent recurrent layers 115, and language-specific output layers 120a, 120b, and 120c. Transcribed training data 135 is also shown.

[0039] Input feature layer 110 receives raw transcribed data. The raw transcribed data is a representation of spoken or produced sounds (such as utterances). In an automatic speech recognition system, utterances are transcribed into symbols associated with the language of the utterance. For example, English utterances are transcribed into English, Japanese utterances into Japanese, etc. The transcribed data is pushed (e.g., processed) by a multilingual network 105. The multilingual network can be a neural network, and as shown in this illustrative embodiment, the layers of the neural network perform various functions of language processing. Shared, independent recurrent layers process the raw transcribed data by performing operations such as transliteration. The transliteration function uses symbols from other languages ​​to represent the utterance. Because human sounds share a degree of commonality, English utterances can be transliterated into symbols from different languages ​​(e.g., Hindi, Mongolian, Javanese, Spanish, Mandarin) to create a transliteration data pool. The shared language-related recurrent layer 115 provides dated transliterations to language-specific output layers 120a, 120b, 120c, etc. In this illustrative embodiment, output layer 120 outputs the raw transcribed data, while output layers 120a and 120c are transliteration output layers, meaning that there is a language-specific output of transliterated data. The training data 135 for transcription includes data that has undergone additional processing before being added back to the layers of the multilingual network for further processing.

[0040] Figure 2 The examples shown are consistent with the illustrative embodiments and include various types of training data and filtering metrics. Figure 1 An overview of multilingual networks 200. For example, Figure 2 Different types of input data that can be provided to the input feature layer 210 are shown. Training data 205, which may represent one or more utterances in linguistic symbols, is shown as one of the inputs to the input feature layer 210. Additional data, such as first untranscribed data 207 in the same language as the training data of the utterance transcription and second untranscribed data 209 in a different language than the utterance transcription data, are shown. One or both of the untranscribed data can be input together with the original transcribed data to improve the training of the multilingual acoustic model in terms of accuracy and capability in training the acoustic model used in the automatic speech recognition system.

[0041] Now, referring to the above... Figure 1 and Figure 2 The description, Figure 2 The language-specific output layers 220a, 220b, and 220c shown provide output pools of transliterated data. These output data pools have a filtering metric 225, which is applied to select data that will increase the accuracy of the acoustic model's training. Several functions exist that can be used as the filtering metric 225.

[0042] One such filtering method, according to filtering metric 225, involves performing symbol counting on the transliterated data for each language-specific output and selecting the group with the highest or relatively high symbol count (relative to languages ​​with lower symbol counts). Higher symbol counts can provide selection of transliterated data that enables more accurate training of the multilingual acoustic model.

[0043] Another filtering function that can be performed according to filtering metric 225 is to compare the ratio of symbols in the transliterated data to the ratio of symbols in the original transcribed data of the discourse, and select one or more portions of the output transliterated data that have a higher ratio of symbols.

[0044] Multilingual data augmentation can then be performed by adding one or more selected portions of the transliterated output data 230 to the original transcribed training data 235. Figure 2The languages ​​of three language-specific output layers are identified, where 235a and 235c are the languages ​​output by the corresponding layers 220a and 220c and have undergone filtering metrics and data augmentation. The original transcribed data 235b, after data augmentation, is the output of the language-specific layer 220b after additional processing. The augmented data 235a, 235b, and 235c are then fed back into the multilingual network for further processing. Multiple iterations can be performed to create training data for the multilingual acoustic model, thereby improving the accuracy of the acoustic model. In low-resource settings, the teachings of this disclosure provide more accurate training of the multilingual acoustic model while conserving computational resources.

[0045] Figure 3 Table 305 shows the acoustic model performance consistent with the illustrative embodiments. Figure 3 Experiments with low-resource languages ​​are identified (labeled as conditions 401, 402, 403, and 404). These monolingual models include Mongolian, Javanese, Duro, and Georgian, respectively. The label "C1" indicates the use of all transliterated data, while "D1" indicates data selected using a filtering metric, "E1" indicates 50 hours of data using the filtering metric, and "F1" indicates 100 hours of data using the filtering metric. It can be seen that by using transliterated training data, the percentage of word error rate decreased for each language. With increasing resources, each language showed an improvement in the percentage of word error rate.

[0046] Figure 4 The acoustic model performance is shown to be consistent with the illustrative embodiment. Figure 3 Similarly, baseline performance 405 and model performance 415 using untranscribed training data are shown. Model performance using untranscribed cross-lingual data 420 is also shown.

[0047] Baseline performance 405 includes both monolingual and multilingual conditions. Model performance using untranscribed training data 415 includes semi-supervised (ST) transcripts. For example, a multilingual network can be used to transcribe untranscribed data. When the untranscribed data corresponds to data used to train the multilingual network, the network can generate transcripts for that data. Since the labels are automatically generated without human annotation, it is called semi-supervised data. Additionally, model performance 415 includes transliterated (TL) language symbols. Regarding model performance using untranscribed cross-lingual data 420, "CSR1" includes the languages ​​Cebuano, Kazakh, Telegu, and Lithuanian, and "CSR2" includes Pashto, Paraguyan Guarani, Igbo, and Amharic.

[0048] Figure 5 The model performance (WER%) after porting the model to include the new language is shown, consistent with the exemplary embodiments. Figure 5 The model has been ported to include a new language (Italian). It can be seen that performance improves with increasing training time.

[0049] Further information Figure 5 Italian is the new language being trained. For example, A4 is the case when the network is trained only on Italian—the weights of the network are randomly initialized. In B4, the network is again trained only on Italian, but this time it is initialized to form a multilingual network that has already been trained on many languages ​​(but not yet on Italian). In this case, the network is better initialized, so B4 is better than A4. C4 is the case when training a multilingual system. Italian is now one of the multilingual languages, so the trained model can be used to process Italian. Due to multilingual training, C4 is better than either A4 or B4. D4 is the case of training a better multilingual network. The model is now trained using additional transliterated data.

[0050] It should be understood that acoustic models trained for ASR are typically categorized into two types—hybrid acoustic models or end-to-end acoustic models. The teachings of this disclosure can be applied to both types. Therefore, while Connectionist Temporal Classification (CTC) is a training method that can be used in end-to-end acoustic model training, this disclosure is not limited to end-to-end acoustic model training.

[0051] Example process

[0052] Having seen the foregoing overview of the example architecture, it may now be helpful to consider a more advanced discussion of the example process. To this end, in conjunction with... Figures 1-5 , Figure 6 A flowchart 600, consistent with illustrative embodiments, is described, illustrating a seismic imaging operation including configuration and execution phases. Process 600 is shown as a collection of boxes in a logic flowchart, representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, these boxes represent computer-executable instructions that, when executed by one or more processors, perform the described operations. Typically, computer-executable instructions may include routines, programs, objects, components, data structures, etc., that perform functions or implement abstract data types. In each process, the order in which operations are described is not intended to be construed as limiting, and any number of described boxes may be combined and / or executed in parallel in any order to implement the process.

[0053] Now for reference Figure 6At operation 605, the multilingual network is trained on a training language set using training data from the original transcription. This training data from the original transcription is, for example, utterances transcribed into its native language. For example, English symbols and graphemes are used to transcribe English utterances. However, it should be understood that other symbols can be used. Phonemes, subwords, whole words, etc., can also be used in place of or alongside graphemes. In this example, the original transcription setup is a low-resource setup, meaning that tens of hours of training data exist (as opposed to hundreds or thousands of hours of training data in typical resource settings).

[0054] At operation 610, transliteration is performed by processing multiple multilingual data sets. In transliteration, utterances in one language are represented by symbols in a language different from the language in which the utterances are made. In low-resource settings, the transliteration data pool can be output from a multilingual network.

[0055] At operation 615, a filtering metric is applied to the pool of transliterated data output from the multilingual network to select one or more portions of the transliterated data for retraining the acoustic model. These portions are selected to retrain the acoustic model to improve accuracy (i.e., reduce the word error rate) and to make the acoustic model more robust by increasing the amount of training data. An example of applying the filtering metric is selecting one or more portions of the transliterated data that have a relatively higher symbol count compared to the rest of the transliterated data. For example, in the case of utterances, a higher symbol count can more accurately define the utterance.

[0056] In another example, a filtering metric is applied to the output pool of transliterated data by comparing the ratio of symbols in the transliterated data to the ratio of symbols in the utterance of training data including the original transcription, and selecting one or more portions of the output transliterated data with a higher ratio of symbols. It should be understood that, in one embodiment, if multiple portions are available for selection, the highest ratio may be ordered (e.g., first highest, second highest, third highest, etc.).

[0057] At operation 620, data augmentation is performed by adding one or more selected portions of the transliterated data, chosen via an applied filtering metric, back to the original transcribed data. Data augmentation can provide, for example, multiple representations of the utterance, both in the language of the original transcribed data and in augmented data that can represent the utterance using symbols from languages ​​different from the language of the utterance. For example, the utterance could be in English, but the augmented data could use symbols from Hindi to represent it. Therefore, the augmented data can be considered updated training data.

[0058] At operation 625, the multilingual network trains a new multilingual acoustic model using updated training data. This process ends after operation 625. However, it should be understood that multiple epochs can be performed to increase the accuracy and robustness of the multilingual acoustic model. New languages ​​can be added and additional transliteration data can be generated to further enhance the accuracy and robustness of the multilingual acoustic model. In this disclosure, the baseline multilingual acoustic model can also be retrained using updated training data.

[0059] Figure 7 A functional block diagram of a computer hardware platform is provided (700). Specifically, Figure 7 A specially configured network or host computer platform 700 is shown, which can be used to implement Figure 6 The method shown.

[0060] Computer platform 700 may include a central processing unit (CPU) 704, a hard disk drive (HDD) 706, random access memory (RAM) and / or read-only memory (ROM) 708, a keyboard 710, a mouse 712, a display 714, and a communication interface 716, which are connected to a system bus 702. HDD 706 may include data storage.

[0061] In one embodiment, HDD 706 has the capability to include a stored program that can perform various processes, such as Automatic Speech Recognition 705 (ASR) operations, in a manner described herein. ASR module 705 provides management of the Automatic Speech Recognition process, such as for training a multilingual acoustic model 742. Network processing module 710 processes training data in a low-resource setting to create a baseline multilingual acoustic module. Transliteration module 720 uses symbols from at least one language to represent utterances spoken in another language, and network processing module 720 outputs a pool of transliterated data. Filtering metric module 730 is configured for application to the output pool of transliterated data and is also configured to select one or more portions of the transliterated data for retraining acoustic model 742. Data augmentation module 735 is configured to augment training data by adding one or more selected portions of the output transliterated data back to the original transcribed data to retrain acoustic model 742. Training sample module 740 stores multilingual data to train the network processor module.

[0062] Example cloud platform

[0063] As described above, the functionality associated with environmental and ecological optimization methods may include the cloud. It should be understood that although this disclosure includes a detailed description of cloud computing as discussed below, implementation of the teachings set forth herein is not limited to a cloud computing environment. Rather, embodiments of this disclosure can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0064] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0065] The features are as follows:

[0066] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring manual interaction with the service provider.

[0067] Wide Area Network (WAN) Access: Capabilities are available on the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0068] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. Location independence has significance because consumers typically do not control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0069] Rapid Flexibility: In some cases, the ability to scale outwards and inwards quickly and flexibly can be provided. For consumers, the available capacity often appears unlimited and can be purchased in any quantity at any time.

[0070] Measurement services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both the providers and consumers of the services being utilized.

[0071] The service model is as follows:

[0072] Software as a Service (SaaS): The capability offered to consumers is the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from various client devices through a thin client interface such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions of limited user-specific application configuration settings.

[0073] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer onto cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of possible application hosting environments.

[0074] Infrastructure as a Service (IaaS): This provides consumers with the capability to deliver processing, storage, networking, and other basic computing resources that enable them to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0075] The deployment model is as follows:

[0076] Private cloud: Cloud infrastructure operated solely by an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0077] Community cloud: Cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0078] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud services.

[0079] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported together (e.g., cloud bursting for load balancing between clouds).

[0080] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure of a network of interconnected nodes.

[0081] Now for reference Figure 8This paper depicts an illustrative cloud computing environment 800 utilizing cloud computing. As shown, the cloud computing environment 800 includes a cloud 850 with one or more cloud computing nodes 810, and local computing devices used by cloud consumers can communicate with the cloud computing nodes. These local computing devices are, for example, personal digital assistants (PDAs) or cellular phones 854A, desktop computers 854B, laptop computers 854C, and / or automotive computer systems 854N. Nodes 810 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 800 to provide cloud consumers with infrastructure, platforms, and / or software-as-a-service that eliminates the need for them to maintain resources on their local computing devices. It should be understood that... Figure 8 The types of computing devices 854A-N shown are for illustrative purposes only, and computing node 810 and cloud computing environment 850 can communicate with any type of computing device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0082] Now for reference Figure 9 This demonstrates the 800 (cloud computing environment) Figure 8 The 900 provides a set of functional abstraction layers. It should be understood beforehand that... Figure 9 The components, layers, and functions shown are for illustrative purposes only, and embodiments of this disclosure are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0083] The hardware and software layer 960 includes hardware and software components. Examples of hardware components include: a mainframe 961; a RISC (Reduced Instruction Set Computer) based server 962; a server 963; a blade server 964; a storage device 965; and a network and networking component 966. In some embodiments, the software components include network application server software 967 and database software 968.

[0084] The virtualization layer 970 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 971; virtual storage 972; virtual networks 973, including virtual private networks; virtual applications and operating systems 974; and virtual clients 975.

[0085] In one example, management layer 980 can provide the functionality described below. Resource provisioning 981 provides dynamic procurement of computing resources and other resources utilized to perform tasks within the cloud computing environment. Metering and pricing 982 provides cost tracking for resource utilization within the cloud computing environment, as well as invoicing or issuing invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, and protection for data and other resources. User portal 983 provides access to the cloud computing environment for consumers and system administrators. Service level management 984 provides cloud resource allocation and management to ensure the required service level is met. Service level agreement (SLA) planning and fulfillment 985 provides pre-scheduling and procurement of cloud resources, where future needs are anticipated according to the SLA.

[0086] Workload layer 990 provides examples of functionalities that can be leveraged in a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 991; software development and lifecycle management 992; virtual classroom education delivery 993; data analytics and processing 994; transaction processing 995; and an ASR module 996 for performing seismic imaging on a hybrid cloud-based platform, as discussed herein.

[0087] in conclusion

[0088] Various embodiments of the present teachings have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technology on the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0089] While the content and / or other examples considered to be in their best form have been described above, it should be understood that various modifications may be made therein, and the subject matter disclosed herein can be implemented in various forms and examples, and the teachings can be applied to many applications, of which only a few have been described herein. The appended claims are intended to claim protection for any and all applications, modifications, and variations that fall within the true scope of this teaching.

[0090] The components, steps, features, purposes, benefits, and advantages discussed herein are merely illustrative. None of them, or the discussion relating to them, is intended to limit the scope of protection. While various advantages have been discussed herein, it will be understood that not all embodiments are necessarily intended to include all advantages. Unless otherwise stated, all measurements, values, scores, locations, sizes, dimensions, and other specifications set forth in this specification, including in the appended claims, are approximate and not precise. They are intended to have a reasonable scope consistent with the functionality associated with them and with the conventions of the art to which they pertain.

[0091] Many other embodiments have also been considered. These embodiments include those with fewer, additional, and / or different components, steps, features, purposes, benefits, and advantages. These embodiments also include those in which components and / or steps are arranged and / or ordered in different ways.

[0092] The flowcharts and diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of various implementations according to this disclosure.

[0093] While the foregoing has been described in conjunction with exemplary embodiments, it should be understood that the term "exemplary" means only as an example, and not the best or optimal. Apart from what has just been stated above, whether or not it is stated in the claims, nothing stated or described is intended or should be construed as causing any component, step, feature, purpose, benefit, advantage, or equivalent to be made public.

[0094] It should be understood that, unless otherwise specified herein, the terms and expressions used herein have the general meaning consistent with those in the respective fields of investigation and research to which they pertain. Relational terms such as "first" and "second" may be used merely to distinguish one entity or action from another, without necessarily requiring or implying any such actual relationship or order between these entities or actions. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but may also include other elements not expressly listed or inherent to such a process, method, article, or apparatus. Without further constraints, an element preceded by "a" or "an" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes that element.

[0095] This summary of disclosure is provided to allow the reader to quickly determine the nature of the technical disclosure. It should be understood that it is not intended to interpret or limit the scope or meaning of the claims. Furthermore, in the foregoing detailed description, it can be seen that various features are grouped together in various embodiments for the purpose of fluent disclosure. This method of disclosure should not be construed as reflecting an intention that the claimed embodiments have more features than those expressly recited in each claim. Rather, as reflected in the appended claims, the inventive subject matter lies in fewer than all features of a single disclosed embodiment. Therefore, the following claims are thereby incorporated into the detailed description, wherein each claim is independently claimed as a separate subject matter.

Claims

1. A computer-implemented method of building a multilingual acoustic model for automatic speech recognition in a low-resource setting, the method comprising: training a multilingual network on a training language set with originally transcribed training data to create a baseline multilingual acoustic model; performing transliteration by processing a plurality of multilingual data types from the language set through the multilingual network and outputting a pool of transliterated data; applying a filtering metric to the pool of transliterated data output from the multilingual network to select one or more portions of the transliterated data for retraining of the acoustic model, wherein applying the filtering metric to the pool of transliterated data is performed by selecting the one or more portions of output transliterated data having a relatively higher symbol count compared to remaining portions of the transliterated data; performing data augmentation by adding the one or more selected portions of the pool of transliterated data back to the originally transcribed training data to obtain updated training data; and training a new multilingual acoustic model through the multilingual network using the updated training data.

2. The computer-implemented method of claim 1, further comprising: retraining the baseline multilingual acoustic model with the updated training data.

3. The computer-implemented method of claim 1, wherein: the originally transcribed training data is from a low-resource language; the multilingual network comprises a neural network comprising a plurality of language-specific output layers configured to model a symbol set for each language respectively; and the neural network outputs language-specific portions of the transliterated data to at least one respective language-specific output layer.

4. The computer-implemented method of claim 3, wherein the adding the one or more selected portions of the pool of transliterated data back to the originally transcribed training comprises relabeled data comprising new copies of data using symbols of other languages.

5. The computer-implemented method of claim 3, wherein the training of the multilingual network on a training language set is performed with the low-resource language of the originally transcribed training data comprising tens of hours of the originally transcribed data.

6. The computer-implemented method of claim 3, further comprising generating semi-supervised labels in response to processing untranscribed data by the neural network.

7. The computer-implemented method of claim 1, wherein the processing of the plurality of multilingual data types comprises processing transcribed training data, untranscribed data from the same training language set, and untranscribed data from different languages.

8. The computer-implemented method of claim 1, further comprising: adding a new language to the multilingual network; and outputting transliterated data for the new language.

9. A computer-implemented method of building a multilingual acoustic model for automatic speech recognition in a low-resource setting, the method comprising: training a multilingual network on a training language set with originally transcribed training data to create a baseline multilingual acoustic model; ​ ​ transliteration is performed by processing a plurality of multilingual data types from the language set via the multilingual network and outputting a pool of transliterated data; applying a filtering metric to the pool of transliterated data output from the multilingual network to select one or more portions of the transliterated data for retraining of the acoustic model, wherein applying the filtering metric to the pool of transliterated data is performed by comparing a ratio of symbols in the transliterated data to symbols in utterances of training data comprising the original transcription and selecting one or more portions of output transliterated data having a higher ratio of symbols; performing data augmentation by adding the one or more selected portions of the pool of transliterated data back to the training data of the original transcription to obtain updated training data; and training a new multilingual acoustic model using the updated training data via the multilingual network.

10. An automatic speech recognition system configured for transliteration-based data augmentation of a multilingual acoustic model in a low-resource setting, the system comprising: a processor; a memory coupled to the processor, the memory storing instructions to cause the processor or perform actions, the actions comprising: training a multilingual network on a training language set with training data of an original transcription to create a baseline multilingual acoustic model; performing transliteration by processing a plurality of multilingual data types from the language set via the multilingual network and outputting a pool of transliterated data; applying a filtering metric to the pool of transliterated data output from the multilingual network to select one or more portions of the transliterated data for retraining of the acoustic model, wherein applying the filtering metric to the pool of transliterated data is performed by selecting the one or more portions of output transliterated data having a relatively higher count of symbols compared to a remaining portion of the transliterated data; performing data augmentation by adding the one or more selected portions of the output transliterated data back to the training data of the original transcription to obtain updated training data; and training a new multilingual acoustic model using the updated training data.

11. The system of claim 10, wherein the instructions cause the processor to perform additional actions, the additional actions comprising: retraining the baseline multilingual acoustic model with the updated training data.

12. The system of claim 10, wherein: the multilingual network comprises a neural network comprising a plurality of language-specific output layers configured to model a symbol set for each language respectively; and the neural network is configured to output language-specific portions of the transliterated data to at least one respective language-specific output layer.

13. The system of claim 10, wherein the processing of the plurality of multilingual data types comprises processing transcribed training data, untranscribed data from the same training language set, and untranscribed data from a different language.

14. The system of claim 13, wherein the instructions cause the processor to perform additional actions, the additional actions comprising: ​ ​ adding a new language to the multilingual network: and outputting transliteration data in the new language.

15. An automatic speech recognition system configured for transliteration-based data augmentation of a multilingual acoustic model in a low-resource setting, the system comprising: a processor; a memory coupled to the processor, the memory storing instructions to cause the processor or perform actions, the actions comprising: training a multilingual network on a training language set with originally transcribed training data to create a baseline multilingual acoustic model; performing transliteration by processing a plurality of multilingual data types from the language set through the multilingual network and outputting a pool of transliterated data; applying a filtering metric to the pool of transliterated data output from the multilingual network to select one or more portions of the transliterated data for retraining of the acoustic model, wherein applying the filtering metric to the pool of transliterated data is performed by comparing a ratio of symbols in the transliterated data to symbols in utterances comprising the originally transcribed training data and selecting one or more portions of output transliterated data having a higher ratio of symbols; performing data augmentation by adding the one or more selected portions of the output transliterated data back to the originally transcribed training data to obtain updated training data; and training a new multilingual acoustic model using the updated training data.

16. A non-transitory computer-readable storage medium tangibly embodying computer-readable program code having computer-readable instructions that, when executed, cause a computer device to perform a method of constructing a multilingual acoustic model for automatic speech recognition in a low-resource setting, the method comprising: training a multilingual model with end-to-end recurrent layers on a pooled dataset of extracted data from multiple languages; transliterating transcribed training data by forward passing acoustic features of each utterance through the trained network; transliterating untranscribed training data by forward passing data belonging to a language used to train the network through the network; applying a filtering metric comprising at least one of a count of symbols in a transliterated output of an utterance or a ratio of a count of symbols in a transliterated language to a count of symbols of a reference transcript; and performing data augmentation by adding transliterated data to the language training pool and training a new multilingual model. training a baseline of the multilingual model for a fixed number of epochs in a first operation.

17. The computer-readable storage medium of claim 16, wherein, performing the data augmentation includes using unsupervised data and performing the transliteration across the language training pool to improve an acoustic model in a low-resource setting.

18. The computer-readable storage medium of claim 16, wherein, the filtering metric is applied to the pool of transliterated data output from the multilingual network to select one or more portions of the transliterated data for retraining of the acoustic model, wherein applying the filtering metric to the pool of transliterated data is performed by comparing a ratio of symbols in the transliterated data to symbols in utterances comprising the originally transcribed training data and selecting one or more portions of output transliterated data having a higher ratio of symbols;

Images