Training data sequence for RNN-T based global English model

Abstract

Training Data Sequence for RNN-T-Based Global English Model A computer-implemented method for preparing training data for a speech recognition model is provided, comprising: obtaining multiple speech datasets, each speech dataset having different acoustic features; and sorting sentences from the multiple speech datasets such that similar sentences from different speech datasets are located closely together while imposing a weak constraint on the length of the speech to train the speech recognition model.

Description

Technical Field

[0001] The present invention generally relates to machine learning, and more specifically, to a method and system for constructing an efficient training data sequence for a global English model based on a Recurrent Neural Network Transducer (RNN-T).

Background Art

[0002] End-to-end models for Automatic Speech Recognition (ASR) have gained popularity in recent years as a way to fold the separate components of traditional ASR systems (e.g., acoustic, pronunciation, and language models) into a single neural network. Examples of such models include Connectionist Temporal Classification (CTC)-based models, Recurrent Neural Network Transducer (RNN-T), and attention-based seq2seq models. Among these models, RNN-T is the most suitable streaming end-to-end recognizer, which has shown competitive performance compared to conventional systems.

[0003] Before delving into RNN-T, it's important to understand that speech recognition continues to evolve to meet the agile demands of mobile environments without constraints. New speech recognition architectures or improvements to existing ones are constantly being developed with the aim of improving the quality of ASR systems. To explain, speech recognition initially employed multiple models, each with its own specific purpose. For example, an ASR system might have included an Acoustic Model (AM), a Pronunciation Model (PM), and a Language Model (LM). The Acoustic Model mapped speech segments (e.g., speech frames) to phonemes. The Pronunciation Model concatenated these phonemes to form words, while the Language Model was used to represent the likelihood of a given phrase (e.g., the probability of a sequence of words). However, although these individual models worked together, each model was trained independently and often manually designed on different datasets.

[0004] While the separate model approach allowed speech recognition systems to be fairly accurate, especially when the training corpus (e.g., the set of training data) for a given model corresponded to the model's effectiveness, the need to train each separate model independently introduced its own complexity, leading to architectures with integrated models. These integrated models aimed to directly map speech waveforms (e.g., input sequences) to output sentences (e.g., output sequences) using a single neural network. This resulted in a sequence-to-sequence approach that generates a sequence of words (or graphemes) given a sequence of speech features. Examples of sequence-to-sequence models include attention-based models and listen-attend-spell (LAS) models. LAS models use a listener component, an attendant component, and a speller component to transcribe spoken utterances into text. Here, the listener is a recurrent neural network (RNN) encoder that receives speech input (e.g., a time-frequency representation of the speech input) and maps it to higher-level feature representations. The attendant learns the alignment between input features and predicted subword units (e.g., graphemes or word fragments). Therefore, we pay attention to higher-level features. Speller is an attention-based RNN decoder that generates character sequences from input by producing a probability distribution for a given set of words. Its integrated structure allows all components of the model to be trained together as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose architecture is built entirely of neural networks. A complete neural network functions without external and / or manually designed components (e.g., finite-state transducers, vocabulary, or text normalization modules).In addition, when training E2E models, these models generally do not require bootstrapping from decision trees or time alignment from a separate system.

[0005] Early E2E models were accurate and proved to be superior to individually trained models, but these E2E models, such as LAS models, worked by reviewing the entire input sequence before generating output text, thus not enabling streaming of output as input was received. Without streaming capability, LAS models cannot perform real-time voice transcription. Due to this flaw, deploying LAS models for speech applications that are highly latency-sensitive or require real-time voice transcription can cause problems.

[0006] In addition, speech recognition systems that have acoustic, pronunciation, and language models, or that are configured with such models together, may rely on decoders that must search a relatively large search graph associated with these models. A large search graph does not justify hosting this type of speech recognition system entirely on-device. Here, when a speech recognition system is hosted "on-device," the device that receives the speech input uses its processor to perform the functions of the speech recognition system. For example, if a speech recognition system is hosted entirely on-device, the device's processor does not need to interact with any off-device computing resources to perform the functions of the speech recognition system. A device that performs speech recognition that is not entirely on-device relies on remote computing (e.g., remote computing systems or cloud computing), and thus on-device connectivity, to perform at least some of the functions of the speech recognition system. For example, a speech recognition system performs decoding on a large search graph using a network connection to a server-based model.

[0007] Unfortunately, by relying on remote connections, speech recognition systems are vulnerable to latency issues and / or the inherent unreliability of communication networks. To improve the usefulness of speech recognition by circumventing these problems, speech recognition systems have once again evolved into a sequence-to-sequence model known as the recurrent neural network transducer (RNN-T). Unlike other sequence-to-sequence models that do not employ an attention mechanism and generally need to process the entire sequence (e.g., speech waveform) to produce an output (e.g., a sentence), RNN-T continuously processes input samples to stream output symbols, a feature particularly attractive for real-time communication. For example, speech recognition with RNN-T can output characters one by one as spoken.

[0008] Therefore, there is a need for a more efficient process for training data sequencing using RNN-T. [Overview of the project]

[0009] According to one embodiment, a computer implementation method is provided for preparing training data for a speech recognition model. The computer implementation method comprises the steps of acquiring multiple speech datasets, each speech dataset having different acoustic features, and sorting sentences from the multiple speech datasets in order to train a speech recognition model, while imposing loose constraints on speech length, so that similar sentences from different speech datasets are placed close together.

[0010] According to another embodiment, a computer program product is provided for preparing training data for a speech recognition model. The computer program product comprises a computer-readable storage medium in which program instructions are embodied, and the program instructions are executable by the computer to cause the computer to acquire multiple speech datasets, each speech dataset having different acoustic features, and to sort sentences from the multiple speech datasets so that similar sentences from different speech datasets are placed close together, while imposing loose constraints on speech length, in order to train a speech recognition model.

[0011] In yet another embodiment, a system is provided for preparing training data for a speech recognition model. The system comprises memory and one or more processors communicating with the memory, the one or more processors being configured to acquire multiple speech datasets, each speech dataset having different acoustic features, and to sort sentences from the multiple speech datasets so that similar sentences from different speech datasets are placed close together, while imposing loose constraints on speech length, in order to train a speech recognition model.

[0012] According to another embodiment, a computer implementation method for preparing training data for a speech recognition model is provided. The computer implementation method comprises the steps of: acquiring multiple speech datasets, each speech dataset having different acoustic features; sorting sentences from the multiple speech datasets so that similar sentences from different speech datasets are placed close together in order to train a speech recognition model; and grouping the similar sentences from different speech datasets into minibatches, where each minibatch contains pairs of sentences from different English dialects.

[0013] In yet another embodiment, a computer program product is provided for preparing training data for a speech recognition model. The computer program product comprises a computer-readable storage medium in which program instructions are embodied, and the program instructions cause the computer to acquire multiple speech datasets, each having different acoustic features, and to sort sentences from the multiple speech datasets so that similar sentences from different speech datasets are placed close together in order to train a speech recognition model, and to group the similar sentences from different speech datasets into minibatches, where each minibatch contains pairs of sentences from different English dialects, and is therefore executable by the computer.

[0014] In a preferred embodiment, multiple speech datasets, each having different acoustic features, are sampled from a data pool, thereby the sampled speech datasets containing multiple sets of similar sentences.

[0015] In another preferred embodiment, score penalties are presented to control various sentences.

[0016] In yet another preferred embodiment, the similar sentence is a similar sentence having a different dialect of the target language.

[0017] In yet another preferred embodiment, the speech recognition model is a global speech recognition model for the target language.

[0018] In yet another preferred embodiment, similar sentences from different audio datasets are grouped into minibatches.

[0019] In yet another preferred embodiment, each minibatch of the minibatches contains a pair of sentences between different English dialects.

[0020] In yet another preferred embodiment, each minibatch of the minibatches contains a similar amount of dialect data.

[0021] In yet another preferred embodiment, the similarity between different English dialects of similar sentences from different speech datasets is given by the following formula.

[0022]

Number

[0023] In yet another preferred embodiment, the similarity score-dependent penalty is given by the following formula.

[0024] P(d)=γe κd -γ(d>0) Here, γ and κ are hyperparameters.

[0025] It should be noted that the exemplary embodiments are described with reference to different subjects. In particular, some embodiments are described with reference to method-type claims, while other embodiments are described with reference to apparatus-type claims. However, those skilled in the art will infer that, unless otherwise notified, any combination of features belonging to one type of subject, in addition to any combination of features between different subjects, particularly between the features of method-type claims and the features of apparatus-type claims, is also considered to be described herein.

[0026] These features and advantages, as well as other features and advantages, will become apparent from the following detailed description of their exemplary embodiments. This detailed description should be read in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0027] The present invention provides details in the following description of preferred embodiments with reference to the following drawings. [Figure 1]This is a block / flow diagram of an exemplary system for organizing training data sequences based on the metric that similar sentences with different dialects are placed in close proximity, with weak constraints on speech length, for building a global English model (GEM) according to an embodiment of the present invention. [Figure 2] This is a block / flow diagram of an exemplary method for organizing a training data sequence based on the metric that similar sentences with different dialects are placed in close proximity, with weak constraints on speech length, for building a Global English Model (GEM) according to an embodiment of the present invention. [Figure 3] This is a block / flow diagram of an exemplary method for preparing training data for a speech recognition model according to an embodiment of the present invention. [Figure 4] This invention illustrates exemplary data sorting by employing an exemplary method, in contrast to conventional methods, according to embodiments of the present invention. [Figure 5] This invention presents a system for sorting sentences from multiple audio datasets according to an embodiment of the present invention. [Figure 6] This is a block / flow diagram of an exemplary processing system for organizing training data sequences based on the metric that similar sentences with different dialects are placed in close proximity, with weak constraints on speech length, for building a Global English Model (GEM) according to an embodiment of the present invention. [Figure 7] This is a block / flow diagram of an exemplary cloud computing environment according to an embodiment of the present invention. [Figure 8] This is a schematic diagram of an exemplary abstraction model layer according to an embodiment of the present invention.

[0028] Throughout the drawing, identical or similar reference numerals represent identical or similar elements. [Modes for carrying out the invention]

[0029] Embodiments of the present invention provide a method and device for constructing an efficient training data sequence for a recurrent neural network transducer (RNN-T) based global English model. RNN-T models are typically trained with an RNN-T loss aimed at improving the log-likelihood of the training data. However, there is little research work investigating sequential training criteria for RNN-T models.

[0030] Currently, RNN-T models are built separately for each language. Even in the case of English, because there are strong dialects (accents) in each English-speaking country, multiple models have been independently created to achieve sufficient performance for practical services. For example, American English (US), Australian English (AU), and British English (UK) models are currently deployed as separate languages. However, from the standpoint of usability and maintenance costs, it is more practical to build and deploy a single, unified English model (referred to herein as the Global English Model (GEM)) that handles multiple English dialects in a single model. One beneficial aspect of GEM construction is to construct efficient training data that includes multiple dialects with good balance in terms of data size. Typically, these datasets are unbalanced.

[0031] Exemplary embodiments of the present invention mitigate such problems by introducing a method that favorably facilitates better training data collection (sorting and sampling) for building accurate global English models.

[0032] While the present invention is described in relation to a given illustrative architecture, it should be understood that other architectures, structures, substrate materials, process features, and stages / blocks may vary within the scope of the invention. For clarity, it should be noted that certain features may not be shown in all drawings. This is not intended to be construed as limiting any particular embodiment, or illustration, or the scope of the claims.

[0033] Figure 1 is a block / flow diagram of an exemplary system for organizing training data sequences based on the metric that similar sentences with different dialects are placed in close proximity, with weak constraints on speech length, for building a Global English Model (GEM) according to an embodiment of the present invention.

[0034] In the conventional method 5, data 10 is provided for random sampling 12, data sorting based on speech length 14, and model training 16.

[0035] Data 10 may be, for example, text or voice messages in Australian English, text or voice messages in British English, and text or voice messages in American English.

[0036] In contrast, an exemplary embodiment introduces a method 20 in which data 10 is favorably provided to a data sampler 22 for data sampling based on sentence similarity between dialects, then to a data sorter 24 for data sorting based on sentence similarity between dialects, and then model training 26 is performed.

[0037] Data 10 may be, for example, text or voice messages in Australian English, text or voice messages in British English, and text or voice messages in American English.

[0038] Thus, Method 20 organizes a training data sequence for building a Global English Model (GEM) based on the metric that similar sentences with different dialects are advantageously placed in close proximity with weak constraints on speech length. The advantage is that each mini-batch contains pairs of similar sentences between different English dialects. Each mini-batch advantageously contains a similar amount of dialect data. For better GEM construction, the same metric can also be applied to data sampling from large data pools, including realistic field data such as IBM Watson® speech-to-text (STT) customer data. The advantage of the present invention lies in how realistic big data can be efficiently utilized and organized into better training data for GEMs. A beneficial aspect of data sampling is introducing score penalties to control for a variety of sentences. This prevents the predictive network training from overfitting to biased texts resulting from strong constraints on word sequences.

[0039] IBM Watson® STT technology enables fast and accurate speech transcription in multiple languages ​​for a variety of use cases, including but not limited to customer self-service, agent assistance, and speech analysis. IBM Watson® STT is an application programming interface (API) cloud service that allows people to convert written text into natural-sounding speech in various languages ​​and voices, for example, within existing applications such as Watson® Assistant.

[0040] Figure 2 is a block / flow diagram of an exemplary method for organizing a training data sequence based on the metric that similar sentences with different dialects are placed in close proximity, with weak constraints on speech length, for building a Global English Model (GEM) according to an embodiment of the present invention.

[0041] In block 30,

number

number

[0042] In block 32, the metric for audio length

number

number

[0043] In block 34, the shortest unprocessed vocalization.

number

number

[0044] In block 36, set n=1.

[0045] In block 38, the utterance with the highest similarity is used as a better training sample for GEM construction, and other dialects

number

[0046] In block 40, it is determined whether n=N. If "no", proceed to block 42, where n is set to n+1. If "yes", proceed to block 44.

[0047] In block 44,

number

[0048]

number

[0049] A minimum amount of dialect training data is used as the base set, but it is not limited to this.

[0050]

number

[0051] The degree of similarity between dialects is,

number

[0052] Here, F(a,b) is the distance between sentences a and b based on word vectors of n word sequences, and P(d) is a similarity score-dependent penalty to avoid constructing biased text data.

[0053] P(d) = γe κd It is favorably given by -γ(d>0).

[0054] Here, γ and κ are hyperparameters.

[0055] Figure 3 is a block / flow diagram of an exemplary method for preparing training data for a speech recognition model according to an embodiment of the present invention.

[0056] In block 50, multiple audio datasets are acquired, and each audio dataset has different acoustic features.

[0057] In block 52, sentences from multiple speech datasets are favorably sorted so that similar sentences from different speech datasets are placed close together, while imposing weak constraints on speech length, in order to train the speech recognition model.

[0058] Furthermore, multiple speech datasets, each with different acoustic features, are favorably sampled from the data pool, while score penalties are presented to control various sentences, thereby ensuring that the sampled speech datasets contain multiple sets of similar sentences. In addition, the similar sentences are similar sentences with different dialects of the target language, and the speech recognition model is a global speech recognition model for the target language.

[0059] Figure 4 shows an exemplary data sorting method by employing an exemplary method according to an embodiment of the present invention, in comparison with a conventional method.

[0060] Block 60 shows a random selection from each dialect.

[0061] The first few sentences are in Australian English (au), the next few sentences are in British English (uk), and the last few sentences are in American English (us).

[0062] Thus, regardless of the words in a sentence, the length of a sentence, or any other characteristic of a sentence, sentences are grouped according to what type of English they are (e.g., au, uk, us).

[0063] Block 70 shows sentences sorted by audio length.

[0064] The first sentence (THANK YOU) is the shortest and is listed at the top, while the last sentence (OKAY I JUST WANTED TO ASK YOU TO STAY ON THE LINE FOR A MOMENT WE ARE HERE UNTIL NINE PM) is the longest and is listed at the bottom.

[0065] Thus, sentences are listed by length, regardless of any other elements.

[0066] Block 80 sorts sentences favorably by similarity, according to an exemplary embodiment.

[0067] For example, the first group 82 contains three sentences that include the phrase "THANK YOU." Regardless of dialect or length, the phrase "THANK YOU." appears in all three sentences, so such sentences are grouped together as 82 (based solely on similarity).

[0068] The second group, 84, also contains three sentences. Each sentence contains the phrase "I WILL RING." Since the phrase "I WILL RING." appears in all three sentences, regardless of dialect or length, such sentences are grouped together as 84 (based solely on similarity).

[0069] The third group, 86, also contains three sentences. Each sentence contains the phrase "WANTS TO DO" ​​or "WANTS TO KNOW." Since such phrases are similar and appear in all three sentences, regardless of dialect or length, these sentences are grouped together as 86 (based solely on similarity).

[0070] The fourth group, 88, also contains three sentences. Each sentence contains the phrase "OKAY," such as "OKAY WE'RE HERE," "I AH OKAY USED," or "OKAY I JUST WANTED TO ASK." Because such phrases are similar and appear in all three sentences, regardless of dialect or length, these sentences are grouped together as 88 (based solely on similarity).

[0071] Therefore, similar sentences with different dialects are advantageously placed close to each other and, consequently, grouped together (for example, in minibatches). In other words, the proximity or similarity of words or phrases is analyzed and evaluated in order to determine grouping or minibatches. Each group 82, 84, 86, and 88 can be called a minibatch. A minibatch may contain, for example, three sentences. However, a minibatch may contain any number of sentences from 3 to 10.

[0072] Figure 5 shows a system for sorting sentences from multiple audio datasets according to an embodiment of the present invention.

[0073] In one example, a first speech dataset 90 having acoustic features 92 is obtained, a second speech dataset 100 having acoustic features 102 is obtained, and a third speech dataset 110 having acoustic features 112 is obtained. Sentences from speech datasets 90, 100, and 110 are sorted favorably by a sorter 115 in terms of similarity or proximity in order to efficiently train a speech recognition model 120. Similar sentences can be grouped into multiple minibatches as described above with reference to Figure 5.

[0074] Furthermore, a weak constraint is imposed on the length of the audio, and therefore, similarity features, variables, or parameters become more favorably dominant when determining minibatches.

[0075] Figure 6 is a block / flow diagram of an exemplary processing system for organizing a training data sequence based on the metric that similar sentences with different dialects are placed in close proximity, with weak constraints on speech length, for building a Global English Model (GEM) according to an embodiment of the present invention.

[0076] Figure 6 shows a block diagram of the components of system 200, including computing device 205. It should be understood that Figure 6 provides only an example of one implementation and does not imply any limitation regarding environments in which different embodiments may be implemented. Many modifications can be made to the illustrated environment.

[0077] The computing device 205 includes a communication fabric 202, which provides communication between the computer processor 204, memory 206, persistent storage 208, communication unit 210, and input / output (I / O) interface 212. The communication fabric 202 can be implemented using any architecture designed to pass data and / or control information between processors (e.g., microprocessors, communication and network processors, etc.), system memory, peripheral devices, and any other hardware components in the system. For example, the communication fabric 202 can be implemented using one or more buses.

[0078] Memory 206, cache memory 216, and persistent storage 208 are computer-readable storage media. In this embodiment, memory 206 includes random access memory (RAM) 214. In another embodiment, memory 206 may be flash memory. In general, memory 206 may include any suitable volatile or non-volatile computer-readable storage media.

[0079] In some embodiments of the present invention, program 225 is included and operated by the AI ​​accelerator chip 222 as a component of the computing device 205. In other embodiments, program 225 is stored in persistent storage 208 for execution by the AI ​​accelerator chip 222 (for implementing training data sequences for RNN-T) via one or more memories of memory 206 in conjunction with one or more of the respective computer processors 204. In this embodiment, persistent storage 208 includes a magnetic hard disk drive. Instead of, or in addition to, a magnetic hard disk drive, persistent storage 208 may include a solid-state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0080] The media used by persistent storage 208 can also be removable. For example, a removable hard drive can be used for persistent storage 208. Other examples include optical and magnetic disks, thumb drives, and smart cards inserted into the drive for transfer to another computer-readable storage medium, which is also part of persistent storage 208.

[0081] In these examples, the communication unit 210 provides communication with other data processing systems or devices, including resources of a distributed data processing environment. In these examples, the communication unit 210 includes one or more network interface cards. The communication unit 210 can provide communication through the use of either or both physical and wireless communication links. The deep learning program 225 can be downloaded to persistent storage 208 through the communication unit 210.

[0082] The I / O interface 212 enables the input and output of data to and from other devices that can be connected to the computing system 200. For example, the I / O interface 212 can provide connection to an external device 218 such as a keyboard, keypad, touchscreen, and / or any other suitable input device. The external device 218 may also include portable computer-readable storage media such as thumb drives, portable optical or magnetic disks, and memory cards.

[0083] The display 220 provides a mechanism for displaying data to the user and may be, for example, a computer monitor.

[0084] Figure 7 is a block / flow diagram of an exemplary cloud computing environment according to an embodiment of the present invention.

[0085] While this invention includes a detailed description of cloud computing, it should be understood that the implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of this invention can be implemented in conjunction with any other type of computing environment that is currently known or will be developed in the future.

[0086] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and deployed with minimal management effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0087] The characteristics are as follows: On-demand self-service: Cloud consumers can unilaterally provision computing power, such as server time and network storage, automatically as needed, without requiring human interaction with service providers. Broad network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use with heterogeneous thin-client or thick-client platforms (e.g., mobile phones, laptops, and PDAs®). Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Consumers generally do not have control or knowledge of the exact location of the resources provided, but there is location independence in that they may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center). Rapid resilience: Capabilities are provisioned quickly and flexibly, sometimes automatically, allowing for rapid scaling out or rapid release and rapid scaling in. Often, to consumers, the available capacity for provisioning appears unlimited and can be purchased in any quantity at any time. Measurement Services: Cloud systems automatically control and optimize resource usage by leveraging measurement capabilities appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts) at a certain level of abstraction. Resource utilization can be monitored, controlled, and reported, providing transparency to both service providers and consumers of the services being used.

[0088] The service model is as follows: Software as a Service (SaaS): The capability offered to consumers is the use of a provider's applications running on cloud infrastructure. These applications are accessible from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with the exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The capability offered to consumers is the ability to deploy applications they have created or acquired, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they do control the configuration of the deployed applications and, in some cases, the application hosting environment. Infrastructure as a Service (IaaS): The ability provided to consumers is to provision processing, storage, networking, and other basic computing resources, allowing consumers to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do control the operating system, storage, and deployed applications, and, in some cases, have limited control over selected networking components (e.g., host firewalls).

[0089] The deployment model is as follows: Private Cloud: Cloud infrastructure is operated solely for an organization. It may be managed by that organization or a third party, and may reside on-premises or off-premises. Community Cloud: Cloud infrastructure is shared by multiple organizations to support a specific community with shared interests (e.g., mission, security requirements, policies, and compliance considerations). It may be managed by that organization or a third party and may reside on-premises or off-premises. Public cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by organizations that sell cloud services. Hybrid Cloud: This cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain unique entities but are joined together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing across clouds).

[0090] Cloud computing environments are service-oriented, emphasizing statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing lies an infrastructure that includes a network of interconnected nodes.

[0091] Referring here to Figure 7, an illustrative cloud computing environment 450 is shown to enable a use case of the present invention. As shown, the cloud computing environment 450 includes one or more cloud computing nodes 410 to which local computing devices used by a cloud consumer, such as a personal digital assistant (PDA) or cellular phone 454A, a desktop computer 454B, a laptop computer 454C, and / or an automotive computer system 454N, can communicate. The nodes 410 can communicate with each other. They may be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above. This makes it possible to provide the cloud computing environment 450 as an infrastructure, platform, and / or software as a service to cloud consumers, without requiring them to maintain resources on their local computing devices for that purpose. The types of computing devices 454A to 454N shown in Figure 7 are intended to be illustrative only, and it should be understood that the computing node 410 and the cloud computing environment 450 can communicate with any type of computerized device via any type of network and / or network addressable connection (e.g., using a web browser).

[0092] Figure 8 is a schematic diagram of an exemplary abstraction model layer according to an embodiment of the present invention. It should be understood that the components, layers, and functions shown in Figure 8 are illustrative only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0093] The hardware and software layer 560 includes hardware and software components. Examples of hardware components include a mainframe 561; a RISC (Reduced Instruction Set Computer) architecture-based server 562; a server 563; a blade server 564; a storage device 565; and network and networking components 566. In some embodiments, the software components include network application server software 567 and database software 568.

[0094] The virtualization layer 570 provides an abstraction layer, from which the following examples of virtual entities may be provided: virtual servers 571; virtual storage 572; virtual networks including virtual private networks 573; virtual applications and operating systems 574; and virtual clients 575.

[0095] In one example, the management layer 580 may provide the functions described below. Resource provisioning 581 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Measurement and pricing 582 provides cost tracking as resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection of data and other resources. User portal 583 provides consumers and system administrators with access to the cloud computing environment. Service level management 584 provides allocation and management of cloud computing resources to ensure that the required service levels are met. Service level agreement (SLA) planning and execution 585 provides pre-arrangements and procurement of cloud computing resources where future requirements are expected to conform to the SLA.

[0096] The workload layer 590 provides examples of functions that can be utilized in a cloud computing environment. Examples of workloads and functions that can be provided from this layer include mapping and navigation 541; software development and lifecycle management 592; virtual classroom education delivery 593; data analysis processing 594; transaction processing 595; and training data sequences 20 for RNN-T.

[0097] The present invention may be a system, method, and / or a computer program product. The computer program product may include a computer-readable storage medium (or a set of mediums) having computer-readable program instructions for causing a processor to execute an aspect of the present invention.

[0098] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any preferred combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage mediums includes: portable computer diskettes, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), static random access memory, portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices, such as punch cards or raised structures in grooves recording instructions, and any preferred combination of the foregoing. When used herein, a computer-readable storage medium should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0099] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers them for storage in a computer-readable storage medium within each computing / processing device.

[0100] The computer-readable program instructions that perform the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or fully on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can be personalized by executing computer-readable program instructions using state information of computer-readable program instructions in order to perform an aspect of the present invention.

[0101] Aspects of the present invention will be described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0102] These computer-readable program instructions can be provided to the processor of at least one of a general-purpose computer, a dedicated computer, or other programmable data processing device to create a machine, thereby creating means for implementing functions / operations specified in one or more blocks or modules of a flowchart and / or block diagram, through which instructions executed via the processor of the computer or other programmable data processing device. Furthermore, these computer-readable program instructions can be stored in a computer-readable storage medium, which can instruct a computer, programmable data processing device, and / or other device to function in a specific manner, thereby creating a product containing instructions that implement modes of functions / operations specified in one or more blocks or modules of a flowchart and / or block diagram.

[0103] Computer-readable program instructions can also be loaded into a computer, other programmable data processing device, or other device to execute a series of operational blocks / stages on the computer, other programmable device, or other device, thereby generating a computer implementation process in which the instructions executed on the computer, other programmable device, or other device implement the functions / operations specified in one or more blocks or modules of a flowchart and / or block diagram.

[0104] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of the system, method, and computer program product according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions that implement a specified logical function. In some alternative implementations, the functions described in the blocks may occur in an order different from the order shown in the figures. For example, two consecutively shown blocks may actually be executed substantially simultaneously, depending on the functionality involved, or the blocks may, in some cases, be executed in reverse order. It should also be noted that each block in the block diagram and / or flowchart diagram, and combinations of blocks in the block diagram and / or flowchart diagram, may be implemented by a dedicated hardware-based system that performs a specified function or operation, or executes a combination of dedicated hardware and computer instructions.

[0105] Any reference in this specification to “one embodiment” or “embodiment” of the Principle, and to other variations thereof, means that the specific features, structures, characteristics, etc. described in relation to that embodiment are included in at least one embodiment of the Principle. Therefore, the phrases “in one embodiment” or “in an embodiment,” and any other variations appearing in various places throughout this specification, do not necessarily all refer to the same embodiment.

[0106] Please understand that the use of any of the following " / ", "and / or", and "at least one of" is intended to encompass the selection of only the first enumerated option (A), or only the second enumerated option (B), or both options (A and B), for example, in the cases of "A / B", "A and / or B", and "at least one of A and B". As a further example, in the cases of "A, B, and / or C", and "at least one of A, B, and C", such phrases are intended to encompass the selection of only the first enumerated option (A), or only the second enumerated option (B), or only the third enumerated option (C), or only the first and second enumerated options (A and B), or only the first and third enumerated options (A and C), or only the second and third enumerated options (B and C), or all three options (A, B, and C). This can be extended to many of the listed items, as will be readily apparent to those skilled in the art in this and related fields.

[0107] Preferred embodiments of a method and system for constructing efficient training data sequences for a recurrent neural network transducer (RNN-T) based global English model (this is illustrative and not intended to be limiting) have been described, but it should be noted that modifications and variations may be made in light of the above teachings by those skilled in the art. Accordingly, it should be understood that in the particular embodiments described, modifications may be made that fall within the scope of the invention as outlined by the appended claims. Thus, aspects of the invention have been described with the details and specificities required by patent law, but the matters to be claimed and the matters to be protected by the patent certificate are described in the appended claims.

Claims

1. A computer implementation method for preparing training data for a speech recognition model, In the stage of acquiring multiple audio datasets, each audio dataset contains audio data in a different dialect; and To train the speech recognition model, a step is to sort sentences from multiple speech datasets such that similar sentences from different speech datasets are placed close together, while imposing constraints on the length of the speech, wherein the sorting step includes sorting the speech dataset of a base dialect based on the length of the speech and extracting similar speech data from speech datasets of other dialects. A computer implementation method comprising the above.

2. The computer implementation method according to claim 1, further comprising the step of sampling from a data pool a plurality of speech datasets, each having speech data of a different dialect, wherein the sampled speech dataset includes a plurality of sets of similar sentences.

3. The computer implementation method according to claim 2, wherein, in the step of sorting sentences from the plurality of speech datasets, a score penalty dependent on the distance between the speech data of the base dialect and the speech data of the other dialects is presented.

4. The computer implementation method according to claim 1, wherein the similar sentence is a similar sentence having different dialects of the target language.

5. The computer implementation method according to claim 4, wherein the speech recognition model is a global speech recognition model for the target language.

6. The computer implementation method according to claim 1, wherein the similar sentences from the different audio datasets are grouped into minibatches.

7. The computer implementation method according to claim 6, wherein each of the minibatches includes a pair of sentences between different English dialects.

8. The computer implementation method according to claim 6, wherein each of the minibatches contains a similar amount of dialect data.

9. The computer implementation method according to claim 1, wherein the similarity between different English dialects of the similar sentences from different audio datasets is calculated based on the distance between the audio data of the base dialect and the audio data of the other dialect, which are based on word vectors of n word sequences, and a similarity score-dependent penalty P(d) to avoid forming biased text data.

10. The aforementioned similarity score-dependent penalty P(d) is, P(d=γe κｄ -γ(d>00) The computer implementation method according to claim 9, where γ and κ are hyperparameters, given by [the specified method].

11. A computer program for preparing training data for a speech recognition model, wherein the computer program is provided to a computer: Procedure for obtaining multiple audio datasets, each audio dataset containing audio data in a different dialect; and A procedure for training the speech recognition model, comprising sorting sentences from multiple speech datasets such that similar sentences from different speech datasets are placed close together, while imposing constraints on speech length, wherein the sorting procedure includes sorting a base dialect speech dataset based on speech length and extracting similar speech data from other dialect speech datasets. A computer program designed to execute something.

12. The computer program according to claim 11, wherein a plurality of speech datasets, each having speech data of a different dialect, are sampled from a data pool, so that the sampled speech dataset includes a plurality of sets of similar sentences.

13. The computer program according to claim 12, wherein in a step of sorting sentences from a plurality of speech datasets, a score penalty dependent on the distance between the speech data of the base dialect and the speech data of the other dialects is presented.

14. The computer program according to claim 11, wherein the similar sentence is a similar sentence having different dialects of the target language.

15. The computer program according to claim 14, wherein the speech recognition model is a global speech recognition model for the target language.

16. The computer program according to claim 11, wherein similar sentences from the different audio datasets are grouped into minibatches.

17. The computer program according to claim 16, wherein each of the minibatches includes a pair of sentences between different English dialects.

18. The computer program according to claim 16, wherein each of the minibatches contains a similar amount of dialect data.

19. The computer program according to claim 11, wherein the similarity between different English dialects of the aforementioned similar sentences from different audio datasets is calculated based on the distance between the audio data of the base dialect and the audio data of the other dialect, which are based on word vectors of n word sequences, and a similarity score-dependent penalty P(d) to avoid forming biased text data.

20. The aforementioned similarity score-dependent penalty P(d) is, P(d=γe κｄ -γ(d>00) A computer program according to claim 19, provided by, where γ and κ are hyperparameters.

21. A system for preparing training data for speech recognition models, memory; and One or more processors communicating with the aforementioned memory The one or more processors are provided with, Obtain multiple audio datasets, each containing audio data in a different dialect; and To train the speech recognition model, sentences from multiple speech datasets are sorted so that similar sentences from different speech datasets are placed close together, while imposing constraints on speech length, wherein the sorting includes sorting the speech dataset of a base dialect based on speech length and extracting similar speech data from speech datasets of other dialects. A system that is configured in such a way.

22. The system according to claim 21, wherein the similarity between different English dialects of the similar sentences from different audio datasets is calculated based on the distance between the audio data of the base dialect and the audio data of the other dialect, based on word vectors of n word sequences, and a similarity score-dependent penalty P(d) to avoid forming biased text data.

23. A computer implementation method for preparing training data for a speech recognition model, During the acquisition of multiple audio datasets, each audio dataset contains audio data from different dialects; A step of training the speech recognition model is to sort sentences from multiple speech datasets such that similar sentences from different speech datasets are placed close together, while imposing constraints on speech length, wherein the sorting step includes sorting a base dialect speech dataset based on speech length and extracting similar speech data from other dialect speech datasets; and The step of grouping similar sentences from the different audio datasets into minibatches, where each minibatch contains a pair of sentences from different English dialects. A computer implementation method comprising the above.

24. The computer implementation method according to claim 23, wherein the similarity between different English dialects of the similar sentences from different audio datasets is calculated based on the distance between the audio data of the base dialect and the audio data of the other dialect, which are based on word vectors of n word sequences, and a similarity score-dependent penalty P(d) to avoid forming biased text data.

25. A computer program for preparing training data for a speech recognition model, wherein the computer program is provided to a computer: The procedure for obtaining multiple audio datasets, each containing audio data from a different dialect; A procedure for training the speech recognition model, comprising sorting sentences from multiple speech datasets such that similar sentences from different speech datasets are placed close together, while imposing constraints on speech length, wherein the sorting procedure includes sorting a base dialect speech dataset based on speech length and extracting similar speech data from other dialect speech datasets; and A procedure for grouping similar sentences from the different audio datasets into minibatches, wherein each minibatch contains a pair of sentences from different English dialects. A computer program designed to execute something.

Images