Sorting data to generate an RNN-T model

Abstract

A computer-implemented method for preparing training data for a speech recognition model is provided, the method including obtaining a plurality of sentences from a corpus; dividing each phoneme in each sentence of the plurality of sentences into three hidden states; calculating a score for each sentence of the plurality of sentences based on an amount of variation in durations of the three hidden states for each phoneme in the sentence; and sorting the plurality of sentences by using the calculated scores.

Description

[Technical field]

[0001] The present invention relates generally to machine learning, and more particularly to a method and system for data sorting to generate a recurrent neural network transducer (RNN-T) model. [Background technology]

[0002] End-to-end models for automatic speech recognition (ASR) have gained popularity in recent years as a way to combine the separate components of traditional ASR systems (e.g., acoustic, pronunciation, and language models) into a single neural network. Examples of such models include connectionist temporal classification (CTC)-based models, recurrent neural network transducers (RNN-T), and attention-based seq2seq models. Among these models, RNN-T is the most streaming-friendly end-to-end recognizer and has shown competitive performance compared to traditional systems. Summary of the Invention

[0003] According to one aspect of the invention, a computer-implemented method for preparing training data for a speech recognition model is provided, the computer-implemented method including obtaining a plurality of sentences from a corpus, splitting each phoneme in each sentence of the plurality of sentences into three hidden states, calculating a score for each sentence of the plurality of sentences based on an amount of variation in durations of the three hidden states for each phoneme in the sentence, and sorting the plurality of sentences by using the calculated scores.

[0004] According to another aspect of the present invention, there is provided a computer program product for preparing training data for a speech recognition model, the computer program product including a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a computer to cause a computer to obtain a plurality of sentences from a corpus, split each phoneme in each sentence of the plurality of sentences into three hidden states, calculate a score for each sentence of the plurality of sentences based on an amount of variation in durations of the three hidden states for each phoneme in the sentence, and sort the plurality of sentences using the calculated scores.

[0005] According to yet another aspect of the present invention, there is provided a system for preparing training data for a speech recognition model, the system including a memory and one or more processors in communication with the memory configured to retrieve a plurality of sentences from a corpus, split each phoneme in each sentence of the plurality of sentences into three hidden states, calculate a score for each sentence of the plurality of sentences based on an amount of variation in durations of the three hidden states for each phoneme in the sentence, and sort the plurality of sentences by using the calculated scores.

[0006] It should be noted that the exemplary embodiments are described in relation to different subject matters. In particular, some embodiments are described in relation to method type claims and other embodiments are described in relation to device type claims. However, a person skilled in the art will infer from the above and following description that, unless otherwise stated, any combination of features belonging to one type of subject matter, as well as any combination between features relating to different subject matters, in particular any combination between features of a method type claim and a feature of a device type claim, is considered to be described in this document.

[0007] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which should be read in connection with the accompanying drawings.

[0008] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which: [Brief description of the drawings]

[0009] [Figure 1] 4A-4C are block / flow diagrams of an exemplary sequence of phonemes with balanced durations between states and an exemplary sequence of phonemes with unequal durations between states, in accordance with an embodiment of the present invention. [Diagram 2] FIG. 2 is a block / flow diagram of an exemplary method for selecting sentences from a training corpus and splitting phonemes in a sequence into three hidden states according to an embodiment of the present invention. [Diagram 3] 11A-11C illustrate the calculation of a uniform score when statistics from clear speech are not available and when statistics from clear speech are available according to an embodiment of the present invention. [Figure 4] 1 is a block / flow diagram of an exemplary method for preparing training data for a speech recognition model, according to an embodiment of the present invention. [Diagram 5] FIG. 1 illustrates a system for preparing training data for a speech recognition model, according to an embodiment of the present invention. [Figure 6] 1 is a block / flow diagram of an exemplary processing system for preparing training data for a speech recognition model, in accordance with an embodiment of the present invention. [Figure 7] FIG. 2 is a block / flow diagram of an exemplary cloud computing environment, in accordance with an embodiment of the present invention. [Figure 8] 2 is a schematic diagram of an exemplary abstract model layer according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Throughout the drawings, the same or similar reference numbers represent the same or similar elements.

[0011] Embodiments in accordance with the present invention provide methods and devices for constructing efficient training data sequences for Recurrent Neural Network Transducer (RNN-T) based models. Exemplary embodiments advantageously sort the training data set by evaluating the change in duration of the hidden states for each phoneme.

[0012] Before delving into RNN-T, speech recognition continues to evolve to meet the non-linear and rapid demands of the mobile environment. New speech recognition architectures or improvements to existing architectures continue to be developed to improve the quality of ASR systems. To explain, speech recognition initially employed multiple models, with each model having a dedicated purpose. For example, ASR systems included an acoustic model (AM), a pronunciation model (PM), and a language model (LM). The acoustic model mapped segments of audio (e.g., frames of audio) to phonemes. The pronunciation model connected these phonemes to form words, and the language model was used to represent the likelihood of a given phrase (e.g., the likelihood of a sequence of words). Furthermore, although these separate models worked together, each model was trained independently and was often manually designed on a different dataset.

[0013] While the separate model approach enabled speech recognition systems to be highly accurate, especially when the training corpus (e.g., a body of training data) for a given model satisfied the model's validity, the need to train separate models independently introduced its own complexities, leading to architectures using integrated models. These integrated models attempted to directly map audio waveforms (e.g., input sequence) to output sentences (e.g., output sequence) using a single neural network. This resulted in sequence-to-sequence approaches, whereby a sequence of words (or graphemes) was generated given a sequence of audio features. Examples of sequence-to-sequence models include "attention-based" models and "listen-attend-spell" (LAS) models. LAS models transcribe spoken utterances using a listener component, an attender component, and a speller component. Here, the listener is a recurrent neural network (RNN) encoder that receives audio input (e.g., a time-frequency representation of the speech input) and maps the audio input to a higher-level feature representation. The Attender focuses on higher-level features to learn alignments between input features and predicted subword units (e.g., graphemes or word fragments). The Speller is an attention-based RNN decoder that generates character sequences from the input by creating a probability distribution over a set of hypothesized words. In an integrated architecture, all components of the model may be trained together as a single end-to-end (E2E) neural network. Here, E2E models refer to models whose architecture consists entirely of neural networks. A complete neural network can function without external or manually designed components (e.g., finite-state transducers, lexicons, or text normalization modules) or both.Additionally, when E2E models are trained, they generally do not require bootstrapping from decision trees or time alignment from a separate system.

[0014] While early E2E models demonstrated improved accuracy and training over individually trained models, these E2E models, such as the LAS model, worked by reviewing the entire input sequence before generating output text and therefore did not allow for streaming output as the input was received. Without streaming capability, the LAS model is unable to perform real-time speech transcription. Due to this deficiency, deploying LAS models for speech applications that are latency sensitive or require real-time speech transcription can be problematic.

[0015] In addition, speech recognition systems having acoustic, pronunciation, and language models, or such models configured together, may rely on a decoder that must search a relatively large search graph associated with these models. For large search graphs, hosting this type of speech recognition system entirely on-device is not conducive. Now, when a speech recognition system is hosted "on-device," the device that receives the audio input uses its processor to perform the functions of the speech recognition system. For example, when a speech recognition system is hosted entirely on-device, the device's processor does not need to coordinate with any off-device computing resources to perform the functions of the speech recognition system. Devices that perform speech recognition that are not entirely on-device rely on remote computing (e.g., a remote computing system or cloud computing), and thus on-line connectivity, to perform at least some functions of the speech recognition system. For example, the speech recognition system uses a network connection with a server-based model to perform decoding on a large search graph.

[0016] Unfortunately, the reliance on remote connections makes speech recognition systems vulnerable to latency issues and / or inherent unreliability of communication networks. To improve the usefulness of speech recognition by circumventing these issues, speech recognition systems have evolved into a form of sequence-to-sequence model, again known as recurrent neural network transducer (RNN-T). Unlike other sequence-to-sequence models, which do not employ attention mechanisms and generally need to process entire sequences (e.g., audio waveforms) to produce an output (e.g., a sentence), RNN-T processes input samples continuously and streams output symbols. This is a feature that is particularly attractive for real-time communication. For example, speech recognition using RNN-T may output characters as they are spoken one by one.

[0017] Thus, there is a need for a more efficient process for training data sequences using RNN-T. Exemplary embodiments of the present invention introduce a method to improve sorting of training data by analyzing and evaluating the change in duration of the hidden states for each phoneme.

[0018] Although the present invention is described with respect to a given example architecture, it should be understood that other architectures, structures, substrate materials, and process features and steps / blocks may vary within the scope of the present invention. It should be noted that for clarity, certain features may not be shown in all drawings. This is not intended to be construed as a limitation of any particular embodiment or example, or of the scope of the claims.

[0019] FIG. 1 is a block / flow diagram of an exemplary sequence of phonemes with balanced durations between states and an exemplary sequence of phonemes with unequal durations between states, according to an embodiment of the present invention.

[0020] The exemplary embodiment introduces a method for organizing training data sequences based on the assumption that utterances with balanced phoneme context-dependent states are acoustically easier. In other words, utterances with imbalanced durations of initial, middle, and final states per phoneme are considered as more difficult cases from an acoustic point of view. For example, spontaneous speech tends to contain imbalanced durations during hidden states and is more difficult than reading-style speech.

[0021] A uniform score based on the duration between the three hidden states is obtained by the difference in the number of frames in each state to estimate how acoustically difficult the utterance is. It is used as a criterion for sorting the training samples. The duration of each hidden state is checked, and the exemplary method ignores silence-related phonemes (X, VN, NS) when the alignment changes for acoustically difficult utterances.

[0022] The sequence of phonemes 10 includes a first phoneme 12 (Ab), a second phoneme 14 (Am), a third phoneme 16 (Ae), a fourth phoneme 18 (sb), a fifth phoneme 20 (sm), and a sixth phoneme 22 (se). The first phoneme 12 has a duration 32, the second phoneme 14 has a duration 34, the third phoneme 16 has a duration 36, the fourth phoneme 18 has a duration 38, the fifth phoneme 20 has a duration 40, and the sixth phoneme 22 has a duration 42.

[0023] The periods 32, 34, 36, 38, 40, 42 of the phonemes 12, 14, 16, 18, 20, 22 are evenly balanced. In other words, each phoneme has a balanced period between states. In other words, "balanced" refers to the regularity or uniformity or homogeneity or similarity of each period. Thus, speech with balanced phonemes may be called clean speech, i.e. speech that is more structured, organized, regular, precise, logical, or measured, or a combination thereof. Structured or clean speech is easier to understand, and therefore acoustically easier. The frames of each state are more equal or equivalent (less difference between states).

[0024] In contrast, the sequence of phonemes 50 includes a first phoneme 72 (Ab), a second phoneme 74 (Am), a third phoneme 76 (Ae), a fourth phoneme 78 (sb), a fifth phoneme 80 (sm), and a sixth phoneme 82 (se). The first phoneme 72 has a duration 52, the second phoneme 74 has a duration 54, the third phoneme 76 has a duration 56, the fourth phoneme 78 has a duration 58, the fifth phoneme 80 has a duration 60, and the sixth phoneme 82 has a duration 62.

[0025] The durations 72, 74, 76, 78, 80, 82 of the phonemes 52, 54, 56, 58, 60, 62 are not evenly balanced. In other words, each phoneme has an unbalanced duration between states. In other words, "unbalanced" refers to the irregularity or inconsistency or non-uniformity or dissimilarity of each duration. Thus, speech with unbalanced phonemes may be called unclean speech, i.e. speech that is unstructured, unorganized, non-regular, non-precise, non-logical, non-measured, or a combination thereof. Instead, unclean speech is more chaotic, spontaneous, noisy, unorganized, and messy or disorganized. Unstructured speech (unclean speech) is more difficult to understand and therefore acoustically challenging. The frames of each state are not proportional or comparable or are less consistent (there is a large difference between states). For example, phoneme 78 has a frame of 8 and phoneme 82 has a frame of 1. As a result, the difference between such frames is large.

[0026] The uniformity score, described below with reference to Figures 2 and 3, is based on the duration between states and is obtained by the difference in the number of frames for each state.

[0027] FIG. 2 is a block / flow diagram 90 of an exemplary method for selecting sentences from a training corpus and splitting phonemes in a sequence into three hidden states according to an embodiment of the present invention.

[0028] For score calculation, a sentence 92 is selected from the training corpus and each phoneme in the sentence is split or separated into three hidden states. For example, phoneme 94 is selected from an acoustic representation 93 of sentence 92. Phoneme 94 is split into three hidden states: hidden state 1 (94A), hidden state 2 (94B), and hidden state 3 (94C). Hidden state 1 may be referred to as the initial state, hidden state 2 may be referred to as the middle state, and hidden state 3 may be referred to as the final state. Hidden state 1 is presented as acoustic representation 96A, hidden state 2 is presented as acoustic representation 96B, and hidden state 3 is presented as acoustic representation 96C.

[0029] A uniform score is then calculated for each sentence, as described below with reference to FIG. 3, and the training samples are sorted in ascending order based on the calculated uniform scores.

[0030] FIG. 3 illustrates the calculation of a uniform score when statistics from clear speech are not available and when statistics from clear speech are available, according to an embodiment of the present invention.

[0031] In the first scenario 100, the phoneme-independent uniform score is calculated when statistics from clean speech are not available:

number

[0032] d n,x is the duration of the xth state (x can be either b: initial, m: medial, or e: final) of the nth phoneme, and μ n d n,x where N is the total number of phonemes in the sentence.

[0033] In the second scenario 102, the phoneme-dependent uniform score is calculated when statistics from clean speech are available:

number

[0034] λ k(n),x is the ideal duration of the x-th state of phoneme k(n) to which the n-th phoneme should be referred, estimated beforehand from clean speech, and k(n) is a function that maps the n-th phoneme to the corresponding phoneme.

[0035] FIG. 4 is a block / flow diagram of an exemplary method for preparing training data for a speech recognition model according to an embodiment of the present invention.

[0036] At block 120, a number of sentences are obtained from the corpus.

[0037] In block 122, each phoneme in each sentence is divided into three hidden states.

[0038] At block 124, a score is calculated for each sentence based on the amount of variation in the durations of the three hidden states for each phoneme in the sentence.

[0039] In block 126, the sentences are sorted (eg, in ascending order) using the calculated scores.

[0040] FIG. 5 illustrates a system for preparing training data for a speech recognition model according to an embodiment of the present invention.

[0041] Corpus 130 is presented including a collection of written text from a number of different sources. Sentences 140 may be extracted from corpus 130. In one example, sentence 3 may be selected from sentences 140. Phonemes 150 are derived from sentence 3.

[0042] A phoneme may be defined as the smallest class of sounds that lead to a difference in meaning in a particular language. For example, in English, / l / and / r / are two phonemes because substituting one for the other produces words with different meanings, as is evident in "royal" and "loyal." Note that phonemes are usually indicated by two slashes.

[0043] Phonemes vary significantly in the way they are pronounced (the variations are called "phones"), that is, people speak with different accents, different speeds, and different vocal pitches (e.g., male versus female voices). In fact, it turns out that even a word as short and everyday as "the" can be pronounced in many entirely different ways.

[0044] Each of the phonemes 150 is divided into three hidden states. For example, phoneme 1 may be / c / , phoneme 2 may be / a / , and phoneme 3 may be / t / . Phoneme 1 is divided into three hidden states, phoneme 2 is divided into three hidden states, phoneme 3 is divided into three hidden states, and so on. For simplicity, phoneme 3 is shown divided into three hidden states 160. Hidden state 1 is an initial state, hidden state 2 is an intermediate state, and hidden state 3 is a final state. Hidden state 1 has period A, hidden state 2 has period B, and hidden state 3 has period C. The initial state represents a first acoustic wave, the intermediate state represents a second acoustic wave, and the final state represents a third acoustic wave, with each acoustic wave potentially having a different frequency.

[0045] The uniform score calculator 170 is used to calculate the uniform score based on when statistics from the clean speech are available or not available. The statistics may refer to various data or information or samples of data related to a number of variables or parameters.

[0046] In one example, the uniformity score is calculated from the average duration of the three states of the phoneme 172. In another example, the uniformity score is calculated from an ideal duration 174 estimated a priori from clean speech.

[0047] The training samples may be sorted by a sorting component 180. The training samples may be sorted, for example, in ascending order of the proposed uniform scores 182, 184, 186, 188.

[0048] FIG. 6 is a block / flow diagram of an exemplary processing system for preparing training data for a speech recognition model, according to an embodiment of the present invention.

[0049] 6 illustrates a block diagram of components of a system 200, including a computing device 205. It should be understood that FIG. 6 is merely provided as an illustration of one embodiment and is not intended to suggest any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the illustrated environment may be made.

[0050] Computing device 205 includes a communications fabric 202 that provides communications between computer processor 204, memory 206, persistent storage 208, communications unit 210, and input / output (I / O) interface 212. Communications fabric 202 may be implemented in any architecture designed to pass data and / or control information between processors (such as microprocessors, communications and network processors), system memory, peripheral devices, and any other hardware components in the system. For example, communications fabric 202 may be implemented with one or more buses.

[0051] Memory 206, cache memory 216, and persistent storage 208 are computer-readable storage media. In this embodiment, memory 206 includes random access memory (RAM) 214. In another embodiment, memory 206 may be flash memory. In general, memory 206 may include any suitable volatile or non-volatile computer-readable storage medium.

[0052] In some embodiments of the present invention, the program 225 is included and operated by the AI ​​accelerator chip 222 as a component of the computing device 205. In other embodiments, the program 225 is stored in the persistent storage 208 for execution by the AI ​​accelerator chip 222 in conjunction with one or more of the respective computer processors 204 (to implement the training data sequences for the RNN-T including the data sorting techniques described herein) via one or more memories of the memory 206. In this embodiment, the persistent storage 208 includes a magnetic hard disk drive. As an alternative to or in addition to a magnetic hard disk drive, the persistent storage 208 may include a solid state hard drive, a semiconductor storage device, a read only memory (ROM), an erasable programmable read only memory (EPROM), a flash memory, or any other computer readable storage medium capable of storing program instructions or digital information.

[0053] The media used by persistent storage 208 may also be removable. For example, a removable hard drive may be used for persistent storage 208. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer-readable storage medium that is also part of persistent storage 208.

[0054] In these examples, the communication unit 210 provides communication with other data processing systems or devices, including resources of a distributed data processing environment. In these examples, the communication unit 210 includes one or more network interface cards. The communication unit 210 may provide communication through the use of either or both physical and wireless communication links. The deep learning program 225 may be downloaded to the persistent storage 208 through the communication unit 210.

[0055] I / O interface 212 allows for the input and output of data with other devices that may be connected to computing system 200. For example, I / O interface 212 may provide a connection to external devices 218, such as a keyboard, a keypad, a touch screen, or some other suitable input device, or a combination thereof. External devices 218 may also include portable computer-readable storage media, such as, for example, thumb drives, portable optical or magnetic disks, and memory cards.

[0056] Display 220 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0057] FIG. 7 is a block / flow diagram of an exemplary cloud computing environment in accordance with an embodiment of the present invention.

[0058] Although the present invention includes detailed descriptions of cloud computing, it should be understood that implementations of the teachings recited herein are not limited to cloud computing environments. Rather, embodiments of the present invention may be practiced in conjunction with any other type of computing environment now known or later developed.

[0059] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0060] The characteristics are as follows:

[0061] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capabilities such as server time and network storage automatically as needed, without the need for human interaction with the service provider.

[0062] Broad network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., cell phones, laptops, and PDAs).

[0063] Resource Sharing: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Location independence is implied in that consumers generally do not have control or knowledge over the exact location of the resources provided, but may be able to specify location at a higher level of abstraction (e.g., country, state, or data center).

[0064] Rapid Scalability: Capabilities can be quickly and elastically provisioned, in some cases automatically, to instantly scale out, and quickly released to instantly scale in. To the consumer, the capabilities available for provisioning often appear unlimited and can be purchased in any quantity at any time.

[0065] Services are meterable: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used.

[0066] The service model is as follows:

[0067] Software as a Service (SaaS): The capability provided to the consumer is the use of the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0068] Platform as a Service (PaaS): The capability provided to the consumer is to deploy consumer-created or consumer-acquired applications, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, to the extent possible, the application hosting environment configuration.

[0069] Infrastructure as a Service (IaaS): The capability provided to the consumer is the supply of processing, storage, networking, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and possibly limited control over select networking components (e.g., host firewalls).

[0070] The deployment model is as follows:

[0071] Private Cloud: The cloud infrastructure is operated exclusively for an organization. It is managed by the organization or a third party and can be on-premise or off-premise.

[0072] Community Cloud: The cloud infrastructure is shared by multiple organizations to support a specific community with shared concerns (e.g., mission, security requirements, policy, and compliance considerations). The cloud infrastructure is managed by the organizations or a third party and can be on-premise or off-premise.

[0073] Public Cloud: The cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0074] Hybrid Cloud: The cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain a unique entity but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0075] A cloud computing environment is stateless, low-coupling, modular, and service-oriented with a focus on semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0076] Referring now to FIG. 7, an exemplary cloud computing environment 450 for enabling use cases of the present invention is shown. As shown, the cloud computing environment 450 includes one or more cloud computing nodes 410 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 454A, a desktop computer 454B, a laptop computer 454C, or an automobile computer system 454N, or combinations thereof, may communicate. The nodes 410 may communicate with each other. They may be physically or virtually grouped in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or combinations thereof, as described above (not shown). This enables the cloud computing environment 450 to offer infrastructure, platform, or software, or combinations thereof, as a service without the cloud consumer having to maintain resources on the local computing device. It should be understood that the types of computing devices 454A-N shown in FIG. 7 are intended to be exemplary only, and that the computing node 410 and cloud computing environment 450 may communicate with any type of computerized device via any type of network and / or network addressable connections (e.g., using a web browser).

[0077] 8 is a schematic diagram of an exemplary abstract model layer according to an embodiment of the present invention. It should be understood in advance that the components, layers, and functions shown in FIG. 8 are intended to be merely illustrative, and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0078] Hardware and software layer 560 includes hardware and software components. Examples of hardware components include mainframe 561, reduced instruction set computer (RISC) architecture based servers 562, servers 563, blade servers 564, storage devices 565, and networks and networking components 566. In some embodiments, software components include network application server software 567 and database software 568.

[0079] The virtualization layer 570 provides an abstraction layer over which the following examples of virtual entities may be provided: virtual servers 571, virtual storage 572, virtual networks including virtual private networks 573, virtual applications and operating systems 574, and virtual clients 575.

[0080] In one embodiment, management layer 580 may provide the functionality described below. Resource provisioning 581 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 582 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one embodiment, these resources may include application software licenses. Security provides protection for data and other resources as well as identity verification for cloud consumers and tasks. User portal 583 provides consumers and system administrators with access to the cloud computing environment. Service level management 584 provides cloud computing resource allocation and management so that requested service levels are met. Service level agreement (SLA) planning and fulfillment 585 provides pre-positioning and procurement of cloud computing resources where future requirements are anticipated according to SLAs.

[0081] The workload layer 590 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this layer include mapping and navigation 541, software development and lifecycle management 592, virtual classroom instructional delivery 593, data analytics processing 594, transaction processing 595, and data sorting for RNN-T models 596.

[0082] The present invention may be a system, a method, or a computer program product, or a combination thereof. The computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0083] A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memories, read-only memories, erasable programmable read-only memories (EPROMs or Flash memories), static random access memories, portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves that allow instructions to be recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through electrical wires.

[0084] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0085] The computer readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or object oriented programming languages ​​such as Smalltalk®, C++, and traditional procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may execute completely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electronic circuitry to perform aspects of the invention.

[0086] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0087] These computer readable program instructions may be provided to at least one processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to manufacture a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for implementing the functions / operations specified in one or more blocks or modules of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored in a computer readable storage medium that may direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium having instructions stored therein includes an article of manufacture including instructions for implementing aspects of the functions / operations specified in one or more blocks or modules of the flowcharts and / or block diagrams.

[0088] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to execute a series of operational blocks / steps to create a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / operations specified in one or more blocks or modules of the flowcharts and / or block diagrams.

[0089] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may occur out of the order described in the drawings. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in the reverse order depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or executes a combination of dedicated hardware and computer instructions.

[0090] References in the specification to "one embodiment" or "an embodiment" of the present principles, and other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment of the present principles. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment," as well as any other variations thereof, appearing in various places throughout the specification are not necessarily all referring to the same embodiment.

[0091] Use of any of the following " / ", "and / or" and "at least one of" should be understood to be intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B), for example, "A / B", "A and / or B", and "at least one of A and B". As a further example, "A, B, and / or C" and "at least one of A, B, and C" are intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A and B and C). This may be extended when multiple items are listed, as would be readily apparent to one of ordinary skill in the art and related arts.

[0092] Although preferred embodiments of a method and system for data sorting to generate a recurrent neural network transducer (RNN-T) model have been described (intended as illustrative and not limiting), it should be noted that modifications and variations may occur to those skilled in the art in light of the above teachings. It is therefore to be understood that changes may be made in the particular embodiments described which are within the scope of the invention as outlined by the appended claims.

Claims

1. 1. A computer-implemented method for preparing training data for a speech recognition model, comprising: Obtaining a number of sentences from the corpus; splitting each phoneme in each sentence of the plurality of sentences into three hidden states; calculating a score for each sentence of the plurality of sentences based on the amount of variation in durations of the three hidden states for each phoneme in the sentence; sorting the plurality of sentences using the calculated scores; and 4. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein each score is calculated by using an average duration of the three hidden states for each phoneme in the sentence.

3. 2. The computer-implemented method of claim 1, wherein each score is calculated by using durations of each of the three hidden states of each phoneme in the sentence previously estimated from clean speech.

4. 4. The computer-implemented method of claim 1, wherein the scores are phoneme-independent uniform scores if statistics from clean speech are not available.

5. The phoneme independent uniform score is [0010] is given as d ｎ，ｘ is the duration of the xth state of the nth phoneme, where x can be initial (b), medial (m), or final (e), and μ ｎ But, d ｎ，ｘ 5. The computer-implemented method of claim 4, wherein n is the average duration of the three hidden states of the nth phoneme obtained by:

6. 6. The computer-implemented method of claim 1, wherein the score is a phoneme-dependent uniform score when statistics from clean speech are available.

7. The phoneme-dependent uniform score is [0025] is given as λ ｋ（ｎ），ｘ 7. The computer-implemented method of claim 6, wherein x(n) is an ideal duration of the x-th state of phoneme k(n) for which the n-th phoneme is to be referenced, estimated a priori from the clean speech, and k(n) is a function that maps the n-th phoneme to a corresponding phoneme.

8. 1. A computer program product for preparing training data for a speech recognition model, the computer program product including a computer readable storage medium having program instructions embodied therein, the program instructions causing a computer to: Get multiple sentences from the corpus, Segmenting each phoneme in each sentence of the plurality of sentences into three hidden states; calculating a score for each sentence of the plurality of sentences based on an amount of variation in duration of the three hidden states for each phoneme in the sentence; A computer program product executable by the computer to cause the plurality of sentences to be sorted by using the calculated scores.

9. 9. The computer program product of claim 8, wherein each score is calculated by using an average duration of the three hidden states for each phoneme in the sentence.

10. 9. The computer program product of claim 8, wherein each score is calculated by using durations of each of the three hidden states of each phoneme in the sentence previously estimated from clean speech.

11. 11. A computer program product according to any one of claims 8 to 10, wherein the score is a phoneme independent uniform score if statistics from clean speech are not available.

12. The phoneme independent uniform score is [0030] is given as d ｎ，ｘ is the duration of the xth state of the nth phoneme, where x can be initial (b), medial (m), or final (e), and μ ｎ But, d ｎ，ｘ 12. The computer program product of claim 11, wherein N is the average duration of the three hidden states of the nth phoneme obtained by:

13. 13. A computer program product according to any one of claims 8 to 12, wherein the score is a phoneme-dependent uniform score if statistics from clean speech are available.

14. The phoneme-dependent uniform score is [0045] is given as λ ｋ（ｎ），ｘ 14. The computer program product of claim 13, wherein x(n) is an ideal duration of the x-th state of phoneme k(n) for which the n-th phoneme is to be referenced, previously estimated from the clean speech, and k(n) is a function that maps the n-th phoneme to a corresponding phoneme.

15. 1. A system for preparing training data for a speech recognition model, comprising: Memory, one or more processors in communication with the memory, Take multiple sentences from the corpus, Dividing each phoneme in each sentence of the plurality of sentences into three hidden states; calculating a score for each sentence of the plurality of sentences based on an amount of variation in durations of the three hidden states for each phoneme in the sentence; sorting the plurality of sentences by using the calculated scores; A processor configured to: A system comprising:

16. 16. The system of claim 15, wherein each score is calculated by using an average duration of the three hidden states for each phoneme in the sentence.

17. 16. The system of claim 15, wherein each score is calculated by using the durations of each of the three hidden states of each phoneme in the sentence previously estimated from clean speech.

18. 18. The system of claim 15, wherein the score is a phoneme independent uniform score if statistics from clean speech are not available.

19. The phoneme independent uniform score is [0050] is given as d ｎ，ｘ is the duration of the xth state of the nth phoneme, where x can be initial (b), medial (m), or final (e), and μ ｎ But, d ｎ，ｘ 20. The system of claim 18, wherein n is the average duration of the three hidden states of the nth phoneme obtained by:

20. The score is a phoneme-dependent uniform score when statistics from clean speech are available, the phoneme-dependent uniform score being [006] is given as λ ｋ（ｎ），ｘ 20. The system of claim 15, wherein x(n) is an ideal duration of the x-th state of a phoneme k(n) for which the n-th phoneme is to be referred, estimated beforehand from the clean speech, and k(n) is a function that maps the n-th phoneme to a corresponding phoneme.

21. A computer program comprising program code means adapted to perform the method according to any one of claims 1 to 7, when said program runs on a computer.

Images