Accuracy of a data stream conversion function for recurrent neural networks

Abstract

A computer-implemented method for training models, the method comprising: Training (220) a second continuous neural speech recognition model, which has a bidirectional coding function (520), to output the same symbols from an output probability grid of the second continuous neural speech recognition model as from an output probability grid of a trained first continuous neural speech recognition model, which has a unidirectional coding function (420); and Creating a third continuous neural speech recognition model that has a unidirectional coding function (420) by training the third continuous speech recognition model as a student in a knowledge distillation procedure using the trained second continuous neural speech recognition model as a teacher (230), wherein the first and third continuous neural speech recognition models transfer output data from an RNN conversion function in a data stream, and the second continuous neural speech recognition model provides the output data offline.

Description

BACKGROUND

[0001] The present invention relates generally to information processing and, more particularly, to improving the accuracy of a streaming recurrent neural network (RNN) transducer data stream transduction function for end-to-end speech recognition.

[0002] End-to-end training of RNN transducer models (RNN-T models) does not require frame-level mappings between audio and output symbols. As a result, corresponding nodes in the posterior lattices defined by the joint networks of different RNN-T models may emit different symbols, introducing a number of new challenges for knowledge distillation between RNN-T models. In particular, these discrepancies in the posterior lattices are pronounced between an offline RNN-T model and a datastream RNN-T model, because the datastream RNN-T model emits the symbols later than the offline RNN-T model.

[0003] The paper "Towards real-time mispronunciation detection in kids' speech" by Peter Plantinga and Eric Fosler-Lussier, published in the 2019 IEEE automatic speech recognition and understanding workshop (ASRU), December 14-18, 2019. IEEE, 2019. pp. 790-696. ISBN 978-1-7281-0306-8, describes a deep learning mispronunciation detection approach that uses two loss terms to improve model alignments. One loss is an "alignment loss" term that boosts the output only when features do not resemble silence. The other loss term uses a unidirectional model as a teacher model to align the bidirectional model.

[0004] The paper “Improved training for online end-to-end speech recognition systems” by Suyoun Kim et al., published on November 6, 2017, at https: / / arxiv.org / pdf / 1711.02212v1 [accessed November 8, 2024], describes an initialization strategy that uses teacher-student learning to transfer knowledge from a large, well-trained, offline end-to-end speech recognition model to an online end-to-end model. SUMMARY

[0005] According to aspects of the present invention, a computer-implemented method for training models is provided. The method comprises training a second end-to-end neural speech recognition model having a bidirectional encoder to output the same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of a trained first end-to-end neural speech recognition model having a unidirectional encoder.The method also includes creating a third end-to-end neural speech recognition model having a unidirectional coding function by training the third end-to-end neural speech recognition model as a student in a knowledge distillation process using the trained second end-to-end neural speech recognition model as a teacher.

[0006] According to further aspects of the present invention, a computer program product for training models is provided. The computer program product comprises a non-transitory, computer-readable storage medium embodying program instructions, the program instructions being executable by a computer to cause the computer to perform a method. The method comprises training a second end-to-end neural speech recognition model having a bidirectional encoder to output the same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of a trained first end-to-end neural speech recognition model having a unidirectional encoder.The method also includes creating a third end-to-end neural speech recognition model having a unidirectional coding function by training the third end-to-end neural speech recognition model as a student in a knowledge distillation process using the trained second end-to-end neural speech recognition model as a teacher.

[0007] According to yet further aspects of the present invention, a computer processing system for training models is provided. The computer processing system includes a memory unit for storing program code. The computer processing system further includes a hardware processor operatively connected to the memory unit to execute the program code to train a second end-to-end neural speech recognition model having a bidirectional coding function to output the same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of a trained first end-to-end neural speech recognition model having a unidirectional coding function.The hardware processor also executes the program code to create a third end-to-end neural speech recognition model that has a unidirectional coding function by training the third end-to-end neural speech recognition model as a student in a knowledge distillation process using the trained second end-to-end neural speech recognition model as a teacher.

[0008] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which should be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the following description, details of the preferred embodiments are provided with reference to the following figures, in which: Fig. 1 shows a block diagram illustrating an exemplary data processing unit according to an embodiment of the present invention; Fig. 2 to 3 illustrate an exemplary method for improving the accuracy of a data stream RNN mapping function for end-to-end speech recognition, according to an embodiment of the present invention; Fig. 4 shows an exemplary unidirectional RNN-T architecture according to an embodiment of the present invention; Fig. 5 shows an exemplary bidirectional RNN-T architecture according to an embodiment of the present invention; Fig. 6 shows a block diagram further illustrating the output probability grid of Fig. 4 and Fig. 5 according to an embodiment of the present invention; Fig. 7 shows a block diagram illustrating the steps performed in a block of the method 200 of Fig. 2 according to an embodiment of the present invention; Fig. 8 shows a block diagram illustrating the steps described in another block of the method 200 of Fig. 2 according to an embodiment of the present invention; Fig. 9 shows a block diagram illustrating the steps in yet another block of the method 200 of Fig. 2 according to an embodiment of the present invention; Fig. 10 shows a block diagram further showing the in a block of Fig. 2 graphically illustrates the RNN-T elements included in accordance with an embodiment of the present invention; Fig. 11 shows a block diagram illustrating an exemplary cloud computing environment having one or more cloud computing nodes with which local computing devices used by cloud users exchange data, according to an embodiment of the present invention; and Fig. 12 shows a block diagram illustrating a set of functional abstraction layers provided by a cloud computing environment, according to an embodiment of the present invention. DETAILED DESCRIPTION

[0010] Embodiments of the present invention relate to improving the accuracy of a streaming recurrent neural network (RNN) transducer data stream transducer for end-to-end speech recognition.

[0011] According to one embodiment of the present invention, a method is proposed for training an RNN-T model such that the nodes of its posterior lattice emit the same symbols as the corresponding nodes of the posterior lattice of the already trained RNN-T model. This method can be used to train an offline RNN-T model that can serve as a suitable teacher for training a student data stream RNN-T model.

[0012] It should be noted that although the embodiments presented herein primarily describe RNN-T models, other models are also possible, such as, but not limited to, transformer transducer, RNN transducer with stateless prediction network, and so on.

[0013] Fig. Figure 1 shows a block diagram illustrating an exemplary computing unit 100 according to an embodiment of the present invention. The computing unit 100 is configured to improve the accuracy of a data stream RNN mapping function model for end-to-end speech recognition.

[0014] The computing device 100 may be embodied as any type of computing device capable of performing the functions described herein, including, but not limited to, a computer, a server, a rack-mounted server, a blade server, a workstation, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile computing device, a portable computing device, a network device, a web device, a distributed computing system, a processor-based system, and / or a consumer electronics device. Additionally or alternatively, the computing device 100 may be embodied as one or more computing sleds, storage sleds, or other racks, sleds, computing enclosures, or other components of a physically compartmentalized computing device. As described in Fig. 1, the computing device 100 includes, by way of example, the processor 110, an input / output subsystem 120, a memory 130, a data storage unit 140, and a data transmission subsystem 150, and / or other components and units typically found in a server or similar computing device. It should be understood that in other embodiments, the computing device 100 may also include other or additional components, for example, those typically found in a server computer (e.g., various input / output units). Furthermore, in some embodiments, one or more of the example components may be integrated into or otherwise form part of another component. For example, in some embodiments, the memory 130 or portions thereof may be integrated into the processor 110.

[0015] Processor 110 may be embodied as any type of processor capable of performing the functions described herein. Processor 110 may be embodied as a single processor, multiple processors, central processing units (CPU(s)), graphics processing unit(s) (GPU(s)), single or multi-core processor(s), digital signal processor(s), microcontroller, or other processor(s) or processing / control circuit(s).

[0016] The memory 130 may be embodied as any type of volatile or non-volatile memory or data storage suitable for performing the functions described herein. In operation, the memory 130 may store various data and software used during operation of the computing device 100, for example, operating systems, applications, programs, libraries, and drivers. The memory 130 is coupled to the processor 110 for data exchange via the I / O subsystem 120, which may be embodied in the form of circuits and / or components to enable input / output operations with the processor 110, the memory 130, and other components of the computing device 100. For example, the I / O subsystem 120 may be embodied in the form of memory control hubs, input / output control hubs, platform control hubs, integrated control circuits, firmware units, data transmission links (e.g.,The I / O subsystem 120 may be embodied or include interface devices (e.g., point-to-point connections, bus connections, wires, cables, optical fibers, printed circuit traces, etc.) and / or other components and subsystems to enable the input / output operations. In some embodiments, the I / O subsystem 120 may be part of a system-on-a-chip (SOC) and may be integrated onto a single integrated circuit chip along with the processor 110, the memory 130, and other components of the computing device 100.

[0017] The data storage unit 140 may be embodied as any type of unit or units configured for the short- or long-term storage of data, such as memory units and circuits, memory cards, hard disk drives, solid-state media, or other data storage units. The data storage unit 140 may store program code for one or more datastream RNN mapping function models (or for improving the accuracy of a datastream RNN mapping function model) for end-to-end speech recognition. The data transmission subsystem 150 of the data processing unit 100 may be embodied in the form of any network interface controller or other data transmission circuitry, unit, or collection thereof suitable for enabling data transmissions between the data processing unit 100 and other remotely located units over a network.The data transmission subsystem 150 may be configured to implement one or more data transmission technologies (e.g., wired or wireless data transmission) and associated protocols (e.g., Ethernet, InfiniBand. ® , Bluetooth ® , Wi-Fi ® , WiMAX, etc.) to carry out such data transmission.

[0018] As illustrated, computing device 100 may also include one or more peripheral devices 160. Peripheral devices 160 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 160 may include a display, a touch screen, graphics circuitry, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0019] Of course, the data processing unit 100 may also include other elements (not shown) as can be easily considered by a person skilled in the art, and certain elements may also be omitted. For example, the data processing unit 100 may include various other input units and / or output units, depending on the particular implementation thereof, as will be readily understood by a person skilled in the art. For example, various types of wireless and / or wired input units and / or output units may be used. Furthermore, additional processors, controllers, memories, etc., may be employed in various configurations. In another embodiment, a cloud configuration may further be used (see, for example, Fig. 11 to 12). These and other variations of the processing system 100 will be readily contemplated by one skilled in the art in light of the teachings of the present invention provided herein.

[0020] As used herein, the term "hardware processor subsystem" or "hardware processor" may refer to a processor, memory (e.g., RAM, cache(s), etc.), software (including, but not limited to, memory management software), or combinations thereof that work together to perform one or more specific tasks. In suitable embodiments, the hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution units, etc.). The one or more data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or computational element-based controller (e.g., logic gates, etc.). The hardware processor subsystem may include one or more on-board memories (e.g., caches, dedicated memory arrays, read-only memories, etc.).In some embodiments, the hardware processor subsystem may include one or more memories that may be on-board or off-board, or that are specifically dedicated for use by the hardware processor subsystem (e.g., ROM, RAM, BIOS (basic input / output system), etc.).

[0021] In some embodiments, the hardware processor subsystem may include and execute one or more software elements. The one or more software elements may include an operating system and / or one or more applications and / or specific code to achieve a particular result.

[0022] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specific result. Such circuitry may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0023] These and other variations of a hardware processor subsystem are also contemplated according to embodiments of the present invention

[0024] Fig. 2 through 3 illustrate an exemplary method 200 for improving the accuracy of a data stream RNN mapping function for end-to-end speech recognition, according to an embodiment of the present invention.

[0025] In block 210, a first end-to-end neural speech recognition model is trained, which has a unidirectional coding function. In one embodiment, the first end-to-end neural speech recognition model can be considered a reference for an end-to-end neural speech recognition model.

[0026] In block 220, a second end-to-end neural speech recognition model is trained, which has a bidirectional encoding function to output the same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of the trained first end-to-end neural speech recognition model. In one embodiment, the training may be performed with respect to a loss for the end-to-end neural speech recognition model (RNN-T loss) and a cross-entropy loss. In one embodiment, the loss for the end-to-end neural speech recognition model is determined by reference to labels in the training data. In one embodiment, the cross-entropy loss is calculated using the first best (1-best) output symbols of the unidirectional model trained in block 210.In one embodiment, each of the nodes of the output probability lattice may represent a softmax operation. In one embodiment, the second end-to-end neural speech recognition model has a stronger or more complex configuration (e.g., a deeper neural network) than the first end-to-end neural speech recognition model.

[0027] In one embodiment, block 220 may include one or more of blocks 220A and 220B.

[0028] In block 220A, a first best symbol is determined for each node of the output probability lattice. This is achieved by inputting the training audio data to the first end-to-end reference speech recognition model and selecting the first best symbol from each node of the output a posteriori lattice.

[0029] In block 220B, a loss for the end-to-end neural speech recognition model and also a cross-entropy loss with respect to the first best symbols are minimized.

[0030] In 220C, the loss for the end-to-end neural speech recognition model and the cross-entropy loss are minimized until both losses are sufficiently small, for example, below respective thresholds.

[0031] In block 230, a third end-to-end neural speech recognition model, which has a unidirectional coding function, is trained as a student in a knowledge distillation process using the trained second end-to-end neural speech recognition model as a teacher. In one embodiment, the third end-to-end neural speech recognition system can be trained until the similarity between the output probability lattices of the third and second end-to-end neural speech recognition models is within a certain similarity range.

[0032] In one embodiment, block 230 includes one or more of blocks 230A, 230B, 230C, and 230D.

[0033] In block 230A, training audio data is input to the trained second end-to-end neural speech recognition model to determine the corresponding output probability lattice.

[0034] In block 230B, less likely nodes of the output probability lattice are hidden with respect to a minimum probability threshold.

[0035] In block 230C, (i) the loss for the end-to-end neural speech recognition model and (ii) a relative divergence across non-obscured portions of the output probability grid to an output probability grid of the third end-to-end speech recognition model are jointly minimized. The relative divergence may be a Kullback-Leibler divergence or another divergence.

[0036] In block 230D, the output probability lattice of the third end-to-end neural speech recognition model is evaluated based on a similarity to an output probability lattice of the second end-to-end speech recognition system. For example, the similarity may be based on KL divergence and / or the like.

[0037] In block 240, speech recognition is performed by performing a beam search over an output probability lattice of the trained third end-to-end neural speech recognition model.

[0038] In one embodiment, the first and third end-to-end neural speech recognition models transmit output data from an RNN implementation function in a data stream, and the second end-to-end neural speech recognition model provides the output data offline.

[0039] The first end-to-end neural speech recognition model can be considered the reference model. The second end-to-end neural speech recognition model can be considered the teacher model. The third end-to-end neural speech recognition model can be considered the student model.

[0040] The first, second, and third end-to-end neural speech recognition models each use an RNN-T (Recurrent Neural Network Transducer) architecture. The first and third end-to-end neural speech recognition models are unidirectional models, while the second end-to-end neural speech recognition model is bidirectional. The architectures are described in more detail below.

[0041] Fig. 4 shows an exemplary unidirectional RNN-T architecture 400 according to an embodiment of the present invention.

[0042] The unidirectional RNN-T architecture 400 includes a prediction network 410, a unidirectional coding block 420, a shared network 430, a softmax block 440, and an output probability grid 450.

[0043] The unidirectional coding block 420 receives an input sequence of functions x = (x1, ..., x T ).

[0044] Based on a search over an output probability lattice defined by P(y|t,u), the unidirectional RNN-T architecture 400 generates an output sequence y. In one embodiment, y is a set of possible symbols, t is a time index, and u is a history of transmitted symbols.

[0045] In detail, the unidirectional coding block 420 receives the input sequence of functions x = (x1, ..., x T ) and gives a coding htenc out of.

[0046] The prediction network 410 receives a previous prediction y u - 1 and gives a prediction hupred out of.

[0047] The common network 430 receives htenc and hupred and gives z t,u out of.

[0048] The softmax block receives z t,u and a softmax of z t,u .

[0049] The output probability grid defined by P(y|t,u) receives the softmax of z t,u and returns the sequence y = (y1, ... , y u ). out of.

[0050] Fig. 5 shows an exemplary bidirectional RNN-T architecture 500 according to an embodiment of the present invention.

[0051] The RNN-T architecture 500 includes a prediction network 510, a bidirectional coding block 520, a shared network 530, a softmax block 540, and an output probability grid 550.

[0052] Fig. Figure 6 shows a block diagram further illustrating the output probability grid of Fig. 4 and Fig. 5 according to an embodiment of the present invention.

[0053] The output probability grid 450 / 550 can be viewed relative to an x-axis and a y-axis such that x = (x1, ... , x T ) and y = (y1, ..., y U ).

[0054] Each node in the output probability grid 450 / 550 represents a softmax of z t,u represents.

[0055] Speech recognition with the RNN-T architecture 400 is performed by beam searching over the output probability grid 450 / 550.

[0056] Fig. 7 shows a block diagram illustrating the steps performed in block 210 of the method 200 of Fig. 2 according to an embodiment of the present invention.

[0057] The elements included in block 210 are a training data database 710 and a unidirectional RNN-T 720. The unidirectional RNN-T 720 may be constructed to have a similar architecture to the RNN-T architecture 300 of Fig. 3.

[0058] Fig. Figure 8 shows a block diagram illustrating the steps performed in block 220 of the method 200 of Fig. 2 according to an embodiment of the present invention.

[0059] The elements included in block 220 are the training data database 710 and the unidirectional RNN-T 720 as well as the bidirectional RNN-T 730. The bidirectional RNN-T 720 may be constructed to have a similar architecture to the RNN-T architecture 500 of Fig. 5.

[0060] The first best symbol for each node of the output probability lattice is output by the unidirectional RNN-T 720. This is used to train the bidirectional RNN-T 730.

[0061] Fig. 9 shows a block diagram illustrating the steps performed in block 230 of the method 200 of Fig. 2 according to an embodiment of the present invention.

[0062] The elements included in block 230 are the training data database 710, the unidirectional RNN-T 720, the bidirectional RNN-T 730, and another unidirectional RNN-T 740. The unidirectional RNN-T 740 may be constructed to have a similar architecture to the RNN-T architecture 300 of Fig. 3.

[0063] The output of the trained bidirectional RNN-T (730) is used to train the unidirectional RNN-T (below). A posteriori is computed for each node of the output grid of 730.

[0064] Fig. 10 shows a block diagram further illustrating the block 220 of Fig. 2 according to an embodiment of the present invention.

[0065] The elements contained in block 220 are the training data database 1010, the unidirectional RNN-T 1020 and the bidirectional RNN-T 1030.

[0066] The unidirectional RNN-T 1020, which can be considered as a reference RNN-T, includes a unidirectional prediction network 1020A, a unidirectional coding network 1020B, and an output probability grid 1020C.

[0067] The bidirectional RNN-T 1030, which can be considered as a teacher RNN-T, includes a deep prediction network 1030A, a bidirectional / deep coding network 1030B, and an output probability grid 1030C.

[0068] The reference number 1020C denotes the output grid of the reference model, while 1020D is the first best output selected from the output grid 1020C. Therefore, on the right side of Fig. 10 the same model 1020 is involved, only essentially presented at different time steps with respect to determining the first best outputs.

[0069] It should be noted that some of the blocks of the output probability grids can be hidden, even if this is not possible in Fig. 10 is not shown. In one embodiment, hidden blocks are not taken into account when training models. This hiding is used when training the third network with reference to the second network.

[0070] In operation, the unidirectional RNN-T 1020 and the bidirectional RNN-T 1030 receive training samples from the training data database 1010. The unidirectional RNN-T 1020 selects the first best outputs for each node of the output probability lattice 1020C. Cross-entropy is minimized between the bidirectional RNN-T 1030 and the unidirectional RNN-T 1040.

[0071] It should be understood that, although this disclosure includes a detailed description of cloud computing, implementation of the teachings herein is not limited to a cloud computing environment. Rather, embodiments of the present invention may be practiced in conjunction with any type of computing environment now known or later invented.

[0072] Cloud computing is a service delivery model for enabling seamless, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a service provider. This cloud model can include at least five characteristics, at least three service models, and at least four implementation models.

[0073] The properties are as follows: On-demand self-service: A cloud user can unilaterally and automatically provision data processing functions such as server time and network storage as needed, without requiring human interaction with the service provider. Broad Network Access: Functions are available over a network and accessed through standard mechanisms that support use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs). Resource pooling: The provider's computing resources are pooled to serve multiple users using a multi-tenant model, with various physical and virtual resources dynamically allocated and reassigned as needed. There is a perceived location independence, as the user generally has no control or knowledge over the exact location of the provided resources, but may be able to specify a location at a higher level of abstraction (e.g., country, state, or data center). Rapid Elasticity: Features can be deployed quickly and elastically for rapid horizontal scaling out, in some cases automatically, and released quickly for rapid scale-in. To the user, the features available for deployment often appear unlimited, and they can be purchased in any quantity at any time. Measured Service: Cloud systems automatically control and optimize resource usage by leveraging a measurement function at a certain level of abstraction appropriate for the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, creating transparency for both the provider and the user of the service.

[0074] The service models are as follows: Software as a Service (SaaS): The functionality provided to the user is to use the provider's applications running in a cloud infrastructure. The applications are accessible from various client devices via a thin client interface such as a web browser (e.g., web-based email). The user does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, with the possible exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The functionality provided to the user is to deploy applications created or obtained by the user, using programming languages ​​and tools supported by the provider, into the cloud infrastructure. The user does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but has control over the deployed applications and possibly over configurations of the application hosting environment. Infrastructure as a Service (IaaS): The functionality provided to the user consists of providing processing, storage, networking, and other basic computing resources, allowing the user to deploy and run any software, including operating systems and applications. The user does not manage or control the underlying cloud infrastructure, but has control over operating systems, storage, deployed applications, and possibly limited control over selected network components (e.g., host firewalls).

[0075] The deployment models are as follows: Private Cloud: The cloud infrastructure is operated solely for an organization. It can be managed by the organization or a third party and can be located on its own premises or on a third-party site. Community Cloud: The cloud infrastructure is shared by multiple organizations and supports a specific user community with common concerns (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organizations or a third party and can be located on their own premises or on someone else's premises. Public Cloud: The cloud infrastructure is made available to the general public or a large industry group and is owned by an organization that sells cloud services. Hybrid Cloud: Cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain separate entities but are interconnected by a standardized or proprietary technology that enables data and application portability (e.g., cloud targeting for load balancing between clouds).

[0076] A cloud computing environment is service-oriented, focusing on state independence, low coupling, modularity, and semantic interoperability. At the heart of cloud computing lies an infrastructure consisting of a network of interconnected nodes.

[0077] With reference to Fig. 11 depicts the illustrative cloud computing environment 1150. As shown, the cloud computing environment 1150 includes one or more cloud computing nodes 1110 with which local computing devices used by cloud users, such as the personal digital assistant (PDA) or mobile phone 1154A, the desktop computer 1154B, the laptop computer 1154C, and / or the automotive computer system 1154N, can communicate. The nodes 1110 can communicate with each other. They can be physically or virtually grouped into one or more networks, such as private, community, public, or hybrid clouds (not shown), as described above, or a combination thereof. This enables the cloud computing environment 1150 to offer infrastructure, platforms, and / or software as a service, for which a cloud user does not need to maintain resources on a local computing device.It is understood that the types of in . Fig. 11 are intended to be illustrative only, and that the computing nodes 1110 and the cloud computing environment 1150 may communicate with any type of computer-based device via any type of network and / or via any type of network-accessible connection (e.g., using a web browser).

[0078] With reference to Fig. 12 shows a set of functional abstraction layers used by the cloud computing environment 1150 ( Fig. 11). It is understood that the information in Fig. The components, layers, and functions shown in Figure 12 are intended to be illustrative only, and embodiments of the invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0079] A hardware and software layer 1260 includes hardware and software components. Examples of hardware components include mainframe computers 1261; Reduced Instruction Set Computer (RISC) architecture-based servers 1262; servers 1263; blade servers 1264; storage devices 1265; and networks and network components 1266. In some embodiments, software components include network application server software 1267 and database software 1268.

[0080] The virtualization layer 1270 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 1271; virtual storage 1272; virtual networks 1273, including virtual private networks, virtual applications and operating systems 1274; and virtual clients 1275.

[0081] In one example, the management layer 1280 may provide the functions described below. Resource provisioning 1281 provides for the dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 1282 provides for cost tracking of using resources within the cloud computing environment, as well as billing or invoicing for the use of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud users and tasks, as well as protection for data and other resources. A user portal 1283 provides users and system administrators with access to the cloud computing environment.Service level management 1284 provides for the allocation and management of cloud computing resources so that the required service objectives are met. Service level agreement (SLA) planning and fulfillment 1285 provides for the pre-arranging and procurement of cloud computing resources for which future requirements are anticipated, according to an SLA.

[0082] A workload layer 1290 provides examples of the functionality for which the cloud computing environment can be used. Examples of workloads and functions that can be provided by this layer include: mapping and navigation 1291; software development and lifecycle management 1292; providing education in virtual classrooms 1293; data analytics processing 1294; transaction processing 1295; and increasing accuracy in an RNN-T for end-to-end speech recognition 1296.

[0083] The present invention may be a system, a method, and / or a computer program product at any possible technical level of integration. The computer program product may comprise a computer-readable storage medium (or media) having computer-readable program instructions stored thereon for causing a processor to carry out aspects of the present invention.

[0084] The computer-readable storage medium may be any physical device capable of retaining and storing instructions for use by an instruction-executing system. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of the computer-readable storage medium includes the following: a portable computer diskette, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM).Flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device such as punched cards or raised structures in a groove on which instructions are stored, and any suitable combination thereof. A computer-readable storage medium, as used herein, shall not be construed as carrying transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., pulses of light traveling through an optical fiber cable), or electrical signals carried through a wire.

[0085] Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to respective computing / processing units or to an external computer or storage unit via a network such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, fiber optic transmission lines, wireless transmission, routers, firewalls, switching units, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing unit receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing unit.

[0086] Computer-readable program instructions for performing operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as SMALLTALK, C++, or the like, as well as conventional procedural programming languages ​​such as the C programming language or similar programming languages.The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, over the Internet using an Internet service provider).In some embodiments, electronic circuits, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute the computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuits to perform aspects of the present invention.

[0087] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, may be implemented by computer-readable program instructions.

[0088] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce a means for implementing the functions / steps defined in the flowchart and / or block diagram block(s).These computer-readable program instructions may also be stored on a computer-readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that implement aspects of the function / step defined in the block(s) of the flowchart and / or block diagrams.

[0089] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of process steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-executable process such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / steps defined in the block(s) of flowcharts and / or block diagrams.

[0090] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions comprising one or more executable instructions for performing the particular logical function(s). In some alternative implementations, the functions specified in the block may occur in a different order than shown in the figures. For example, two blocks shown in succession may actually execute substantially concurrently, or the blocks may sometimes execute in reverse order depending on the corresponding functionality.It is further to be understood that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by special purpose hardware-based systems that perform the specified functions or steps, or by combinations of special purpose hardware and computer instructions.

[0091] Reference in the specification to "a single embodiment" or "an embodiment" of the present invention, as well as other variations thereof, means that a particular function, structure, or feature described in connection with the embodiment is included in at least one embodiment of the present invention. Therefore, the appearances of the phrase "in a single embodiment" or "in an embodiment" and any other variations thereof in various places in the specification do not necessarily refer to the same embodiment.

[0092] It should be noted that the use of " / ", "and / or" and "at least one of", for example, in the case of "A / B", "A and / or B" or "at least one of A and B", is intended to cover only the selection of the first listed option (A), or only the selection of the second listed option (B), or both options (A and B). As a further example, in the case of "A, B and / or C" or "at least one of A, B and C", this wording is intended to cover only the selection of the first listed option (A), or only the selection of the second listed option (B), or only the selection of the third listed option (C), or only the selection of the first and second listed options (A and B), or only the selection of the first and third listed options (A and C), or only the selection of the second and third listed options (B and C), or all three options (A and B and C).This can be extended to any number of points, as will be readily apparent to a person skilled in this and related techniques.

[0093] Having described preferred embodiments of a system and method (which are intended to be illustrative and not limiting), it is understood that modifications and variations may be made by those skilled in the art in light of the above teachings. Therefore, it is to be understood that changes may be made to the particular disclosed embodiments without departing from the scope of the invention as described in the appended claims. Having thus described aspects of the invention with the particularity and specificity required by the patent laws, what is claimed and intended to be protected by the specification is set forth in the appended claims.

Claims

[1] A method for training models implemented on a computer, the method comprising: Training (220) a second end-to-end neural speech recognition model having a bidirectional coding function (520) to output the same symbols from an output probability grid of the second end-to-end neural speech recognition model as from an output probability grid of a trained first end-to-end neural speech recognition model having a unidirectional coding function (420); and Creating a third end-to-end neural speech recognition model having a unidirectional coding function (420) by training the third end-to-end neural speech recognition model as a student in a knowledge distillation process using the trained second end-to-end neural speech recognition model as a teacher (230), wherein the first and third end-to-end neural speech recognition models transmit output data from an RNN implementation function in a data stream, and the second end-to-end neural speech recognition model provides the output data offline. [2] The computer-implemented method of claim 1, wherein training the second end-to-end speech recognition model comprises: Determining a first best symbol for each node of the output probability lattice of the first end-to-end neural speech recognition model by inputting the training audio data to the first end-to-end reference speech recognition model, selecting the first best symbol from each node of the output probability lattice, and outputting the first best symbol for training the bidirectional coding function for each node of the unidirectional coding function; and Minimize a loss for the end-to-end neural speech recognition model and also a cross-entropy loss with respect to the first best symbol. [3] A computer-implemented method according to claim 2, wherein a weighted summation of the cross-entropy loss and the loss for the end-to-end speech recognition model is minimized. [4] The computer-implemented method of claim 2, wherein training the third end-to-end neural speech recognition model comprises: Hiding less probable nodes of the output probability grid of the second end-to-end neural speech recognition model with respect to a minimum probability threshold; and jointly minimizing, i, the loss for the end-to-end neural speech recognition model and, ii, a relative divergence over non-obscured parts of the output probability lattice of the second end-to-end neural speech recognition model to an output probability lattice of the third end-to-end speech recognition model. [5] A computer-implemented method according to claim 4, wherein the relative divergence is a Kullback-Leibler divergence. [6] The computer-implemented method of claim 1, wherein training the second end-to-end neural speech recognition model further comprises inputting training audio data to the trained first end-to-end neural speech recognition model to determine the output probability lattice of the second end-to-end neural speech recognition model. [7] The computer-implemented method of claim 1, wherein each of the nodes of the output probability lattice of the second end-to-end neural speech recognition model represents a softmax operation. [8] The computer-implemented method of claim 1, further comprising performing speech recognition by performing a beam search over an output probability lattice of the trained third end-to-end neural speech recognition model. [9] The computer-implemented method of claim 1, wherein the second end-to-end neural speech recognition model comprises a more complex neural network than the first end-to-end neural speech recognition model. [10] The computer-implemented method of claim 1, further comprising evaluating the output probability lattice of the third end-to-end neural speech recognition model based on a similarity to the output probability lattice of the second end-to-end speech recognition system. [11] The computer-implemented method of claim 1, wherein the third end-to-end neural speech recognition system is trained until the similarity between the output probability lattices of the third and second end-to-end neural speech recognition models is within a certain similarity range. [12] A computer-implemented method according to claim 1, wherein the method is performed by a speech recognition system. [13] The computer-implemented method of claim 1, wherein at least one of the first, second, and third neural speech recognition models comprises a model of a recurrent neural network mapping function. [14] The computer-implemented method of claim 1, wherein the second end-to-end neural speech recognition model comprises an ensemble of end-to-end neural speech recognition models, and the method further comprises selecting a best teacher model from the ensemble in response to an overlap of search paths through an output probability lattice of each of the end-to-end neural speech recognition models in the ensemble. [15] A computer program product for training models, the computer program product comprising a non-transitory computer-readable storage medium embodying program instructions, the program instructions being executable by a computer to cause the computer to perform a method comprising: Training (220) a second end-to-end neural speech recognition model having a bidirectional coding function (520) to output the same symbols from an output probability grid of the second end-to-end neural speech recognition model as from an output probability grid of a trained first end-to-end neural speech recognition model having a unidirectional coding function (420); and Creating a third end-to-end neural speech recognition model having a unidirectional coding function (420) by training the third end-to-end neural speech recognition model as a student in a knowledge distillation process using the trained second end-to-end neural speech recognition model as a teacher (230), wherein the first and third end-to-end neural speech recognition models transmit output data from an RNN implementation function in a data stream, and the second end-to-end neural speech recognition model provides the output data offline. [16] The computer program product of claim 15, wherein training the second end-to-end speech recognition model comprises: Determining a first best symbol for each node of the output probability lattice of the first end-to-end neural speech recognition model by inputting the training audio data to the first end-to-end reference speech recognition model, selecting the first best symbol from each node of the output probability lattice, and outputting the first best symbol for training the bidirectional coding function for each node of the unidirectional coding function; and Minimize a loss for the end-to-end neural speech recognition model and also a cross-entropy loss with respect to the first best symbol. [17] The computer program product of claim 16, wherein a weighted summation of the cross-entropy loss and the loss for the end-to-end speech recognition model is minimized. [18] The computer program product of claim 16, wherein training the third end-to-end neural speech recognition model comprises: Hiding less probable nodes of the output probability grid of the second end-to-end neural speech recognition model with respect to a minimum probability threshold; and jointly minimizing, i, the loss for the end-to-end neural speech recognition model and, ii, a relative divergence over non-obscured parts of the output probability lattice of the second end-to-end neural speech recognition model to an output probability lattice of the third end-to-end speech recognition model. [19] A computer processing system for training models, the computer processing system comprising: a memory unit for storing program code; and a hardware processor operatively connected to the memory unit, wherein program code is stored on the memory unit, wherein executing the program code stored on the memory unit causes the hardware processor to Training (220) a second end-to-end neural speech recognition model having a bidirectional coding function (520) to output the same symbols from an output probability grid of the second end-to-end neural speech recognition model as from an output probability grid of a trained first end-to-end neural speech recognition model having a unidirectional coding function (420); and Creating a third end-to-end neural speech recognition model having a unidirectional coding function (420) by training the third end-to-end neural speech recognition model as a student in a knowledge distillation process using the trained second end-to-end neural speech recognition model as a teacher (230), wherein the first and third end-to-end neural speech recognition models transmit output data from an RNN implementation function in a data stream, and the second end-to-end neural speech recognition model provides the output data offline.

Images