Accuracy of streaming RNN transducers

Abstract

A computer-based method for model training is provided. [Solution] This method includes training a second end-to-end neural speech recognition model having a bidirectional encoder so that it outputs the same symbols from the output stochastic grid of a second end-to-end neural speech recognition model as those from the output stochastic grid of a first end-to-end neural speech recognition model having a unidirectional encoder. This method also includes constructing a third end-to-end neural speech recognition model having a unidirectional encoder by training a third end-to-end neural speech recognition model as a student, using the trained second end-to-end neural speech recognition model as a teacher, in a knowledge distillation method.

Description

[Technical Field]

[0001] The present invention generally relates to information processing, and more particularly to improving the accuracy of streaming recurrent neural network (RNN) transducers for end-to-end neural speech recognition. [Background technology]

[0002] End-to-end training of RNN transducer (RNN-T) models does not require frame-level alignment between audio symbols and output symbols. As a result, corresponding nodes in the posterior lattice defined by a joint network of different RNN-T models may emit different symbols, which presents a new set of challenges for knowledge distillation between RNN-T models. These inconsistencies in the posterior lattice are particularly pronounced between offline and streaming RNN-T models because streaming RNN-T models emit symbols later than offline RNN-T models. [Overview of the project]

[0003] According to aspects of the present invention, a computer implementation method for model training is provided. This method includes training a second end-to-end neural speech recognition model having a bidirectional encoder so that it outputs the same symbols from the output stochastic grid of a second end-to-end neural speech recognition model as those from the output stochastic grid of a first end-to-end neural speech recognition model having a unidirectional encoder. The method also includes constructing a third end-to-end neural speech recognition model having a unidirectional encoder by training a third end-to-end neural speech recognition model as a student, using the trained second end-to-end neural speech recognition model as a teacher, in a knowledge distillation method.

[0004] According to another aspect of the present invention, a computer program product for model training is provided. This computer program product includes a non-transient computer-readable storage medium having program instructions embodied therewith, which can be executed by a computer to cause the computer to perform a method. The method includes training a second end-to-end neural speech recognition model having a bidirectional encoder to output the same symbols from the output stochastic grid of a second end-to-end neural speech recognition model as those from the output stochastic grid of a first end-to-end neural speech recognition model having a unidirectional encoder. The method also includes constructing a third end-to-end neural speech recognition model having a unidirectional encoder by training a third end-to-end neural speech recognition model as a student, using the trained second end-to-end neural speech recognition model as a teacher, in a knowledge distillation method.

[0005] According to yet another aspect of the present invention, a computer processing system for model training is provided. This computer processing system includes a memory device for storing program code. This computer processing system further includes a hardware processor operably coupled to the memory device, the hardware processor executing program code to train a second end-to-end neural speech recognition model having a bidirectional encoder so that it outputs the same symbols from the output stochastic grid of a second end-to-end neural speech recognition model having a bidirectional encoder as the output stochastic grid of a first end-to-end neural speech recognition model having a bidirectional encoder. This hardware processor also executes program code to construct a third end-to-end neural speech recognition model having a bidirectional encoder by training a third end-to-end neural speech recognition model as a student, using the trained second end-to-end neural speech recognition model as a teacher, in a knowledge distillation method.

[0006] These and other features and advantages will become apparent from the following detailed description of the explanatory embodiments, which should be read in relation to the attached drawings.

[0007] The following description provides details of a preferred embodiment with reference to the following figures. [Brief explanation of the drawing]

[0008] [Figure 1] Block diagram showing an exemplary computing device according to one embodiment of the present invention. [Figure 2] This figure shows an exemplary method for improving the accuracy of a streaming RNN transducer for end-to-end neural speech recognition, according to one embodiment of the present invention. [Figure 3]This figure shows an exemplary method for improving the accuracy of a streaming RNN transducer for end-to-end neural speech recognition, according to one embodiment of the present invention. [Figure 4] This figure shows an exemplary unidirectional RNN-T architecture according to one embodiment of the present invention. [Figure 5] This figure shows an exemplary bidirectional RNN-T architecture according to one embodiment of the present invention. [Figure 6] This is a block diagram further showing the output stochastic grid of Figures 4 and 5 according to one embodiment of the present invention. [Figure 7] This is a block diagram illustrating the elements included in the block of method 200 shown in Figure 2, according to one embodiment of the present invention. [Figure 8] This is a block diagram illustrating elements included in another block of Method 200 of Figure 2, according to one embodiment of the present invention. [Figure 9] This is a block diagram illustrating elements included in yet another block of method 200 of Figure 2, according to one embodiment of the present invention. [Figure 10] This is a block diagram illustrating, in a more graphical manner, the RNN-T elements included in the block shown in Figure 2, according to one embodiment of the present invention. [Figure 11] This is a block diagram illustrating a cloud computing environment for illustrative purposes, according to one embodiment of the present invention, having one or more cloud computing nodes with which a local computing device used by a cloud consumer communicates. [Figure 12] This is a block diagram showing a set of functional abstraction layers provided by a cloud computing environment, according to one embodiment of the present invention. [Modes for carrying out the invention]

[0009] Embodiments of the present invention aim to improve the accuracy of streaming recurrent neural network (RNN) transducers for end-to-end neural speech recognition.

[0010] According to embodiments of the present invention, a method is proposed for training an RNN-T model such that the nodes of its posterior distribution grid emit the same symbols as the corresponding nodes of the posterior distribution grid from a pre-trained RNN-T model. This method can be used to train an offline RNN-T model that can serve as an excellent teacher for training student streaming RNN-T models.

[0011] While the RNN-T model is primarily described in the embodiments outlined herein, it should be understood that other models include, but are not limited to, transformer transducers and RNN transducers with stateless predictive networks.

[0012] Figure 1 is a block diagram showing an exemplary computing device 100 according to one embodiment of the present invention. The computing device 100 is configured to improve the accuracy of a streaming RNN transducer model for end-to-end neural speech recognition.

[0013] Computing device 100 can be embodied as any type of computing device or computer device capable of performing the functions described herein, but is not limited to, computers, servers, rack-based servers, blade servers, workstations, desktop computers, laptop computers, notebook computers, tablet computers, mobile computing devices, wearable computing devices, network appliances, web appliances, distributed computing systems, processor-based systems, or consumer electronic devices, or combinations thereof. Additionally or alternatively, computing device 100 may be embodied as one or more computing threads, memory threads, or other racks, threads, computing chassis, or other components of physically subdivided computing devices. As shown in Figure 1, computing device 100 includes, for illustrative purposes, a processor 110, an input / output subsystem 120, memory 130, a data storage device 140, and a communications subsystem 150, or other components and devices commonly found in servers or similar computing devices, or combinations thereof. Of course, in other embodiments, the computing device 100 may include other or additional components, such as those commonly found in a server computer (e.g., various input / output devices). Furthermore, in some embodiments, one or more of the components used for illustrative purposes may be incorporated into another component or otherwise form part of another component. For example, in some embodiments, the memory 130 or a portion thereof may be incorporated into the processor 110.

[0014] Processor 110 may be embodied as any type of processor capable of performing the functions described herein. Processor 110 may be embodied as a single processor, multiple processors, a central processing unit (CPU), a graphics processing unit (GPU), a single-core or multi-core processor, a digital signal processor, a microcontroller, or other processor or processing / control circuitry.

[0015] Memory 130 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, memory 130 may store various data and software used during the operation of computing device 100, such as, for example, an operating system, applications, programs, libraries, and drivers. Memory 130 is communicatively coupled to processor 110 via I / O subsystem 120, which may be embodied as circuitry or components, or a combination thereof, to facilitate input / output operations between processor 110, memory 130, and other components of computing device 100. For example, I / O subsystem 120 may be embodied as, or otherwise include, a memory controller hub, an input / output control hub, a platform controller hub, an integrated control circuit, a firmware device, a communication link (e.g., a point-to-point link, a bus link, a wire, a cable, a light guide, a printed circuit board trace, etc.), or other components and subsystems to facilitate input / output operations, or a combination thereof. In some embodiments, I / O subsystem 120 forms part of a system-on-chip (SOC) and may be incorporated onto a single integrated circuit chip together with processor 110, memory 130, and other components of computing device 100.

[0016] The data storage device 140 can be embodied as one or more devices of any type configured for short-term or long-term storage of data, such as, for example, memory devices and circuits, memory cards, hard disk drives, solid state drives, or other data storage devices. The data storage device 140 can store program code (for improving the accuracy) of a streaming RNN transducer model for end-to-end neural speech recognition. The communication subsystem 150 of the computing device 100 can be embodied as any network interface controller or other communication circuit, device, or collection thereof that can enable communication between the computing device 100 and other remote devices on a network. The communication subsystem 150 can be configured to use any one or more communication technologies (such as wired or wireless communication) and associated protocols (such as Ethernet(R), InfiniBand(R), Bluetooth(R), Wi-Fi(R), WiMAX, etc.) to perform such communication.

[0017] As shown, the computing device 100 can also include one or more peripheral devices 160. The peripheral devices 160 can include any number of additional input / output devices, interface devices, or other peripheral devices, or combinations thereof. For example, in some embodiments, the peripheral devices 160 can include a display, touch screen, graphics circuit, keyboard, mouse, speaker system, microphone, network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0018] Naturally, the computing device 100 may include other elements (not shown) as readily conceivable by those skilled in the art, and certain elements may be omitted. For example, various other input devices or output devices, or both, may be included in the computing device 100 as readily understood by those skilled in the art. For example, various types of wireless or wired, or both, input / output devices may be used. Furthermore, additional processors, controllers, memory, etc., in various configurations may be utilized. In another embodiment, a cloud configuration may be used (see, for example, Figures 11-12). These and other variations of the processing system 100 will readily be conceivable by those skilled in the art, given the teachings of the present invention provided herein.

[0019] As used herein, the terms “hardware processor subsystem” or “hardware processor” may refer to a processor, memory (including RAM, cache, etc.), software (including memory management software), or a combination thereof, that work together to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). One or more data processing elements may be included in a central processing unit, a graphics processing unit, or a separate processor-based or computing element-based controller (e.g., logic gates, etc.), or a combination thereof. A hardware processor subsystem may include one or more onboard memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories that may be onboard or offboard, or that may be dedicated to use by the hardware processor subsystem (e.g., ROM, RAM, Basic Input / Output System (BIOS), etc.).

[0020] In some embodiments, a hardware processor subsystem may include and execute one or more software elements. These software elements may include an operating system and / or one or more applications and / or specific code for achieving a specified result.

[0021] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application-specific integrated circuits (ASICs), FPGAs, or PLAs, or a combination thereof.

[0022] Other variations of these hardware processor subsystems are also intended by embodiments of the present invention.

[0023] Figures 2 and 3 illustrate an exemplary method 200 for improving the accuracy of a streaming RNN transducer for end-to-end neural speech recognition, according to an embodiment of the present invention.

[0024] In block 210, a first end-to-end neural speech recognition model having a one-way encoder is trained. In one embodiment, the first end-to-end neural speech recognition model can be considered a reference end-to-end neural speech recognition model.

[0025] In block 220, a second end-to-end neural speech recognition model having a bidirectional encoder is trained to output the same symbols from the output stochastic grid of the second end-to-end neural speech recognition model as those from the output stochastic grid of the first end-to-end neural speech recognition model that were trained. In one embodiment, training can be performed for the end-to-end neural speech recognition model loss (RNN-T loss) and the cross-entropy loss. In one embodiment, the end-to-end neural speech recognition model loss is obtained by referring to labels in the training data. In one embodiment, the cross-entropy loss is calculated using the 1-best output symbols from the unidirectional model trained in block 210. In one embodiment, each node of the output stochastic grid can represent a softmax operation. In one embodiment, the second end-to-end neural speech recognition model has a more powerful and complex configuration (e.g., a deeper neural network) than the first end-to-end neural speech recognition model.

[0026] In one embodiment, block 220 may include one or more blocks 220A and 220B.

[0027] In block 220A, a one-best symbol is obtained for each node in the output stochastic grid. This is achieved by feeding training audio data into a first reference end-to-end neural speech recognition model and selecting a one-best symbol from each node in the output posterior distribution grid.

[0028] In block 220B, the cross-entropy loss with respect to the end-to-end neural speech recognition model and the 1-best symbol is minimized.

[0029] In 220C, the end-to-end neural speech recognition model loss and cross-entropy loss are minimized until both losses are sufficiently small (for example, until they fall below their respective thresholds).

[0030] In block 230, in the knowledge distillation method, a third end-to-end neural speech recognition model having a one-way encoder is trained as a student by using a trained second end-to-end neural speech recognition model as a teacher. In one embodiment, the third end-to-end neural speech recognition system can be trained until the similarity between the output probability grids of the third and second end-to-end neural speech recognition models falls within a specific similarity range.

[0031] In one embodiment, block 230 includes one or more of blocks 230A, 230B, 230C, and 230D.

[0032] In block 230A, training audio data is input to a trained second end-to-end neural speech recognition model to obtain the corresponding output stochastic grid.

[0033] In block 230B, nodes with low probabilities in the output probability grid are masked relative to the minimum probability threshold.

[0034] In block 230C, (i) the end-to-end neural speech recognition model loss and (ii) the relative divergence of the unmasked portion of the output stochastic grid with respect to the output stochastic grid of a third end-to-end neural speech recognition model are minimized together. The relative divergence may be a Kullback-Leibler divergence or other divergence.

[0035] In Block 230D, the output stochastic grid of a third end-to-end neural speech recognition model is evaluated based on its similarity to the output stochastic grid of a second end-to-end neural speech recognition system. For example, the similarity can be based on KL divergence, or other methods, or both.

[0036] In block 240, speech recognition is performed by running a beam search on the output stochastic grid of a trained third end-to-end neural speech recognition model.

[0037] In one embodiment, the first and third end-to-end neural speech recognition models stream output data from an RNN transducer, while the second end-to-end neural speech recognition model provides the output data offline.

[0038] The first end-to-end neural speech recognition model can be considered a reference model. The second end-to-end neural speech recognition model can be considered a training model. The third end-to-end neural speech recognition model can be considered a student model.

[0039] Each of the first, second, and third end-to-end neural speech recognition models employs a recurrent neural network transducer (RNN-T) architecture. The first and third end-to-end neural speech recognition models are unidirectional, while the second end-to-end neural speech recognition model is bidirectional. The architecture is described further below.

[0040] Figure 4 shows an exemplary unidirectional RNN-T architecture 400 according to one embodiment of the present invention.

[0041] The unidirectional RNN-T architecture 400 includes a prediction network 410, a unidirectional encoder block 420, a joint network 430, a softmax block 440, and an output stochastic grid 450.

[0042] The one-way encoder block 420 uses the input feature sequence x=(x1,...,x T ) is received.

[0043] Based on a search on an output stochastic grid defined by P(y|t,u), the one-way RNN-T architecture 400 generates an output sequence y. In one embodiment, y is the set of possible symbols, t is the time index, and u is the history of emitted symbols.

[0044] In detail, the one-way encoder block 420 uses the input feature sequence x = (x1,...,x T ) receive and encode

number

[0045] Prediction network 410 predicts the previous y u -1 received, predict

number

[0046] Joint Network 430 is

number

number

[0047] Softmax blocks are z t,uand z t,u receives the softmax of t,u .

[0048] The output probability lattice 450 defined by P(y|t,u) receives the softmax of z t,u and outputs a sequence y = (y1,..., y u ).

[0049] FIG. 5 shows an exemplary bidirectional RNN-T architecture 500 according to an embodiment of the present invention.

[0050] The RNN-T architecture 500 includes a prediction network 510, a bidirectional encoder block 520, a joint network 530, a softmax block 540, and an output probability lattice 550.

[0051] FIG. 6 is a block diagram further showing the output probability lattices of FIGS. 4 and 5 according to an embodiment of the present invention.

[0052] The output probability lattices 450 / 550 can be considered with respect to the x-axis and y-axis such that x = (x1,..., x T ) and y = (y1,..., y U ).

[0053] Each node of the output probability lattices 450 / 550 represents the softmax of z t,u .

[0054] Speech recognition by the RNN-T architecture 400 is realized by beam search for the output probability lattices 450 / 550.

[0055] FIG. 7 is a block diagram pictorially showing elements involved in block 210 of method 200 of FIG. 2 according to an embodiment of the present invention.

[0056] The elements involved in block 210 are the training data database 710 and the unidirectional RNN-T 720. The unidirectional RNN-T 720 can be configured to have an architecture similar to the RNN-T architecture 300 in Figure 3.

[0057] Figure 8 is a block diagram illustrating the elements involved in block 220 of method 200 of Figure 2 according to an embodiment of the present invention.

[0058] The elements involved in block 220 are the training data database 710 and the unidirectional RNN-T720, as well as the bidirectional RNN-T730. The bidirectional RNN-T730 can be configured to have an architecture similar to the RNN-T architecture 500 in Figure 5.

[0059] The one-best symbol for each node in the output stochastic grid is output from the unidirectional RNN-T720. This is used to train the bidirectional RNN-T730.

[0060] Figure 9 is a block diagram illustrating the elements involved in block 230 of method 200 of Figure 2, according to an embodiment of the present invention.

[0061] The elements involved in block 230 are a training data database 710, a one-way RNN-T 720, a two-way RNN-T 730, and another one-way RNN-T 740. The one-way RNN-T 740 can be configured to have an architecture similar to the RNN-T architecture 300 in Figure 3.

[0062] The output from the trained bidirectional RNN-T(730) is used to train a unidirectional RNN-T(lower). The "posterior distribution" is calculated for each node in the output grid from 730.

[0063] Figure 10 is a block diagram that further graphically illustrates the RNN-T elements involved in block 220 of Figure 2 according to an embodiment of the present invention.

[0064] The elements involved in block 220 are the training data database 1010, the one-way RNN-T1020, and the two-way RNN-T1030.

[0065] One-way RNN-T1020, which can be considered a reference RNN-T, includes a one-way prediction network 1020A, a one-way encoder network 1020B, and an output stochastic grid 1020C.

[0066] The bidirectional RNN-T1030, which can be considered a teacher RNN-T, includes a deep prediction network 1030A, a bidirectional / deep encoder network 1030B, and an output stochastic grid 1030C.

[0067] Reference number 1020C represents the output grid from the reference model, while 1020D is the 1-best selected from output grid 1020C. Thus, the same model 1020 is involved on the right side of Figure 10, but is shown at essentially different time steps in relation to obtaining the 1-best output.

[0068] Although not shown in Figure 10, it should be understood that some blocks of the output stochastic grid can be masked. In one embodiment, the masked blocks are ignored during model training. This masking is used when training a third network by referencing a second network.

[0069] In terms of operation, the unidirectional RNN-T1020 and the bidirectional RNN-T1030 receive training samples from the training data database 1010. The unidirectional RNN-T1020 selects the best for each node of the output stochastic grid 1020C. Cross-entropy is minimized between the bidirectional RNN-T1030 and the unidirectional RNN-T1020.

[0070] While this disclosure includes a detailed description of cloud computing, it should be understood that the implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment that is currently known or may be developed in the future.

[0071] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly prepared and published with minimal administrative work and interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0072] The characteristics are as follows: On-Demand Self-Service: Cloud consumers can unilaterally prepare computing functions such as server time and network storage automatically as needed, without requiring human interaction with service providers. Extensive network access: Functionality is available over the network and accessible through standard mechanisms that facilitate use with heterogeneous thin-client or thick-client platforms (such as mobile phones, laptops, and PDAs). Resource pooling: A provider's computing resources are pooled to serve multiple consumers with different physical and virtual resources that are dynamically allocated and reallocated as needed, using a multi-tenant model. While consumers generally cannot control or have knowledge of the exact location of the resources provided, location independence is significant in that they may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center). Rapid elasticity: Features can be provided quickly and elastically, sometimes automatically, allowing for rapid scaling out and rapid release for quick scaling in. To consumers, the features available for provisioning often appear unlimited and can be purchased in any quantity at any time. Measured services: Cloud systems automatically control and optimize resource usage by leveraging metric capabilities at an abstraction level appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). This provides transparency to both service providers and consumers by enabling monitoring, control, and reporting of resource usage.

[0073] The service model is as follows: Software as a Service (SaaS): The functionality provided to consumers is the use of the provider's applications running on cloud infrastructure. These applications are accessible from various client devices through thin-client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or individual application functionalities, with the exception of potentially limited user-specific application configuration settings. Platform as a Service (PaaS): The functionality offered to consumers is the deployment of applications created or acquired by the consumer using programming languages ​​and tools supported by the provider to cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure such as networks, servers, operating systems, and storage, but they do have control over the deployed applications and, in some cases, the configuration of the application hosting environment. Infrastructure as a Service (IaaS): The functionality provided to consumers is the provision of processing, storage, networking, and other basic computing resources that enable consumers to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do have control over their operating systems, storage, and deployed applications, and in some cases, limited control over selected network components (e.g., host firewalls).

[0074] The deployment model is as follows: Private Cloud: Cloud infrastructure is operated exclusively for an organization, managed by the organization or a third party, and may reside on-premises or off-premises. Community Cloud: Cloud infrastructure is shared by multiple organizations to support a specific community that shares concerns (e.g., mission, security requirements, policies, and compliance considerations). It may be managed by an organization or a third party and may reside on-premises or off-premises. Public Cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by organizations that sell cloud services. Hybrid Cloud: Cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain separate entities but are bound together by standardized or proprietary technologies (e.g., cloud bursting for load balancing across clouds) that enable data and application portability.

[0075] Cloud computing environments are service-oriented, emphasizing statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure, including a network of interconnected nodes.

[0076] Referring here to Figure 11, an exemplary cloud computing environment 1150 is depicted. As illustrated, the cloud computing environment 1150 includes one or more cloud computing nodes 1110 through which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or cellular phone 1154A, a desktop computer 1154B, a laptop computer 1154C, or an automotive computer system 1154N, or a combination thereof, can communicate. The nodes 1110 can communicate with one another. They may be physically or virtually grouped in one or more networks, or a combination thereof, such as private, community, public, or hybrid clouds as described herein (not shown). This allows the cloud computing environment 1150 to provide infrastructure, a platform, or software, or a combination thereof, as a service, eliminating the need for cloud consumers to maintain resources on their local computing devices. The types of computing devices 1154A-N shown in Figure 11 are for illustrative purposes only, and it should be understood that the computing node 1110 and the cloud computing environment 1150 can communicate with any type of computerized device (for example, using a web browser) via any type of network or network addressable connection, or both.

[0077] Next, referring to Figure 12, a set of functional abstraction layers provided by the cloud computing environment 1150 (Figure 11) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 12 are for illustrative purposes only, and embodiments of the present invention are not limited thereto. As shown in the figure, the following layers and corresponding functions are provided:

[0078] The hardware / software layer 1260 includes hardware and software components. Examples of hardware components include a mainframe 1261, RISC (Reduced Instruction Set Computer) architecture-based servers 1262, 1263, a blade server 1264, a storage device 1265, and network and networking components 1266. In some embodiments, the software components include network application server software 1267 and database software 1268.

[0079] The virtualization layer 1270 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 1271; virtual storage 1272; virtual networks 1273, including virtual private networks; virtual applications and operating systems 1274; and virtual clients 1275.

[0080] For example, the management layer 1280 may provide the following functions: Resource provisioning 1281 provides dynamic procurement of computing and other resources used to perform tasks within the cloud computing environment. Metering and pricing 1282 provides cost tracking as resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. For example, these resources may include application software licenses. Security provides protection of data and other resources, in addition to identity verification for cloud consumers and tasks. The user portal 1283 provides access to the cloud computing environment for consumers and system administrators. Service level management 1284 provides cloud computing resource allocation and management to meet required service levels. Service level agreement (SLA) planning and execution 1285 provides pre-arrangement and procurement of cloud computing resources where future requirements are anticipated in accordance with the SLA.

[0081] Workload layer 1290 provides examples of the functionality that a cloud computing environment can utilize. Examples of workloads and functions that can be provided from this layer include mapping and navigation 1291, software development and lifecycle management 1292, virtual classroom education delivery 1293, data analysis processing 1294, transaction processing 1295, and RNN-T accuracy improvement for end-to-end neural speech recognition 1296.

[0082] The present invention may be a computer program product, or a combination thereof, in the form of a system, method, or integration at any possible level of technical detail. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to perform an aspect of the present invention.

[0083] Computer-readable storage media can be tangible devices that can hold and store instructions used by instruction-executing devices. Computer-readable storage media can be, for example, but are not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any appropriate combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer diskettes, hard disks, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random-access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy(R) disks, mechanically encoded devices such as punch cards or grooved raised structures having instructions recorded thereon, and any appropriate combination of the foregoing. As used herein, computer-readable storage media should not be interpreted as being transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0084] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers or edge servers, or a combination thereof. The network adapter card or network interface of each computing / processing device receives computer-readable program instructions from the network and transfers them for storage in a computer-readable storage medium within the respective computing / processing device.

[0085] The computer-readable program instructions for performing the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may run as a standalone software package entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or a connection to an external computer may be made (for example, via the Internet using an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by personalizing the electronic circuit using state information of computer-readable program instructions in order to perform aspects of the present invention.

[0086] In this specification, aspects of the present invention will be described with reference to flowcharts, block diagrams, or both, of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0087] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing device to generate a machine, where instructions executed via the processor of a computer or other programmable data processing device create means to perform functions / operations specified in one or more blocks of a flowchart or block diagram, or both. These computer-readable program instructions may also be stored in a computer-readable storage medium in which the instructions stored therein can be instructed to function in a particular manner to a computer, programmable data processing device, or other device, or a combination thereof, to constitute a product containing instructions that perform modes of functions / operations specified in one or more blocks of a flowchart or block diagram, or both.

[0088] Computer-readable program instructions may also be loaded into a computer, other programmable data processing device, or other device so that instructions executed on a computer, other programmable device, or other device perform a function / action specified in one or more blocks of a flowchart or block diagram, or both, thereby generating a computer-executed process by causing the computer, other programmable device, or other device to perform a series of action steps.

[0089] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or part of an instruction, which contains one or more executable instructions for performing a specified logical function. In some alternative implementations, the functions described in the blocks may occur in a different order than shown in the diagram. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or they may be executed in reverse order depending on the functionality in which the blocks are involved. It should also be noted that each block in the block diagram and / or flowchart illustration, as well as combinations of blocks in the block diagram and / or flowchart illustration, may be implemented by a special-purpose hardware-based system that performs a specified function or action, or executes a combination of special-purpose hardware instructions and computer instructions.

[0090] Any reference in this specification to “one embodiment” or “embodiment” of the present invention, and to other modifications thereof, means that certain features, structures, characteristics, etc., described in connection with that embodiment are included in at least one embodiment of the present invention. Therefore, the phrases “in one embodiment” or “in an embodiment,” appearing in various places throughout this specification, like other modifications, do not necessarily all refer to the same embodiment.

[0091] Please understand that the use of any of the following " / ", "and / or", and "at least one" is intended to include, for example, "A / B", "A and / or B", and "at least one of A and B", the selection of only the first-listed option (A), or only the second-listed option (B), or the selection of both options (A and B). As further examples, in the case of "A, B, and / or C" and "at least one of A, B, and C", such phrasing is intended to include the selection of only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or only the first-listed and second-listed options (A and B), or only the first and third-listed options (A and C), or only the second and third-listed options (B and C), or the selection of all three options (A, B, and C). This can be extended to many of the listed items, as will be readily apparent to those skilled in the art in this and related fields.

[0092] While preferred embodiments of the system and method (intended to be descriptive and not limiting) have been described above, it should be noted that modifications and variations can be made in light of the above teachings by those skilled in the art. Therefore, it should be understood that modifications may be made in the specific embodiments disclosed that fall within the scope of the invention, as outlined by the attached claims. While aspects of the invention have been described in this manner to the level of detail and specificity required by patent law, anything claimed and desired to be protected by patent is described in the attached claims.

Claims

1. A computer-based method for model training, Training a second end-to-end neural speech recognition model having a bidirectional encoder so that it outputs the same symbols from the output stochastic grid of a second end-to-end neural speech recognition model as those from the output stochastic grid of a first trained end-to-end neural speech recognition model having a unidirectional encoder, and In the knowledge distillation method, the third end-to-end neural speech recognition model having a one-way encoder is constructed by training the third end-to-end neural speech recognition model as a student, using the trained second end-to-end neural speech recognition model as a teacher. A computer implementation method, including

2. Training the second end-to-end neural speech recognition model described above is Obtaining a 1-best symbol for each node of the output stochastic grid of the first end-to-end neural speech recognition model, and In addition to the end-to-end neural speech recognition model loss, the cross-entropy loss related to the 1-best symbol is also minimized. The computer implementation method according to claim 1, including the method described in claim 1.

3. The computer implementation method according to claim 2, wherein the weighted sum of the cross-entropy loss and the end-to-end neural speech recognition model loss is minimized.

4. Training the third end-to-end neural speech recognition model described above is With respect to the minimum probability threshold, masking the nodes with low probabilities in the output probability grid of the second end-to-end neural speech recognition model, and (i) Minimize together the end-to-end neural speech recognition model loss and (ii) minimize the relative divergence of the unmasked portion of the output stochastic grid of the second end-to-end neural speech recognition model with respect to the output stochastic grid of the third end-to-end neural speech recognition model. The computer implementation method according to claim 2, including the method described in claim 2.

5. The computer implementation method according to claim 4, wherein the relative divergence is a Kullback-Leibler divergence.

6. The computer implementation of claim 1, further comprising training the second end-to-end neural speech recognition model by inputting training audio data into the trained first end-to-end neural speech recognition model in order to obtain the output stochastic grid of the second end-to-end neural speech recognition model.

7. The computer implementation method according to claim 1, wherein the first and third end-to-end neural speech recognition models stream output data from an RNN transducer, and the second end-to-end neural speech recognition model provides output data offline.

8. The computer implementation method according to claim 1, wherein each node of the output probability grid of the second end-to-end neural speech recognition model represents a softmax operation.

9. The computer implementation of claim 1, further comprising performing speech recognition by performing a beam search on the output stochastic grid of the trained third end-to-end neural speech recognition model.

10. The computer implementation method according to claim 1, wherein the second end-to-end neural speech recognition model includes a neural network that is more complex than the first end-to-end neural speech recognition model.

11. The computer implementation of claim 1, further comprising evaluating the output stochastic grid of the third end-to-end neural speech recognition model based on its similarity to the output stochastic grid of the second end-to-end neural speech recognition system.

12. The computer implementation of claim 1, wherein the third end-to-end neural speech recognition system is trained until the similarity between the output probability grids of the third and second end-to-end neural speech recognition models falls within a specific similarity range.

13. The computer implementation method according to claim 1, wherein the method is performed by a speech recognition system.

14. The computer implementation method according to claim 1, wherein at least one of the first, second, and third neural speech recognition models comprises a recurrent neural network transducer model.

15. The computer implementation of claim 1, wherein the second end-to-end neural speech recognition model comprises a group of end-to-end neural speech recognition models, and the method further comprises selecting the best training model from the group in accordance with the overlap of search paths through the output probability grids of each of the end-to-end neural speech recognition models in the group.

16. A computer program product for model training, comprising a non-transient computer-readable storage medium having program instructions embodied therewith, wherein the program instructions are executable by a computer, and the computer, Training a second end-to-end neural speech recognition model having a bidirectional encoder so that it outputs the same symbols from the output stochastic grid of a second end-to-end neural speech recognition model as those from the output stochastic grid of a first trained end-to-end neural speech recognition model having a unidirectional encoder, and In the knowledge distillation method, the third end-to-end neural speech recognition model having a one-way encoder is constructed by training the third end-to-end neural speech recognition model as a student, using the trained second end-to-end neural speech recognition model as a teacher. A computer program product that includes, or causes, a method to be executed.

17. Training the second end-to-end neural speech recognition model described above is Obtaining a 1-best symbol for each node of the output stochastic grid of the first end-to-end neural speech recognition model, and In addition to the end-to-end neural speech recognition model loss, the cross-entropy loss related to the 1-best symbol is also minimized. A computer program product according to claim 16, including the following:

18. The computer program product according to claim 17, wherein the weighted sum of the cross-entropy loss and the end-to-end neural speech recognition model loss is minimized.

19. Training the third end-to-end neural speech recognition model described above is With respect to the minimum probability threshold, masking the nodes with low probabilities in the output probability grid of the second end-to-end neural speech recognition model, and (i) the loss of the end-to-end neural speech recognition model and (ii) the relative divergence of the unmasked portion of the output stochastic grid of the second end-to-end neural speech recognition model with respect to the output stochastic grid of the third end-to-end neural speech recognition model are to be minimized together. A computer program product according to claim 17, including the following:

20. A computer processing system for model training, A memory device for storing program code, and The memory device comprises a hardware processor operably coupled to it, the hardware processor executing the program code, The second end-to-end neural speech recognition model, having a bidirectional encoder, is trained to output the same symbols from the output stochastic grid of the second end-to-end neural speech recognition model as those from the output stochastic grid of the first trained end-to-end neural speech recognition model, having a unidirectional encoder, A computer processing system for constructing a third end-to-end neural speech recognition model having a one-way encoder, by training the third end-to-end neural speech recognition model as a student using the trained second end-to-end neural speech recognition model as a teacher in a knowledge distillation method.

Images