External language model information integrated into the neural transducer model

Abstract

A computer-implemented method for training a neural transducer is provided, the method including: obtaining outputs from a trained language model and a seed neural transducer, respectively, by using audio data and transcription data of the audio data as input data, combining the outputs to obtain a supervisory output, and updating parameters of another neural transducer being trained such that the output is closer to the supervisory output. The neural transducer may be a recurrent neural network transducer (RNN-T).

Description

[Technical field]

[0001] The present invention relates generally to machine learning, and more particularly to a method and system for integrating external language model information into a neural transducer model, such as a recurrent neural network transducer (RNN-T) model, for end-to-end speech recognition. [Background technology]

[0002] End-to-end models for automatic speech recognition (ASR) have gained popularity in recent years as a way to combine the separate components of traditional ASR systems (e.g., acoustic, pronunciation, and language models) into a single neural network. Examples of such models include connectionist temporal classification (CTC)-based models, recurrent neural network transducers (RNN-T), and attention-based seq2seq models. Among these models, RNN-T is the most streaming-friendly end-to-end recognizer and has shown competitive performance compared to traditional systems. Summary of the Invention

[0003] According to one aspect of the present invention, there is provided a computer-implemented method for training a neural transducer, the computer-implemented method including: obtaining outputs from a trained language model and a seed neural transducer, respectively, by using audio data and transcription data of the audio data as input data, combining the outputs to obtain a supervisory output, and updating parameters of another neural transducer being trained such that the output is closer to the supervisory output.

[0004] According to another aspect of the present invention, there is provided a computer program product for training a neural transducer, the computer program product including a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a computer to: obtain outputs from a trained language model and a seed neural transducer, respectively, by using audio data and transcription data of the audio data as input data, combine the outputs to obtain a supervisory output, and update parameters of another neural transducer being trained such that the output is closer to the supervisory output.

[0005] According to yet another aspect of the present invention, there is provided a system for training a neural transducer, the system including a memory and one or more processors in communication with the memory, the processors configured to: obtain outputs from a trained language model and a seed neural transducer, respectively, by using audio data and transcription data of the audio data as input data, combine the outputs to obtain a supervisory output, and update parameters of another neural transducer being trained such that the output is closer to the supervisory output.

[0006] It should be noted that the exemplary embodiments are described in relation to different subject matters. In particular, some embodiments are described in relation to method type claims and other embodiments are described in relation to device type claims. However, a person skilled in the art will infer from the above and following description that, unless otherwise stated, any combination of features belonging to one type of subject matter, as well as any combination between features relating to different subject matters, in particular any combination between features of a method type claim and a feature of a device type claim, is considered to be described in this document.

[0007] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which should be read in connection with the accompanying drawings.

[0008] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which: [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block / flow diagram of an exemplary system for integrating external Language Model (LM) information into a neural transducer model, such as a Recurrent Neural Network Transducer (RNN-T), in accordance with an embodiment of the present invention. [Diagram 2] 1 is a block / flow diagram of an exemplary neural transducer architecture and output probability lattice. [Diagram 3] FIG. 1 is a block / flow diagram of a method for training a neural transducer by integrating an external language model (LM) according to an embodiment of the present invention. [Figure 4] FIG. 1 is a block / flow diagram of an exemplary method for integrating external LM information into a neural transducer model such as an RNN-T, in accordance with an embodiment of the present invention. [Diagram 5] FIG. 1 illustrates a practical application example of how neural transducers are used in automatic speech recognition (ASR) for speech-to-text conversion, according to an embodiment of the present invention. [Figure 6] FIG. 1 is a block / flow diagram of an exemplary processing system for integrating external LM information into a neural transducer model such as an RNN-T, in accordance with an embodiment of the present invention. [Figure 7] FIG. 2 is a block / flow diagram of an exemplary cloud computing environment, in accordance with an embodiment of the present invention. [Figure 8] 2 is a schematic diagram of an exemplary abstract model layer according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Throughout the drawings, the same or similar reference numbers represent the same or similar elements.

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments according to the present invention provide methods and devices for improving a neural transducer model by using a posterior distribution from an external language model (LM).

[0012] Before delving into recurrent neural network transducers (RNN-T), speech recognition continues to evolve. New speech recognition architectures or improvements to existing architectures continue to be developed to improve the quality of ASR systems. To explain, speech recognition initially employed multiple models, with each model having a dedicated purpose. For example, ASR systems included an acoustic model (AM), a pronunciation model (PM), and a language model (LM). The acoustic model mapped segments of audio (e.g., frames of audio) to phonemes. The pronunciation model connected these phonemes to form words, and the language model was used to represent the likelihood of a given phrase (e.g., the likelihood of a sequence of words). Furthermore, although these separate models worked together, each model was trained independently and was often manually designed on a different dataset.

[0013] While the separate model approach enabled speech recognition systems to be very accurate, especially when the training corpus (e.g., a body of training data) for a given model satisfied the model's validity, the need to train separate models independently introduced its own complexities, leading to architectures using integrated models. These integrated models attempted to directly map audio waveforms (e.g., input sequence) to output sentences (e.g., output sequence) using a single neural network. This resulted in sequence-to-sequence approaches, whereby a sequence of words (or graphemes) was generated given a sequence of audio features. Examples of sequence-to-sequence models include "attention-based" models and "listen-attend-spell" (LAS) models. LAS models transcribe speech utterances into characters using a listener component, an attender component, and a speller component. Here, the listener is a recurrent neural network (RNN) encoder that receives audio input (e.g., a time-frequency representation of the speech input) and maps the audio input to a higher-level feature representation. The Attender learns alignments between input features and predicted subword units (e.g., graphemes or word fragments) by focusing on higher-level features. The Speller is an attention-based RNN decoder that generates character sequences from the input by creating a probability distribution over a set of hypothesized words. In an integrated architecture, all components of the model may be trained together as a single end-to-end (E2E) neural network. Here, E2E models refer to models whose architecture consists entirely of neural networks. A complete neural network can function without external or manually designed components (e.g., finite-state transducers, lexicons, or text normalization modules) or both.Additionally, when E2E models are trained, they generally do not require bootstrapping from decision trees or time alignment from a separate system.

[0014] While early E2E models demonstrated improved accuracy and training over individually trained models, these E2E models, such as the LAS model, worked by reviewing the entire input sequence before generating output text and therefore did not allow for streaming output as the input was received. Without streaming capability, the LAS model is unable to perform real-time speech transcription. Due to this deficiency, deploying LAS models for speech applications that are latency sensitive or require real-time speech transcription can be problematic.

[0015] In addition, speech recognition systems having acoustic, pronunciation, and language models, or such models configured together, may rely on a decoder that must search a relatively large search graph associated with these models. For large search graphs, hosting this type of speech recognition system entirely on-device is not conducive. Now, when a speech recognition system is hosted "on-device," the device that receives the audio input uses its processor to perform the functions of the speech recognition system. For example, when a speech recognition system is hosted entirely on-device, the device's processor does not need to coordinate with any off-device computing resources to perform the functions of the speech recognition system. Devices that perform speech recognition that are not entirely on-device rely on remote computing (e.g., a remote computing system or cloud computing), and thus on-line connectivity, to perform at least some functions of the speech recognition system. For example, the speech recognition system uses a network connection with a server-based model to perform decoding on a large search graph.

[0016] Unfortunately, the reliance on remote connections makes speech recognition systems vulnerable to latency issues and / or inherent unreliability of communication networks. To improve the usefulness of speech recognition by circumventing these issues, speech recognition systems have evolved into a form of sequence-to-sequence model, again known as recurrent neural network transducer (RNN-T). Unlike other sequence-to-sequence models that do not employ attention mechanisms and generally must process entire sequences (e.g., audio waveforms) to produce an output (e.g., a sentence), RNN-T processes input samples continuously and streams output symbols. This is a feature that is particularly attractive for real-time communication. For example, speech recognition using RNN-T may output characters as they are spoken one by one.

[0017] Therefore, there is a need for a more efficient process for improving speech recognition accuracy. Exemplary embodiments of the present invention introduce a method for improving a neural transducer model (e.g., RNN-T) by using a posterior distribution from an external language model (LM).

[0018] Although the present invention is described with respect to a given example architecture, it should be understood that other architectures, structures, substrate materials, and process features and steps / blocks may vary within the scope of the present invention. It should be noted that for clarity, certain features may not be shown in all drawings. This is not intended to be construed as a limitation of any particular embodiment or example, or of the scope of the claims.

[0019] FIG. 1 is a block / flow diagram of an exemplary system 10 for integrating external language model (LM) information into a neural transducer model, such as a recurrent neural network transducer (RNN-T), in accordance with an embodiment of the present invention.

[0020] The exemplary embodiment introduces a system 10 in which the probability of a language model P_{LM}(y|u) is combined for each P(y|t,u) in the posterior lattice. In this way, for any time index, the LM probability given the previous output symbol is the same. As a result, the same P_{LM}(y|u) can be combined with P(y|t,u) for all time indexes t with the same u. The combined probability is then used as a teacher to train a neural transducer model.

[0021] In particular, referring to FIG. 1, a language model 22 of any type is trained from a large amount of text data (e.g., an external corpus). Using the audio data 14 and its transcription data 12, a posterior lattice P(y|t,u) (32) is calculated from a seed neural transducer model 20. Then, the probability of the language model P_{LM}(y|u) (30) is calculated using the trained language model 22, and P(y|t,u) (32) and P_{LM}(y|u) (30) are combined, for example, by using linear or log-linear interpolation. The parameters of the neural transducer model 24 are updated by minimizing the KL divergence or mean squared error between the output from the neural transducer model being trained (24) and the calculated interpolated output (34).

[0022] Advantages of the present invention preferably include at least better speech recognition accuracy, where the neural transducer model itself is trained to yield an output after interpolation of a seed neural transducer model and an external language model.

[0023] Figure 2 is a block / flow diagram of an exemplary neural transducer architecture and output probability lattice.

[0024] The neural transducer architecture may be, for example, an RNN-T model 50, which includes an encoder (or encoder network) 54, a prediction (or prediction network) 64, and a combination network 70. The encoder network 54 is similar to an acoustic model, which is a set of acoustic features x t A higher representation of

number

number

[0025] The combination network 70 is connected to the encoder network output

number

number

number

[0026] The loss function of RNN-T is the negative log-posterior distribution of the output label sequence y given the input acoustic features x. L = -lnP(y|x)

[0027] Thus, according to the RNN-T model 50, the neural transducer generates an output sequence y based on an input feature sequence x by searching over an output probability lattice defined by P(y|t,u) (74), where x=(x1,...x T ) is the input sequence 52 of audio feature vectors, and y=(y1,...y U ) is the sequence of previous tokens, and z{t,u} are the logits.

[0028] For the output probability lattice 80, each node 88 represents the softmax of z_{t,u}.

[0029] FIG. 3 is a block / flow diagram of a method for training a neural transducer by integrating an external language model (LM) according to an embodiment of the present invention.

[0030] In block 90, the audio data and the transcription data of the audio data are used as input data to obtain outputs from a trained language model and a seed neural transducer, e.g., an RNN-T, respectively.

[0031] In block 92, the outputs are combined to obtain a supervisory output.

[0032] In block 94, the parameters of the other neural transducer being trained are updated so that its output is closer to the supervisory output. In other words, the output from the other neural transducer being trained approaches or approaches the supervisory output. The seed neural transducer 20 (FIG. 1) is fixed. The new neural transducer 24 (FIG. 1) is trained from scratch (random values) or initialized by a seed model. Note that the seed neural transducer is different from the neural transducer being trained.

[0033] FIG. 4 is a block / flow diagram of an exemplary method for integrating external language model (LM) information into a neural transducer model, such as an RNN-T, in accordance with an embodiment of the present invention.

[0034] At block 100, any type of language model is trained from large amounts of text data.

[0035] In block 102, a posterior lattice P(y|t,u) is computed from a seed neural transducer model by using the audio and its transcription.

[0036] In block 104, the language model probability P_{LM}(y|u) is calculated with the trained language model.

[0037] In block 106, P(y|t,u) and P_{LM}(y|u) are combined, for example, by using either linear or log-linear interpolation.

[0038] In block 108, the parameters of the neural transducer model are updated by minimizing the KL divergence or mean squared error between the output from the neural transducer model being trained and the calculated interpolated output. As mentioned above, the seed neural transducer 20 (FIG. 1) is fixed. A new neural transducer 24 (FIG. 1) is trained from scratch (random values) or initialized by a seed model. Note that the seed neural transducer is different from the neural transducer being trained.

[0039] FIG. 5 illustrates a practical application example 110 of how RNN-T is used in automatic speech recognition (ASR) for speech-to-text conversion, according to an embodiment of the present invention.

[0040] Simply put, conversational artificial intelligence (AI) uses natural language to communicate with machines. Deep learning applications of conversational AI are growing day by day, from voice assistants and chatbots to question answering systems that enable customer self-service. The range of industries that apply conversational AI to their solutions is wide and diverse, ranging from finance to healthcare. Conversational AI is a complex system that integrates multiple deep neural networks that must work seamlessly and in harmony to deliver a pleasant user experience with accurate, fast and natural human-to-machine interaction. To achieve these goals, developers are developing applications that solve important problems such as domain adaptation, user analytics, compliance, high-precision speech recognition, user identification, sentiment analysis, among others.

[0041] A conversational AI application uses three subsystems to process and transcribe audio, understand the question asked (derive meaning), generate a response (text) and return the response to the human. These steps are achieved by multiple deep learning solutions working together. First, an automatic speech recognition (ASR) system 112 is used to process the raw audio signal and transcribe text from it. Second, natural language processing (NLP) is used to derive meaning from the transcribed text (ASR output). In this case, the seed neural transducer 20 (Figure 1) is fixed. A new neural transducer 24 (Figure 1) is trained from scratch (random values) or initialized by a seed model. Note that the seed neural transducer is different from the neural transducer being trained. Finally, speech synthesis or speech-to-text 116 is used to produce the text. The generated text may be displayed, for example, on a smartphone 120, a tablet 122, or a computer 124, or a combination thereof. As a result, the exemplary system 10 may be implemented in an ASR 112 to generate text from speech 116, as an exemplary practical application.

[0042] FIG. 6 is a block / flow diagram of an exemplary processing system for preparing training data for a speech recognition model, according to an embodiment of the present invention.

[0043] 6 illustrates a block diagram of components of a system 200, including a computing device 205. It should be understood that FIG. 6 is merely provided as an illustration of one embodiment and is not intended to suggest any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the illustrated environment may be made.

[0044] Computing device 205 includes a communications fabric 202 that provides communications between computer processor 204, memory 206, persistent storage 208, communications unit 210, and input / output (I / O) interface 212. Communications fabric 202 may be implemented in any architecture designed to pass data and / or control information between processors (such as microprocessors, communications and network processors), system memory, peripheral devices, and any other hardware components in the system. For example, communications fabric 202 may be implemented with one or more buses.

[0045] Memory 206, cache memory 216, and persistent storage 208 are computer-readable storage media. In this embodiment, memory 206 includes random access memory (RAM) 214. In another embodiment, memory 206 may be flash memory. In general, memory 206 may include any suitable volatile or non-volatile computer-readable storage medium.

[0046] In some embodiments of the present invention, the program 225 is included and operated by the AI ​​accelerator chip 222 as a component of the computing device 205. In other embodiments, the program 225 is stored in the persistent storage 208 for execution by the AI ​​accelerator chip 222 in conjunction with one or more of the respective computer processors 204 via one or more memories of the memory 206 (for integrating external language model information into a neural transducer model, such as a recurrent neural network transducer (RNN-T) model, for end-to-end speech recognition). In this embodiment, the persistent storage 208 includes a magnetic hard disk drive. As an alternative to or in addition to a magnetic hard disk drive, the persistent storage 208 may include a solid state hard drive, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0047] The media used by persistent storage 208 may also be removable. For example, a removable hard drive may be used for persistent storage 208. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer-readable storage medium that is also part of persistent storage 208.

[0048] In these examples, the communication unit 210 provides communication with other data processing systems or devices, including resources of a distributed data processing environment. In these examples, the communication unit 210 includes one or more network interface cards. The communication unit 210 may provide communication through the use of either or both physical and wireless communication links. The deep learning program 225 may be downloaded to the persistent storage 208 through the communication unit 210.

[0049] I / O interface 212 allows for the input and output of data with other devices that may be connected to computing system 200. For example, I / O interface 212 may provide a connection to external devices 218, such as a keyboard, a keypad, a touch screen, or some other suitable input device, or a combination thereof. External devices 218 may also include portable computer-readable storage media, such as, for example, thumb drives, portable optical or magnetic disks, and memory cards.

[0050] Display 220 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0051] FIG. 7 is a block / flow diagram of an exemplary cloud computing environment in accordance with an embodiment of the present invention.

[0052] Although the present invention includes detailed descriptions of cloud computing, it should be understood that implementations of the teachings recited herein are not limited to cloud computing environments. Rather, embodiments of the present invention may be practiced in conjunction with any other type of computing environment now known or later developed.

[0053] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0054] The characteristics are as follows:

[0055] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capabilities such as server time and network storage automatically as needed, without the need for human interaction with the service provider.

[0056] Broad network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., cell phones, laptops, and PDAs).

[0057] Resource Sharing: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Location independence is implied in that consumers generally do not have control or knowledge over the exact location of the resources provided, but may be able to specify location at a higher level of abstraction (e.g., country, state, or data center).

[0058] Rapid Scalability: Capabilities can be quickly and elastically provisioned, in some cases automatically, to instantly scale out, and quickly released to instantly scale in. To the consumer, the capabilities available for provisioning often appear unlimited and can be purchased in any quantity at any time.

[0059] Services are meterable: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used.

[0060] The service model is as follows:

[0061] Software as a Service (SaaS): The capability provided to the consumer is the use of the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0062] Platform as a Service (PaaS): The capability provided to the consumer is to deploy consumer-created or consumer-acquired applications, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, to the extent possible, the application hosting environment configuration.

[0063] Infrastructure as a Service (IaaS): The capability provided to the consumer is the supply of processing, storage, networking, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and possibly limited control over select networking components (e.g., host firewalls).

[0064] The deployment model is as follows:

[0065] Private Cloud: The cloud infrastructure is operated exclusively for an organization. It is managed by the organization or a third party and can be on-premise or off-premise.

[0066] Community Cloud: The cloud infrastructure is shared by multiple organizations to support a specific community with shared concerns (e.g., mission, security requirements, policy, and compliance considerations). The cloud infrastructure is managed by the organizations or a third party and can be on-premise or off-premise.

[0067] Public Cloud: The cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0068] Hybrid Cloud: The cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain a unique entity but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0069] A cloud computing environment is stateless, low-coupling, modular, and service-oriented with a focus on semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0070] Referring now to FIG. 7, an exemplary cloud computing environment 450 for enabling use cases of the present invention is shown. As shown, the cloud computing environment 450 includes one or more cloud computing nodes 410 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 454A, a desktop computer 454B, a laptop computer 454C, or an automobile computer system 454N, or combinations thereof, may communicate. The nodes 410 may communicate with each other. They may be physically or virtually grouped in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or combinations thereof, as described above (not shown). This enables the cloud computing environment 450 to offer infrastructure, platform, or software, or combinations thereof, as a service without the cloud consumer having to maintain resources on the local computing device. It should be understood that the types of computing devices 454A-N shown in FIG. 7 are intended to be exemplary only, and that the computing node 410 and cloud computing environment 450 may communicate with any type of computerized device via any type of network and / or network addressable connections (e.g., using a web browser).

[0071] 8 is a schematic diagram of an exemplary abstract model layer according to an embodiment of the present invention. It should be understood in advance that the components, layers, and functions shown in FIG. 8 are intended to be merely illustrative, and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0072] Hardware and software layer 560 includes hardware and software components. Examples of hardware components include mainframe 561, reduced instruction set computer (RISC) architecture based servers 562, servers 563, blade servers 564, storage devices 565, and networks and networking components 566. In some embodiments, software components include network application server software 567 and database software 568.

[0073] The virtualization layer 570 provides an abstraction layer over which the following examples of virtual entities may be provided: virtual servers 571, virtual storage 572, virtual networks including virtual private networks 573, virtual applications and operating systems 574, and virtual clients 575.

[0074] In one embodiment, management layer 580 may provide the functionality described below. Resource provisioning 581 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 582 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one embodiment, these resources may include application software licenses. Security provides protection for data and other resources as well as identity verification for cloud consumers and tasks. User portal 583 provides consumers and system administrators with access to the cloud computing environment. Service level management 584 provides cloud computing resource allocation and management so that requested service levels are met. Service level agreement (SLA) planning and fulfillment 585 provides pre-positioning and procurement of cloud computing resources where future requirements are anticipated according to SLAs.

[0075] The Workload Layer 590 provides examples of functionality for which a cloud computing environment may be used. Examples of workloads and functions that may be provided from this layer include mapping and navigation 541, software development and lifecycle management 592, virtual classroom instructional delivery 593, data analytics processing 594, transaction processing 595, and a system 10 for integrating external LM information into a neural transducer, such as an RNN-T.

[0076] The present invention may be a system, a method, or a computer program product, or a combination thereof. The computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0077] A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memories, read-only memories, erasable programmable read-only memories (EPROMs or Flash memories), static random access memories, portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves that allow instructions to be recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through electrical wires.

[0078] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0079] The computer readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or object oriented programming languages ​​such as Smalltalk®, C++, and traditional procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may execute completely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electronic circuitry to perform aspects of the invention.

[0080] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0081] These computer readable program instructions may be provided to at least one processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to manufacture a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for implementing the functions / operations specified in one or more blocks or modules of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored in a computer readable storage medium that may direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium having instructions stored therein includes an article of manufacture including instructions for implementing aspects of the functions / operations specified in one or more blocks or modules of the flowcharts and / or block diagrams.

[0082] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to execute a series of operational blocks / steps to create a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / operations specified in one or more blocks or modules of the flowcharts and / or block diagrams.

[0083] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may occur out of the order described in the drawings. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in the reverse order depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or executes a combination of dedicated hardware and computer instructions.

[0084] References in the specification to "one embodiment" or "an embodiment" of the present principles, and other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment of the present principles. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment," as well as any other variations thereof, appearing in various places throughout the specification are not necessarily all referring to the same embodiment.

[0085] Use of any of the following " / ", "and / or" and "at least one of" should be understood to be intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B), for example, "A / B", "A and / or B", and "at least one of A and B". As a further example, "A, B, and / or C" and "at least one of A, B, and C" are intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A and B and C). This may be extended when multiple items are listed, as would be readily apparent to one of ordinary skill in the art and related arts.

[0086] Although preferred embodiments of a method and system for integrating external language model information into a neural transducer model, such as an RNN-T model, for end-to-end speech recognition have been described (intended as illustrative and not limiting), it should be noted that modifications and variations may be made by those skilled in the art in light of the above teachings. It is therefore to be understood that changes may be made in the particular embodiments described that are within the scope of the invention as outlined by the appended claims.

Claims

1. 1. A computer-implemented method for training a neural transducer, comprising: obtaining outputs from a trained language model and a seed neural transducer, respectively, by using audio data and transcription data of the audio data as input data; combining said outputs to obtain a supervisory output; updating parameters of another neural transducer being trained so that its output is closer to the supervisory output; 4. A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein the neural transducer is a recurrent neural network transducer (RNN-T).

3. The computer-implemented method of claim 1 or 2, wherein the combining of the outputs is performed by using linear interpolation.

4. The computer-implemented method of claim 1 or 2, wherein the combining of the outputs is performed by using log-linear interpolation.

5. 5. The computer-implemented method of claim 1, wherein the output of the neural transducer is a posterior lattice.

6. 6. The computer-implemented method of claim 5, wherein the horizontal axis of the posterior lattice is a time index t, the vertical axis of the posterior lattice is a symbol u in the transcription data, and each node t, u in the posterior lattice represents a posterior distribution having an output for the first u elements of an output sequence by point t in the transcription sequence.

7. 7. The computer-implemented method of claim 6, wherein the combining of the outputs comprises combining the same output of the trained language model with the posterior distributions for all time indexes that have the same u.

8. 1. A computer program product for training a neural transducer, the computer program product including a computer readable storage medium having program instructions embodied therein, the program instructions causing a computer to: obtaining outputs from a trained language model and a seed neural transducer, respectively, by using audio data and transcription data of the audio data as input data; combining said outputs to obtain a supervisory output; updating parameters of another neural transducer being trained so that its output is closer to the supervisory output; a computer program product executable by said computer to

9. 9. The computer program product of claim 8, wherein the neural transducer is a recurrent neural network transducer (RNN-T).

10. 10. A computer program product according to claim 8 or 9, wherein the combining of the outputs is performed by using linear interpolation.

11. 10. A computer program product as claimed in claim 8 or 9, wherein said combining of said outputs is performed by using log-linear interpolation.

12. 12. A computer program product according to any one of claims 8 to 11, wherein the output of the neural transducer is a posterior lattice.

13. 13. The computer program product of claim 12, wherein a horizontal axis of the posterior lattice is a time index t, a vertical axis of the posterior lattice is a symbol u in the transcription data, and each node t, u in the posterior lattice represents a posterior distribution having an output for the first u elements of an output sequence by point t in the transcription sequence.

14. 14. The computer program product of claim 13, wherein the combining of the outputs comprises combining a same output of the trained language model with the posterior distributions for all time indexes that have a same u.

15. 1. A system for preparing training data for training a neural transducer, comprising: Memory, one or more processors in communication with the memory, obtaining outputs from a trained language model and a seed neural transducer, respectively, by using audio data and transcription data of the audio data as input data; combining said outputs to obtain a supervisory output; updating parameters of another neural transducer being trained so that its output is closer to the supervisory output; a processor configured to A system comprising:

16. 16. The system of claim 15, wherein the neural transducer is a recurrent neural network transducer (RNN-T).

17. 17. The system of claim 15 or 16, wherein the combining of the outputs is performed by using linear interpolation.

18. 17. The system of claim 15 or 16, wherein the combining of the outputs is performed by using log-linear interpolation.

19. 19. The system of claim 15, wherein the output of the neural transducer is a posterior lattice.

20. 20. The system of claim 19, wherein a horizontal axis of the posterior lattice is a time index t and a vertical axis of the posterior lattice is a symbol u in the transcription data, each node t,u in the posterior lattice represents a posterior distribution having an output for the first u elements of an output sequence by point t of the transcription sequence, and wherein the combining of the outputs comprises combining the same output of the trained language model with the posterior distribution for all time indexes with the same u.

21. A computer program comprising program code means adapted to perform the method according to any one of claims 1 to 7, when said program runs on a computer.

Images