Customizing Recurrent Neural Network Transducers for Speech Recognition

Abstract

Customizing a recurrent neural network transducer for speech recognition A computer-implemented method for customizing a recurrent neural network transducer (RNN-T) is provided. The computer-implemented method includes synthesizing first domain audio data from first domain text data and feeding the synthesized first domain audio data to a trained encoder of the recurrent neural network transducer (RNN-T) having an initial condition, where the encoder is updated using the synthesized first domain audio data and the first domain text data. The computer-implemented method further includes synthesizing second domain audio data from second domain text data and feeding the synthesized second domain audio data to an updated encoder of the recurrent neural network transducer (RNN-T), where the predictive network is updated using the synthesized second domain audio data and the second domain text data. The computer-implemented method further includes restoring the updated encoder to the initial condition.

Description

[Technical Field]

[0001] The present invention relates generally to speech recognition, and more particularly to a method and system for training an end-to-end speech recognition model.

[0002] Recurrent neural networks (RNNs) are a type of artificial neural network in which the connections between nodes form a time-ordered, directed graph. This allows RNNs to analyze sequence dependencies between attributes such as phonemes. Derived from feedforward neural networks, RNNs can use their internal state (memory) to process variable-length sequences of input. RNNs can also directly encode ordering information. RNNs can receive and process inputs in the same order as the original sequence. This makes them applicable to tasks such as recognition of non-segmented connected handwriting input or speech recognition. The input values ​​in the time series can be actual values ​​or symbols.

[0003] RNNs can have a fixed number of parameters and can process a variable number of inputs. RNNs can have a one-to-one relationship between the number of positions in the sequence and the number of layers in the network. Each layer can have a single input in a specific position in the sequence (e.g., time, stage). The input can thereby interact with the hidden layer depending on the input's position in the sequence. The layer architecture repeats in time, and is therefore called recurrent. RNNs can require a predetermined alignment between the input and output sequences to perform the transformation. This can be limiting because finding the alignment can be a very difficult aspect of the sequence transformation problem.

[0004] An end-to-end (E2E) automatic speech recognition (ASR) system can directly convert a sequence of acoustic features into an output sequence of tokens (phonemes, characters, words, etc.) by mapping acoustic features to an output token sequence. An end-to-end model of ASR can directly output a transcript of a given word in the input audio. Summary of the Invention

[0005] According to an embodiment of the present invention, a computer-implemented method for customizing a recurrent neural network transducer (RNN-T) is provided. The computer-implemented method includes synthesizing first-domain audio data from first-domain text data and feeding the synthesized first-domain audio data to a trained encoder of the recurrent neural network transducer (RNN-T) having initial conditions, where the encoder is updated using the synthesized first-domain audio data and the first-domain text data. The computer-implemented method further includes synthesizing second-domain audio data from second-domain text data and feeding the synthesized second-domain audio data to an updated encoder of the recurrent neural network transducer (RNN-T), where the predictive network is updated using the synthesized second-domain audio data and the second-domain text data. The computer-implemented method further includes restoring the updated encoder to the initial conditions.

[0006] According to another embodiment of the present invention, there is provided a system for customizing a recurrent neural network transducer (RNN-T). The system includes one or more processor devices, a memory in communication with at least one of the one or more processor devices, and a display screen. The memory includes a synthesizer configured to synthesize first-domain audio data from first-domain text data and second-domain audio data from second-domain text data, and an encoder configured to receive the synthesized first-domain audio data generated from the first-domain text data, the encoder being a trained encoder of the recurrent neural network transducer (RNN-T) having initial conditions, the encoder being configured to be updated from the initial conditions using the synthesized first-domain audio data and the first-domain text data, and the encoder being further configured to receive the synthesized second-domain audio data generated from the second-domain text data. The memory further includes an output sequence generator configured to generate an output symbol sequence y based on an input feature sequence x, which is a time-ordered sequence of acoustic features represented as vectors.

[0007] According to yet another embodiment of the present invention, a computer program product for customizing a recurrent neural network transducer (RNN-T) is provided, the computer program product including one or more computer-readable storage media, wherein program instructions are collectively stored on the one or more computer-readable storage media and are executable by a computer. The computer program product, when executed, causes a computer to synthesize first-domain audio data from first-domain text data and to provide the synthesized first-domain audio data to a trained encoder of the recurrent neural network transducer (RNN-T) having initial conditions, where the encoder is updated using the synthesized first-domain audio data and the first-domain text data. The computer program product, when executed, also causes a computer to synthesize second-domain audio data from second-domain text data and to provide the synthesized second-domain audio data to an updated encoder of the recurrent neural network transducer (RNN-T), where a predictor is updated using the synthesized second-domain audio data and the second-domain text data. The computer program product, when executed, also causes the computer to restore the updated encoder to its initial condition.

[0008] According to yet another embodiment of the present invention, there is provided a computer-implemented method for customizing a recurrent neural network transducer (RNN-T), comprising: synthesizing first-domain audio data from first-domain text data; and providing the synthesized first-domain audio data to a trained encoder of the recurrent neural network transducer (RNN-T) having initial conditions, the encoder being updated using the synthesized first-domain audio data and the first-domain text data; and converting the synthesized first-domain audio data into an acoustic embedding. twhere the acoustic embedding a t The computer-implemented method compresses the synthesized first domain audio data into a smaller feature space. t to a joiner, and synthesizing second-domain audio data from the second-domain text data. The computer-implemented method further includes feeding the synthesized second-domain audio data to an updated encoder, where the updated encoder encodes the synthesized second-domain audio data into acoustic embeddings bt, where the acoustic embeddings bt compress the synthesized second-domain audio data into a smaller feature space, and feeding an output sequence from the joiner to a predictor of a recurrent neural network transducer (RNN-T), where the predictor is updated using the output sequence from the synthesized second-domain audio data and the second-domain text data. The computer-implemented method further includes restoring the updated encoder to an initial condition.

[0009] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments, which should be read in connection with the accompanying drawings. [Brief explanation of the drawings]

[0010] The following description provides details of preferred embodiments with reference to the following drawings: [Figure 1] FIG. 1 is a diagram of an architecture for a recurrent neural network transducer (RNN-T) that can be applied to speech recognition, according to an embodiment of the present invention. [Figure 2] FIG. 1 is a block / flow diagram illustrating an algorithm for training a recurrent neural network transducer (RNN-T) for speech recognition, according to an embodiment of the present invention. [Figure 3]FIG. 1 is a block / flow diagram illustrating an algorithm for synthesizing audio features from text, updating the encoder and predictor, and restoring the encoder according to an embodiment of the invention. [Figure 4] FIG. 1 is a diagram of a posterior lattice for an RNN-T, in accordance with an embodiment of the present invention. [Figure 5] 1 is an exemplary processing system to which the method and system may be applied, according to an embodiment of the present invention; [Figure 6] 1 illustrates an exemplary processing system configured to implement one or more neural networks for modeling road layouts, according to an embodiment of the present invention. [Figure 7] FIG. 1 is a block diagram illustrating an exemplary neural network, in accordance with an embodiment of the present invention. [Figure 8] FIG. 1 is a block diagram illustrating an exemplary cloud computing environment having one or more cloud computing nodes with which local computing devices used by cloud consumers communicate, according to an embodiment. [Figure 9] FIG. 2 is a block diagram illustrating a set of functional abstraction layers provided by a cloud computing environment, according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiments of the present invention provide a system and method for customizing a language model to a specific target domain from a sufficiently strong base RNN-T model. The RNN-T can be pre-trained from scratch. The pre-trained RNN-T can be customized by the user for a specific domain.

[0012] A beam search across one neural network can be performed for inference, resulting in low computational cost, a small memory footprint, and a simple inference engine.

[0013] In various embodiments, this approach can accommodate RNN-T based modeling, where the "language model" and "vocabulary" are integrated into one neural network along with other modules and cannot be directly manipulated externally after the network has been trained and deployed.

[0014] Exemplary applications / uses to which the present invention may be applied include, but are not limited to, automatic speech recognition (ASR), natural language translation, and the like.

[0015] Although aspects of the invention are described in terms of given example architectures, it should be understood that other architectures, structures, and process features and steps may vary within the scope of aspects of the invention.

[0016] Referring now to the drawings, wherein like numerals represent the same or similar elements, and initially to FIG. 1 , there is shown a diagram of an architecture for a recurrent neural network transducer (RNN-T) applicable to speech recognition, in accordance with an embodiment of the present invention.

[0017] Automatic speech recognition (ASR) using deep neural networks (DNNs) can use a hybrid framework that implements several models simultaneously. Models for hybrid systems can include an acoustic model (AM) and a language model (LM). An RNN-T model can include three parts: a predictor network that encodes a sequence of labels into a text embedding, an encoder network that encodes a sequence of observation vectors into an acoustic embedding, and a neural network (e.g., a joiner) with a softmax output layer that combines the text and acoustic embeddings.

[0018] Language model customization is a feature that allows developers and users to add application-specific, use-case-specific words, phrases, and sentences for automatic speech recognition (ASR) to a trained language model. In other words, customization modifies a previously trained and deployed language model (LM), thereby providing a flexible and efficient customization capability. In various embodiments, the acoustic model, language model, and vocabulary are explicitly modularized in the hybrid system. Therefore, language model customization is easily achieved because the language model and vocabulary can be directly manipulated and modified even after they are trained and deployed.

[0019] In one or more embodiments, the ASR RNN-T 100 may have three components: an encoder 130 for audio, a predictor 140 for text, and a joiner 150 that combines the outputs of the encoder 130 and the predictor 140. The encoder 130 generates an audio frame at time t, and an acoustic embedding a t The embedding can compress the input feature space into a smaller feature space. An acoustic model (AM) can be incorporated into the encoder 130. The predictor 140 can then use the text embedding t h The text history can be encoded as y up to index h. A language model can be incorporated into the predictor 140. The text embedding vector and the encoder output can be vectors of real values ​​of fixed dimensions. These embeddings can be fed into a joiner 150, which combines them to produce y t,h This yields a probability distribution over the output units in

[0020] In various embodiments, frame-level alignment between audio and output symbols is not utilized to train or customize the RNN-T. Paired text and synthesized audio from the text can be used to train or customize the RNN-T, and an end-to-end model can be trained from paired audio and transcript without pre-computed alignment. The computer-implemented method can include updating the encoder network using the synthesized first-domain audio data and first-domain text data.

[0021] In various embodiments, an input feature sequence x, 110, can be provided to the encoder 130 of the recurrent neural network transducer (RNN-T) 100, where the input feature sequence x can be a time-ordered sequence of acoustic features represented as a vector. In various embodiments, x=(x1, x2,..., x T ) can be an input sequence of any length T. In one or more embodiments, the audio data can be synthesized from text available in the source domain. The text used to synthesize the audio data can be the same text used to train the original RNN-T ASR model. The acoustic model of the encoder 130 is constructed using the acoustic features x t a high-level representation

number

number

[0022] The predictor 140 can function as an RNN language model, which predicts the preceding non-blank target y u-1 Under these conditions, a high-level expression

number

number

[0023] Converting an audio signal into a sequence of words requires being able to distinguish speech sounds (such as phonemes or syllables) despite the obvious distortions created by different voices. RNNs can be applied to the problem of mapping input and output sequences. If the RNN output is probabilistic, a distribution over the output sequence of the same length as the input sequence can be generated. Speech recognition is performed by taking an input sequence x=x1,...,x T Given a word sequence W=w1,...,w, the most likely word sequence is n where T may represent the number of frames in the utterance.

[0024] In various embodiments, the output symbol sequence y, 120 may be fed to a predictive neural network 140 having a length of u-1 and may behave as a language model, where the output symbol sequence y is generated by a recurrent neural network transducer (RNN-T). u-1 denotes a previous prediction (e.g., symbol). The RNN-T predicts the next symbol y based on the previous symbol sequence leading up to u. u In various embodiments, y=(y1, y2,..., y U ) may be a sequence of length U of target output symbols belonging to set Y. The set of target symbols may be letters, phonemes, graphemes, wordpieces, etc.

[0025] In the RNN-T model, an extra blank symbol Φ can be introduced to extend a length-U sequence y into a set of length-(T+U) sequences Φ(y). The sequence length of the symbols or labels (U) and the acoustic feature frame (T) do not have to be the same length.

[0026] Because the output of the predictive neural network 140 is merged with the output of the encoder 130 in the joiner 150 before producing the output symbols, direct manipulation of the predictive network 140 for customization (adding words, phrases, and sentences) is not possible.

[0027] In various embodiments, the input vector x t and the output vector y u may be represented as a real-valued vector of fixed length; for example, in character-based speech recognition, each x t may be a vector of Mel-Frequency Cepstral Coefficients (MFCCs), and each y t can be a one-hot vector that encodes a specific character, and the Mel-Frequency Cepstral Coefficients (MFCCs) represent the spectrum of the short-term power of the sound. t and the output vector y u can be mapped to the input vector x t and the output vector y u may have different lengths T and U.

[0028] In various embodiments, the encoder 130 may be pre-trained first. Prior to training the RNN-T, the encoder network 130 and the prediction network 140 may be initialized.

[0029] In various embodiments, an acoustic model and a cross-entropy language model are obtained in advance, and the acoustic model may be a phoneme-acoustic model. In various embodiments, the cross-entropy language model may be a character cross-entropy language model, a subword cross-entropy language model, or a word cross-entropy language model.

[0030] In various embodiments, the encoder 130 of the end-to-end speech recognition model can be initialized based on the acoustic model. The encoder network 130 can also be updated using paired synthetic audio and associated text from the source domain. The "source" data is the training data used to train the original RNN-T before customization.

[0031] In various embodiments, the predictor network 140 operates as a language modeler that determines / predicts the text associated with the input audio.

[0032] In various embodiments, the prediction network 140 may be a recurrent neural network, and the prediction network 140 may have an input layer, an output layer, and one or more hidden layers. The size of the input layer may be the same as the length of the input vector, where K={k1, k2, ... k K}, and y u ga k k where n is an index with 1≦k≦K. The input can be encoded as a one-hot vector.

[0033] In various embodiments, the output of the prediction network 140 is merged with the output from the encoder network 130 before emitting the output symbols. In a character-based system, the output symbols from the RNN-T (after softmax calculation) are a set of characters and <blank>In various embodiments, <blank>No symbols are used in the history of the prediction network. It is not possible to directly manipulate the prediction network 140 for customization (adding words, phrases, and sentences). The outputs from both the encoder network 130 and the prediction network 140 are used to update the prediction network 140 or the entire RNN-T 100. However, the RNN-T 100 is only customized with text data; if audio data is not available, the output from the encoder network 130 is not available.

[0034] In various embodiments, the output feature sequence y, 180 may be the output sequence generated by the RNN-T 100 based on the input feature sequence x, 110 by searching over the output probability lattice defined by P(y|t,u), where y=(y1, y2,...y U-1 ,y U ) may be an output sequence of length U belonging to the set Y* of all sequences across some output space Y, where P(y|t,u) is the posterior probability of y given "t" and "u", where "t" is the index across time of the time-ordered sequence of acoustic features represented as a vector, and "u" is the index across the output sequence of length U. In various embodiments, y=(y1,y2,...,y u-1 ) is fed to the prediction network 140.

[0035] In various embodiments, a connectionist temporal classification (CTC) model is trained with acoustic features x, represented as vectors as input to obtain a phoneme-acoustic model, and phonemes as output. A neural network trained with this CTC model can be used to initialize the encoder network 130 of the RNN-T 100. The phoneme-acoustic model can be used as an initial acoustic model for the encoder network 130 of the RNN-T 100. The acoustic features can be used as input and a model unit can be set as output to train the initial acoustic model to obtain a target acoustic model.

[0036] In various embodiments, cross-entropy (CE) can be used to train a language model (LM). The CTC acoustic model and the CE language model can be used to initialize the encoder 130 (encoder) and the predictor 140 (decoder) of the end-to-end system, respectively. After initialization, the end-to-end system can have a suitable initial state.

[0037] Training of an RNN-T can start from a pre-trained model, or if there is no pre-training, training can start from random initialization.

[0038] In various embodiments, the language model may include an LSTM layer and a first input embedding layer. LSTMs can be used in both the encoder network 130 and the prediction network 140, although other types of neural networks, such as Transformers, can also be used in either or both of the encoder and prediction networks.

[0039] In various embodiments, the outputs generated by the encoder 130 and the predictor 140 may be combined by a joiner 150, which generates a high-level representation h t enc , h u pre (embedding vectors). The joiner network 150 can be used to compute the weighted sum of the encoder network h t enc and the output from the prediction network h u pre Combining the outputs from the embedding z t,u (logits). In various embodiments, the joiner 150 computes the encoder network output h as the sum of linear transformations of both embeddings. t enc and the predicted network output h u pre is a feedforward network that combines

[0040]

number

[0041] In the formula, W enc and W pre is the weight matrix, and b z is the bias vector,

number

[0042] z t,u is connected to the output layer with a linear transformation:

[0043]

number

[0044] where W is the weight matrix and b y is the bias vector.

[0045] In various embodiments, a softmax function 160 is applied to the output of the joiner 150, which normalizes the output of the joiner neural network 150 to generate a probability (posterior) distribution 170,

number

number

[0046] In various embodiments, the probability distribution 170 can be used to generate the output feature sequence y, 180, which can be generated by searching over the output probability lattice defined by P(y|t,u). The output symbol sequence y, 120 can be used to update the predictor 140. When training the model, beam search may not be used, since both the paired symbol sequence and the input audio features are given, allowing the posterior probability lattice to be calculated. Parameters are updated by minimizing the RNN-T loss, which is defined as the sum of the symbol posterior probabilities over all possible RNN-T alignments:

[0047]

number

[0048] where each sequence

number

number

number

[0049] In various embodiments, end-to-end training can be achieved from transcripts and audio only, eliminating the need for the iterative and lengthy training phases required for hybrid models. In various embodiments, a lexicon linking spelling and pronunciation is also not required. RNN-Ts can be trained from associated pairs of audio data and their transcriptions.

[0050] In one or more embodiments, audio data can be synthesized from text available in the source domain. The text used to synthesize the audio data can be the same text used to train the original RNN-T model. When updating the encoder network, we use the synthesized audio for the text used to train the original RNN-T model. The encoder network can then be updated conditioned on the appropriate output from the predictive network.

[0051] In various embodiments, the encoder network can be updated using pairs of synthesized audio and text from the source domain.

[0052] Audio data can also be synthesized for text in the target domain.

[0053] In various embodiments, the prediction network can be updated using pairs of synthetic audio and text from the target domain. When updating the prediction network, the encoder network is already adapted to the synthetic audio of the source domain. The prediction network can then be updated conditioned on the appropriate output from the encoder network.

[0054] In various embodiments, the encoder network can be restored to its original condition. Although the quality of synthetic audio has improved recently, it is not always worth updating the encoder network for speech recognition using synthetic audio. The final encoder network can be the same as the original encoder network, without being contaminated by synthetic audio from the target domain used in customization. The encoder can be reset to its initial state when the customized features are no longer desired.

[0055] FIG. 2 is a block / flow diagram illustrating an algorithm for training a recurrent neural network transducer (RNN-T) for speech recognition, according to an embodiment of the present invention.

[0056] In one or more embodiments, an algorithm 200 for training a recurrent neural network transducer (RNN-T) may involve creating an input feature sequence x of acoustic features represented as vectors, which may be audio data synthesized from text used to initially train the (original) RNN-T. An encoder network may be trained first. The audio data may be synthesized from text in a source domain, which may be represented by a standard corpus.

[0057] At block 210, a feature sequence x of acoustic features may be synthesized by reading text for the source domain.

[0058] At 215, the feature sequence x of acoustic features may be fed to the RNN-T encoder 130, where the acoustic features may be vectors of Mel-Frequency Cepstral Coefficients (MFCCs).

[0059] At block 220, the encoder may be trained using associated pairs of synthetic audio and source domain text. The encoder generates a hidden vector sequence (h0, h1, .., h T ) can be generated. The input audio features can have length T. The hidden vector h t may be calculated for each audio feature.

[0060] 225, the hidden vector sequence (h0, h1, .., h T ) may be supplied to the joiner 230.

[0061] In block 230, the joiner generates a hidden vector sequence (h0, h1, .., h t ) into the hidden vector sequence (h0, h1, .., h u-1 ) in combination with h T and h u The subscripts t and u

number

[0062] 235, the induced local field z t,u is fed into the softmax function 240.

[0063] At 240, the softmax function generates the posterior probability P(y|t,u) 250.

[0064] At 245, the posterior probability P(y|t,u) is output by the softmax function.

[0065] 250, and the output from 240 is P(y|t,u).

[0066] At 255, P(y|t,u) is provided to an output sequence generator 260 which produces an output symbol sequence y having length U.

[0067] In block 260, an output sequence generator 260 produces an output symbol sequence y having length U.

[0068] At 265, the output symbol sequence y having length U-1 is fed to the predictor neural network 270 to update the predictor neural network 270 and generate 120 y u-1 indicates the previous prediction (e.g., character). Based on the previous word sequence up to u-1, the RNN-T predicts the next symbol y u Predict.

[0069] In block 270, the predictor neural network 270 is updated to predict the hidden vector sequence (h0, h1, .., h U ) is generated.

[0070] FIG. 3 is a block / flow diagram illustrating an algorithm for synthesizing audio features from text, updating the encoder and predictor, and restoring the encoder according to an embodiment of the present invention.

[0071] At block 310, the source domain text to be used for training is identified.

[0072] Block 320 synthesizes source domain training audio data for the encoder from text from the source domain.

[0073] In block 330, the encoder network is updated, but the weights of the predictor neural network and the joiner neural network are kept constant (i.e., fixed).

[0074] At block 340, the target domain text to be used for training is identified.

[0075] At block 350, audio is synthesized for target domain training of predictors from text in the target domain. The synthesized audio for the target domain is fed into the encoder. In various embodiments, only the prediction network is updated; the encoder network is not updated with the synthesized audio for the target domain.

[0076] At block 360, the predictor network is updated while keeping the weights of the encoder neural network and the joiner neural network constant.

[0077] During actual deployment of the customized model, the encoder network is restored to its weights before feeding the target domain audio, at block 370. Restoring the weights to the encoder network to the state trained for the source domain can reset the encoder to its pre-customized state.

[0078] FIG. 4 is a diagram of a posterior lattice for an RNN-T, in accordance with an embodiment of the present invention.

[0079] y=(y1,y2,...,y U ) may denote a sequence of length U of target output symbols to translate linguistically.

[0080] x=(x1,x2,...,x T ) can denote the acoustic feature vector over T time steps.

[0081] Each node 400 is P(y t+u represents the posterior distribution P(y|t,u) defined by

[0082] FIG. 5 is an exemplary processing system 500 to which the method and system may be applied, according to an embodiment of the present invention.

[0083] In various embodiments, processing system 500 may include at least one processor (CPU) 504 and may have a graphics processing unit (GPU) 505 capable of performing vector calculations / manipulations operatively coupled to other components by a system bus 502. Cache 506, read-only memory (ROM) 508, random access memory (RAM) 510, input / output (I / O) adapter 520, audio adapter 530, network adapter 540, user interface adapter 550, and display adapter 560 may be operatively coupled to system bus 502.

[0084] A first storage device 522 and a second storage device 524 are operatively coupled to the system bus 502 by an I / O adapter 520. The storage devices 522 and 524 may be disk storage devices (e.g., magnetic or optical disk storage devices), solid-state devices, magnetic storage devices, etc. The storage devices 522 and 524 may be the same type of storage device or different types of storage devices.

[0085] Speakers 532 are operatively coupled to the system bus 502 by an audio adapter 530. A transceiver 542 is operatively coupled to the system bus 502 by a network adapter 540. A display device 562 is operatively coupled to the system bus 502 by a display adapter 560.

[0086] A first user input device 552, a second user input device 554, and a third user input device 556 are operably coupled to the system bus 502 by a user interface adapter 550. The user input devices 552, 554, and 556 may be any of a keyboard, mouse, keypad, image capture device, motion sensing device, microphone, a device incorporating the functionality of at least two of the foregoing devices, etc. Of course, other types of input devices may also be used while maintaining the spirit of the principles. The user input devices 552, 554, and 556 may be the same type of user input device or different types of user input devices. The user input devices 552, 554, and 556 may be used to input and output information to the system 500.

[0087] In various embodiments, processing system 500 may also include other elements (not shown), or omit certain elements, as would be readily apparent to one skilled in the art. For example, various other input or output devices, or combinations thereof, may be included in processing system 500, depending on the particular implementation, as would be readily apparent to one skilled in the art. For example, various types of wireless or wired input or output devices, or both, may be used. Also, additional processors, controllers, memory, etc., in various configurations, may also be utilized, as would be readily apparent to one skilled in the art. These and other variations of processing system 500 will be readily apparent to those skilled in the art given the teachings of the principles presented herein.

[0088] It should also be understood that system 500 is a computer system computing system for implementing particular embodiments of the present methods / systems. Part or all of processing system 500 may be implemented in one or more of the elements of Figures 1-4. It should also be understood that processing system 500 may perform at least some of the methods described herein, including, for example, at least some of the methods of Figures 1-4.

[0089] FIG. 6 is an exemplary processing system 600 configured to implement one or more neural networks for modeling road layouts, according to an embodiment of the present invention.

[0090] In one or more embodiments, the processing system 600 may be the computer system 500 configured to perform the computer-implemented method of customizing a recurrent neural network transducer for speech recognition.

[0091] In one or more embodiments, the processing system 600 can be the computer system 500 that includes a memory component 670, including, but not limited to, the computer system's random access memory (RAM) 510, a hard drive 522, and / or cloud storage for storing and implementing a computer-implemented method for understanding road layouts from video images. The memory component 670 can also utilize a database for organizing the memory storage.

[0092] In various embodiments, the memory component 670 may include an encoder neural network 610 that may be configured to implement multiple acoustic models configured to model acoustic input and perform automatic speech recognition (ASR). In various embodiments, the encoder neural network 610 may be implemented as a long short-term memory (LSTM) or a bidirectional LSTM (BLSTM). The encoder neural network 610 may also be configured to receive as input an acoustic signal. The input may be a sequential set of audio data received by the microphone 556. The encoder neural network 610 may also be configured to generate an output value that is an embedding.

[0093] In various embodiments, the memory component 670 may include a predictor neural network 620 that is configured to train one or more acoustic models and that can be configured to generate encoder embeddings to perform automatic speech recognition (ASR). In various embodiments, the predictor neural network 620 can be implemented as a long short-term memory (LSTM). The predictor neural network 620 can also be configured to generate output values ​​that are embeddings.

[0094] In various embodiments, the memory component 670 can include a joiner neural network 630 that can be configured to combine two separate input data sets from the encoder and the predictor, where the data is stored as a feature / vector h t and h u The Joiner neural network 630 can be configured to produce an output, which can be a character, a subword, or a word.

[0095] In various embodiments, the memory component 670 may include an output generator 640 configured to produce an output symbol sequence y, 120 having a length u. The output generator 640 may be configured to receive an output from the joiner neural network 630.

[0096] In various embodiments, the memory component 670 may include a softmax function 650 configured to generate predictions from the output values ​​of the Joiner neural network 630 .

[0097] In various embodiments, the memory component 670 may include a synthesizer 660 configured to synthesize first domain audio data from first domain text data and / or second domain audio data from second domain text data.

[0098] FIG. 7 is a block diagram illustrating an exemplary neural network according to another embodiment of the present invention.

[0099] Neural network 700 may include multiple neurons / nodes, and output nodes may communicate using one or more of multiple connections 708. Neural network 700 may include multiple layers, including, for example, one or more input layers 702, one or more hidden layers 704, and one or more output layers 706. In one embodiment, nodes in each layer may be used to apply any function (e.g., input program, input data, etc.) to any previous layer to produce an output, and hidden layers 704 may be used to transform inputs from the input layer (or any other layer) into outputs for nodes at different levels.

[0100] FIG. 8 is a block diagram illustrating an exemplary cloud computing environment having one or more cloud computing nodes with which local computing devices used by cloud consumers communicate, according to an embodiment.

[0101] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0102] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0103] The features are as follows: On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed without requiring human interaction with the service provider. Wide network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (eg, cell phones, laptops, PDAs). Resource Pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, with various physical and virtual resources dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge of the exact location of the resources provided, but do have a sense of location independence in that they may be able to specify location at a higher level of abstraction (e.g., country, state, or data center). Rapid Elasticity: Capacity can be provisioned quickly and elastically, sometimes automatically, quickly scaled out, quickly released, and quickly scaled in. To the consumer, the capacity available for provisioning often appears unlimited and can be purchased in any amount at any point in time. Measured Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services they utilize.

[0104] The service model is as follows: Software as a Service (SaaS): The consumer is offered the ability to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The ability offered to consumers is to deploy applications they create or acquire, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does have control over the applications deployed and, in some cases, the application hosting environment configuration. Infrastructure as a Service (IaaS): The ability offered to consumers is to provision processing, storage, network, and other basic computing resources on which they can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does control the operating system, storage, deployed applications, and in some cases, limited control over selected networking components (e.g., host firewalls).

[0105] The deployment model is as follows: Private Cloud: Cloud infrastructure is operated exclusively for an organization. A private cloud may be managed by the organization or a third party and may exist on-premise or off-premise. Community Cloud: Cloud infrastructure is shared by multiple organizations to support a specific community with shared concerns (e.g., mission, security requirements, policies, compliance considerations). It can be managed by the organization or a third party and can reside on-premises or off-premises. Public Cloud: Cloud infrastructure is made available to the general public or large industry entities and is owned by an organization that sells cloud services. Hybrid Cloud: A cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are joined by standardized or proprietary technologies that allow for data and application portability (e.g., cloud bursting for load balancing between clouds).

[0106] Cloud computing environments are service-oriented, emphasizing statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0107] Referring now to FIG. 8, an exemplary cloud computing environment 950 is illustrated. As shown, cloud computing environment 950 includes one or more cloud computing nodes 910 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 951, a desktop computer 952, a laptop computer 953, and / or an automobile computer system 954, may communicate. The nodes 910 may communicate with each other. They may be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or a combination thereof, as described above. This enables cloud computing environment 950 to provide infrastructure, platform, or software, or a combination thereof, as a service without the cloud consumer having to maintain resources on their local computing device. It should be understood that the types of computing devices 951, 952, 953, 954 shown in FIG. 8 are intended to be illustrative only, and that the computing node 110 and cloud computing environment 950 can communicate with any type of computerized device via any type of network or network-addressable connection or combination thereof (e.g., using a web browser).

[0108] FIG. 9 is a block diagram illustrating a set of functional abstraction layers provided by a cloud computing environment, in accordance with an embodiment of the present invention.

[0109] Referring now to Figure 9, a set of functional abstraction layers provided by cloud computing environment 950 (Figure 7) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 8 are intended to be exemplary only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0110] Hardware and software layer 1060 includes hardware and software components. Examples of hardware components include mainframes 61, RISC (reduced instruction set computer) architecture-based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0111] The virtualization layer 1070 provides an abstraction layer within which examples of virtual entities can be provided: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0112] In one example, management layer 1080 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing and other resources used to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification of cloud consumers and tasks, as well as protection of data and other resources. User portal 83 provides access to the cloud computing environment to consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management so that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides advance arrangement and procurement of cloud computing resources in anticipation of future requirements according to SLAs.

[0113] The workload layer 1090 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this layer include mapping and navigation 91; software development and lifecycle management 92; virtual classroom instruction delivery 93; data analytics processing 94; transaction processing 95; and a recurrent neural network transducer (RNN-T) implementing automatic speech recognition (ASR) 96, where the teacher neural network can be a recurrent neural network configured to train for automatic speech recognition and prepare the student neural network.

[0114] As used herein, the terms “hardware processor subsystem” or “hardware processor” can refer to a processor, memory, software, or combination thereof working together to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem can include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements can be included in a central processing unit, a graphics processing unit, or a separate processor or computing element-based controller (e.g., logic gates, etc.), or a combination thereof. The hardware processor subsystem can include one or more on-board memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, the hardware processor subsystem can include one or more memories that can be on-board, dedicated, or dedicated for use by the hardware processor subsystem (e.g., ROM, RAM, basic input / output system (BIOS), etc.).

[0115] In some embodiments, a hardware processor subsystem may include or execute one or more software elements, which may include an operating system, one or more applications, or both, and / or specific code for achieving a specified result.

[0116] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application specific integrated circuits (ASICs), FPGAs, or PLAs, or a combination thereof.

[0117] These and other variations of hardware processor subsystems are contemplated according to embodiments of the present invention.

[0118] The present invention may be a system, method, or computer program product, or combination thereof, integrated at any conceivable level of technical detail. The computer program product may include a computer-readable storage medium (or multiple computer-readable storage media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0119] A computer-readable storage medium can be a tangible device that can store and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or ridge structures in grooves on which instructions are recorded, and any suitable combination thereof. Computer-readable storage medium, as used herein, should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0120] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device over a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof). This network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium within each computing / processing device.

[0121] Computer-readable program instructions for carrying out operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® or C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, to carry out aspects of the present invention, electronic circuitry including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to customize the electronic circuitry by utilizing state information of the computer-readable program instructions.

[0122] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0123] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus and / or other device to function in a particular manner, whereby a computer-readable storage medium having instructions stored thereon comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0124] Computer-readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams, thereby causing a series of operable steps to be performed on the computer, other programmable apparatus, or other device that produce a computer-implemented process.

[0125] References herein to "one embodiment" or "an embodiment" of the invention, as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment of the invention. Thus, the appearances of the phrases "in one embodiment" or "an embodiment," and any other variations thereof, in various places throughout this specification are not necessarily all referring to the same embodiment.

[0126] For example, it should be understood that the use of any of the following terms, " / ," "or ... or a combination thereof," and "at least one of," in the cases of "A / B," "A or B or a combination thereof," and "at least one of A and B," is intended to encompass the selection of only the first enumerated option (A), or the selection of only the second enumerated option (B), or the selection of both options (A and B). As a further example, in the cases of "A, B, or C or a combination thereof" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first enumerated option (A), or the selection of only the second enumerated option (B), or the selection of only the third enumerated option (C), or the selection of only the first and second enumerated options (A and B), or the selection of only the first and third enumerated options (A and C), or the selection of only the second and third enumerated options (B and C), or the selection of all three options (A, B, and C). This may be extended for as many items as are listed, as would be readily apparent to one skilled in the art.

[0127] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may actually be implemented as a single step, or may be executed concurrently, substantially concurrently, partially, or fully in a time-overlapping manner, or the blocks may possibly be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks included in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations or executes a combination of special-purpose hardware and computer instructions.

[0128] While preferred embodiments of the system and method have been described (which are intended to be illustrative only and not limiting), it should be noted that modifications and variations can be made by those skilled in the art given the above teachings. It is therefore to be understood that changes may be made within the specific embodiments disclosed which are within the scope of the invention and outlined by the appended claims. Having thus described aspects of the invention with the detail and specificity required by the patent laws, it is the appended claims which set forth what is claimed and desired to be protected by Letters Patent. According to the present disclosure, the following items are also disclosed: (Item 1) synthesizing first-domain audio data from the first-domain text data; providing the synthesized first-domain audio data to a trained encoder of a recurrent neural network transducer (RNN-T) having initial conditions, wherein the encoder is updated using the synthesized first-domain audio data and the first-domain text data; synthesizing second domain audio data from the second domain text data; feeding the synthesized second-domain audio data to the updated encoder of the recurrent neural network transducer (RNN-T), wherein a predictive network is updated using the synthesized second-domain audio data and the second-domain text data; and restoring the updated encoder to the initial conditions. 1. A computer-implemented method for customizing a recurrent neural network transducer (RNN-T), comprising: (Item 2) Item 10. The computer-implemented method of item 1, wherein the recurrent neural network transducer (RNN-T) includes a joiner that combines the output of the encoder with the output of a predictor. (Item 3) The joiner generates an induced local field z that is fed into a softmax function. t,u Item 3. The computer-implemented method of item 2, wherein the computer-implemented method produces an output as: (Item 4) Item 4. The computer-implemented method of item 3, wherein the softmax function generates a posterior probability P(y|t,u). (Item 5) The posterior probability generator P(y|t,u) generates an output sequence of length U, y=(y 1 ,y 2 ,...y U-1 ,y U Item 5. The computer-implemented method of item 4, generating an output in which (Item 6) Item 6. The computer-implemented method of item 5, wherein the input feature sequence x is derived from the synthesized first domain audio data. (Item 7) one or more processor devices; memory in communication with at least one of the one or more processor devices; and display screen; of, The memory includes: an encoder configured to receive synthesized first-domain audio data generated from first-domain text data, wherein the encoder is a recurrent neural network transducer (RNN-T) trained encoder having initial conditions, wherein the encoder is configured to be updated from the initial conditions using the synthesized first-domain audio data and the first-domain text data, wherein the encoder is further configured to receive synthesized second-domain audio data generated from second-domain text data; and An output sequence generator that produces an output symbol sequence y based on an input feature sequence x, which is a time-ordered sequence of acoustic features represented as a vector. A system for customizing a recurrent neural network transducer (RNN-T), comprising: (Item 8) 8. The system of claim 7, wherein the memory further comprises a joiner configured to combine an output of the trained encoder with an output of a predictor. (Item 9) The joiner outputs an induced local field z t,u Item 9. The system according to item 8, (Item 10) The memory stores the induced local field z t,u Item 10. The system of item 9, further comprising a softmax function configured to receive and generate an output. (Item 11) The output symbol sequence y=(y 1 ,y 2 ,...y U-1 ,y U ) is an output sequence of length U based on an input feature sequence x, which is a time-ordered sequence of acoustic features represented as vectors. (Item 12) 12. The system of claim 11, wherein the memory further comprises a synthesizer configured to synthesize first-domain audio data from first-domain text data and second-domain audio data from second-domain text data. (Item 13) Item 13. The system of item 12, wherein the encoder is further configured to be restored to the initial condition. (Item 14) one or more computer-readable storage media; and program instructions collectively stored on said one or more computer-readable storage media; Equipped with The program instructions may cause a computer to: synthesizing first-domain audio data from the first-domain text data; providing the synthesized first-domain audio data to a trained encoder of a recurrent neural network transducer (RNN-T) with initial conditions, wherein the encoder is updated using the synthesized first-domain audio data and the first-domain text data; synthesizing second domain audio data from the second domain text data; feeding the synthesized second-domain audio data to the updated encoder of the recurrent neural network transducer (RNN-T), wherein a predictor is updated using the synthesized second-domain audio data and the second-domain text data; and Restoring the updated encoder to the initial conditions. a computer program product for customizing a recurrent neural network transducer (RNN-T), the computer program product being executable by the computer to cause the computer to perform the steps of: (Item 15) Item 15. The computer program product of item 14, wherein the recurrent neural network transducer (RNN-T) includes a joiner that combines an output of the trained encoder with an output of the predictor. (Item 16) The joiner generates an induced local field z that is fed into a softmax function. t,u Item 16. The computer program product of item 15, which produces an output as (Item 17) Item 17. The computer program product of item 16, wherein the softmax function generates a posterior probability P(y|t,u). (Item 18) The posterior probability generator P(y|t,u) generates an output sequence y=(y 1 ,y 2 ,...y U-1 ,y U Item 18. The computer program product of item 17, generating an output that is (Item 19) Item 19. The computer program product of item 18, wherein the first domain audio data to be synthesized is the input feature sequence x. (Item 20) synthesizing first-domain audio data from the first-domain text data; providing the synthesized first-domain audio data to a trained encoder of a recurrent neural network transducer (RNN-T) having initial conditions, wherein the encoder is updated using the synthesized first-domain audio data and the first-domain text data, and converting the synthesized first-domain audio data into an acoustic embedding a t and encode the audio embedding a t compressing the synthesized first domain audio data into a smaller feature space; The acoustic embedding t supplying the same to a joiner; synthesizing second domain audio data from the second domain text data; providing the synthesized second domain audio data to the updated encoder, wherein the updated encoder converts the synthesized second domain audio data into the acoustic embedding b. t where the acoustic embedding b t compressing the synthesized second domain audio data into a smaller feature space; feeding the output sequence from the joiner to a predictor of the recurrent neural network transducer (RNN-T), wherein the predictor is updated using the output sequences from the synthesized second-domain audio data and the second-domain text data; and restoring the updated encoder to the initial conditions. 1. A computer-implemented method for customizing a recurrent neural network transducer (RNN-T), comprising: (Item 21) The joiner is the acoustic embedding a t Item 21. The computer-implemented method of item 20, wherein the embeddings from the predictors are combined through a weighted sum. (Item 22) The joiner generates an induced local field z that is fed into a softmax function. t,u Item 22. The computer-implemented method of item 21, wherein the method produces an output as (Item 23) Item 23. The computer-implemented method of item 22, wherein the softmax function generates a posterior probability P(y|t,u). < / blank> < / blank>

Claims

1. A computer-implemented method for customizing a recurrent neural network transducer (RNN-T) applied to speech recognition, comprising: The recurrent neural network transducer (RNN-T) comprises an encoder for audio, a predictive network for text, and a joiner that combines the outputs of the encoder and the predictive network; synthesizing first domain audio data from the first domain text data; providing the synthesized first-domain audio data to the trained encoder of the recurrent neural network transducer (RNN-T) with initial conditions, wherein the encoder is updated using the synthesized first-domain audio data and the first-domain text data; synthesizing second domain audio data from the second domain text data; providing the synthesized second-domain audio data to the updated encoder of the recurrent neural network transducer (RNN-T), wherein the predictive network of the recurrent neural network transducer (RNN-T) is updated using the synthesized second-domain audio data and the second-domain text data; and restoring the updated encoder to the initial conditions. A computer-implemented method comprising:

2. The joiner generates an induced local field z ｔ，ｕ The computer-implemented method of claim 1 , wherein the computer-implemented method produces an output as:

3. The computer-implemented method of claim 2 , wherein the softmax function generates a posterior probability P(y|t, u).

4. The posterior probability generator P(y|t,u) is an output sequence of length U based on an input feature sequence x, which is a time-ordered sequence of acoustic features represented as a vector, where x is the output sequence y=(y １ , y ２ ,...y Ｕ－１ , y Ｕ 4. The computer-implemented method of claim 3, wherein the computer-implemented method generates an output in which:

5. The computer-implemented method of claim 4 , wherein the input feature sequence x is derived from the synthesized first-domain audio data.

6. A system for customizing a recurrent neural network transducer (RNN-T) applied to speech recognition, comprising: one or more processor devices; memory in communication with at least one of the one or more processor devices; and Display screen; Equipped with The memory includes: an encoder configured to receive synthesized first-domain audio data generated from first-domain text data, the encoder being a trained version of the recurrent neural network transducer (RNN-T) having initial conditions, the encoder being configured to be updated from the initial conditions using the synthesized first-domain audio data and the first-domain text data, the encoder being configured to receive synthesized second-domain audio data generated from second-domain text data, the updated encoder being further configured to be restored to the initial conditions; a predictor configured to receive a preceding output symbol sequence, the predictor being included in the recurrent neural network transducer (RNN-T) and configured to be updated using the synthesized second-domain audio data and the second-domain text data; a joiner included in the recurrent neural network transducer (RNN-T) configured to combine the output of the encoder with the output of the predictor; and an output sequence generator that receives the output from the joiner and produces an output symbol sequence y based on the input feature sequence x, which is a time-ordered sequence of acoustic features represented as a vector; A system having:

7. The joiner outputs an induced local field z ｔ，ｕ The system of claim 6 .

8. The memory stores the induced local field z ｔ，ｕ 8. The system of claim 7, further comprising a softmax function configured to receive and generate an output.

9. The output symbol sequence y=(y １ , y ２ ,...y Ｕ－１ , y Ｕ 9. The system of claim 8, wherein x is an output sequence of length U based on an input feature sequence x, which is a time-ordered sequence of acoustic features represented as a vector.

10. 10. The system of claim 9, wherein the memory further comprises a synthesizer configured to synthesize first domain audio data from first domain text data and second domain audio data from second domain text data.

11. A computer program executable by a computer for customizing a recurrent neural network transducer (RNN-T) applied to speech recognition, comprising: the recurrent neural network transducer (RNN-T) comprises an encoder for audio, a predictor for text, and a joiner that combines the output of the encoder and the output of the predictor; The computer program causes the computer to: synthesizing first domain audio data from the first domain text data; providing the synthesized first-domain audio data to the trained encoder of the recurrent neural network transducer (RNN-T) with initial conditions, wherein the encoder is updated using the synthesized first-domain audio data and the first-domain text data; synthesizing second domain audio data from the second domain text data; providing the synthesized second-domain audio data to the updated encoder of the recurrent neural network transducer (RNN-T), wherein the predictor of the recurrent neural network transducer (RNN-T) is updated using the synthesized second-domain audio data and the second-domain text data; and Restoring the updated encoder to the initial conditions. A computer program for performing the following.

12. The joiner generates an induced local field z ｔ，ｕ 12. The computer program of claim 11, which produces an output as:

13. The computer program product of claim 12 , wherein the softmax function generates a posterior probability P(y|t, u).

14. The posterior probability generator P(y|t,u) generates an output sequence y=(y １ , y ２ ,...y Ｕ－１ , y Ｕ 14. The computer program of claim 13, which generates an output that is 15. The computer program of claim 14, wherein the first domain audio data is the input feature sequence x.

16. synthesizing first domain audio data from the first domain text data; providing the synthesized first-domain audio data to a trained encoder of a recurrent neural network transducer (RNN-T) with initial conditions, wherein the encoder is updated using the synthesized first-domain audio data and the first-domain text data, and converting the synthesized first-domain audio data into an acoustic embedding a ｔ and encoding the audio embedding a ｔ compresses the synthesized first domain audio data into a smaller feature space; The acoustic embedding ｔ to a joiner; synthesizing second domain audio data from the second domain text data; providing the synthesized second domain audio data to the updated encoder, wherein the updated encoder converts the synthesized second domain audio data into the acoustic embedding b. ｔ where the acoustic embedding b ｔ compresses the synthesized second domain audio data into a smaller feature space; feeding the output sequence from the joiner to a predictor of the recurrent neural network transducer (RNN-T), wherein the predictor is updated using the output sequences from the synthesized second-domain audio data and the second-domain text data; and restoring the updated encoder to the initial conditions.

1. A computer-implemented method for customizing a recurrent neural network transducer (RNN-T), comprising:

17. The joiner is ｔ 17. The computer-implemented method of claim 16, wherein the embeddings from the predictors are combined through a weighted sum.

18. The joiner generates an induced local field z ｔ，ｕ 20. The computer-implemented method of claim 17, wherein the method produces an output as:

19. 20. The computer-implemented method of claim 18, wherein the softmax function generates a posterior probability P(y|t, u).

Images