Customization of recurrent neural network transcriber for speech recognition

Abstract

A computer-implemented method for customizing a recurrent neural network transcriber (RNN-T) is provided. The computer-implemented method includes synthesizing first domain audio data from first domain text data and feeding the synthesized first domain audio data into a trained encoder of a recurrent neural network transcriber (RNN-T) with an initial condition, wherein the encoder is updated using the synthesized first domain audio data and the first domain text data. The computer-implemented method also includes synthesizing second domain audio data from second domain text data and feeding the synthesized second domain audio data into the updated encoder of the recurrent neural network transcriber (RNN-T), wherein a prediction network is updated using the synthesized second domain audio data and the second domain text data. The computer-implemented method further includes restoring the updated encoder to the initial condition.

Description

Technical Field

[0001] This invention generally relates to speech recognition, and more particularly to methods and systems for training end-to-end speech recognition models. Background Technology

[0002] A recurrent neural network (RNN) is an artificial neural network in which connections between nodes form a directed graph along a time series. This allows RNNs to analyze sequence dependencies between attributes such as phonemes. Derived from feedforward neural networks, RNNs can use their internal state (memory) to process variable-length input sequences. RNNs can also directly encode ordering information. RNNs can receive and process input in the same order as in the original sequence. This makes them applicable to tasks such as unsegmented, connected handwriting recognition or speech recognition. The input values ​​for the time series can be real-valued or symbolic.

[0003] RNNs can have a fixed number of parameters and can also handle a variable number of inputs. There can be a one-to-one relationship between the number of positions in a sequence and the number of layers in the network. Each layer can have a single input for a specific position in the sequence (e.g., a time step). Thus, the input can interact with the hidden layers based on its position in the sequence. The layer architecture repeats in time, hence the term recursive. RNNs may require a predefined alignment between the input and output sequences to perform transduction. This can be a limitation, as finding the alignment can be a very difficult aspect of the sequence transduction problem.

[0004] End-to-end (E2E) automatic speech recognition (ASR) systems can directly transcribe an acoustic feature sequence into an output symbol (phoneme, character, word, etc.) sequence by mapping acoustic features to an output symbol sequence. An end-to-end model for ASR can directly output a word transcript given an input audio file. Summary of the Invention

[0005] According to embodiments of the present invention, a computer-implemented method for customizing a recurrent neural network transducer (RNN-T) is provided. The computer-implemented method includes synthesizing first-domain audio data from first-domain text data, and feeding the synthesized first-domain audio data into a trained encoder of the recurrent neural network transducer (RNN-T) with initial conditions, wherein the encoder is updated using the synthesized first-domain audio data and the first-domain text data. The computer-implemented method further includes synthesizing second-domain audio data from second-domain text data, and feeding the synthesized second-domain audio data into the updated encoder of the recurrent neural network transducer (RNN-T), wherein the prediction network is updated using the synthesized second-domain audio data and the second-domain text data. The computer-implemented method further includes restoring the updated encoder to its initial conditions.

[0006] According to another embodiment of the present invention, a system for customizing a recurrent neural network transcriber (RNN-T) is provided. The system includes: one or more processor devices; a memory communicating with at least one of the one or more processor devices; and a display screen, wherein the memory includes a synthesizer configured to synthesize first-domain audio data from first-domain text data and second-domain audio data from second-domain text data; and an encoder configured to receive the synthesized first-domain audio data generated from the first-domain text data, wherein the encoder is a trained encoder of a recurrent neural network transcriber (RNN-T) with initial conditions, wherein the encoder is configured to update from the initial conditions using the synthesized first-domain audio data and the first-domain text data, and wherein the encoder is further configured to receive the synthesized second-domain audio data generated from the second-domain text data. The memory also includes an output sequence generator that generates an output symbol sequence y based on an input feature sequence x, the input feature sequence x being a time-ordered sequence of acoustic features represented as vectors.

[0007] According to another embodiment of the present invention, a computer program product for customizing a recurrent neural network transcriber (RNN-T) is provided. The computer program product includes one or more computer-readable storage media and program instructions co-stored on the one or more computer-readable storage media, the program instructions being executable by a computer. When executed, the computer program product causes the computer to synthesize first-domain audio data from first-domain text data and feed the synthesized first-domain audio data into a trained encoder of the recurrent neural network transcriber (RNN-T) with initial conditions, wherein the encoder is updated using the synthesized first-domain audio data and the first-domain text data. When executed, the computer program product also causes the computer to synthesize second-domain audio data from second-domain text data and feed the synthesized second-domain audio data into the updated encoder of the recurrent neural network transcriber (RNN-T), wherein the predictor is updated using the synthesized second-domain audio data and the second-domain text data. When executed, the computer program product also causes the computer to restore the updated encoder to its initial conditions.

[0008] According to another embodiment of the present invention, a computer-implemented method for customizing a recurrent neural network transcriber (RNN-T) is provided. This computer-implemented method includes synthesizing first-domain audio data from first-domain text data and feeding the synthesized first-domain audio data into a trained encoder of a recurrent neural network transcriber (RNN-T) with initial conditions, wherein the encoder is updated using the synthesized first-domain audio data and the first-domain text data, and the encoder encodes the synthesized first-domain audio data into an acoustic embedding. t In the middle, acoustic embedding a t The synthesized first-domain audio data is compressed into a smaller feature space. This computer-implemented method also includes acoustic embedding... t The data is fed to a joiner, and second-domain audio data is synthesized from the second-domain text data. The computer-implemented method further includes feeding the synthesized second-domain audio data into an updated encoder, wherein the updated encoder encodes the synthesized second-domain audio data into an acoustic embedding. t Among them, acoustic embedding b t The synthesized second-domain audio data is compressed into a smaller feature space, and the output sequence from the combiner is fed into the predictor of a recurrent neural network transcriptome (RNN-T), where the predictor is updated using the output sequences from the synthesized second-domain audio data and the second-domain text data. The computer-implemented method also includes restoring the updated encoder to its initial conditions.

[0009] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments of the invention, which are read in conjunction with the accompanying drawings. Attached Figure Description

[0010] The following description will provide details of preferred embodiments with reference to the following figures, in which:

[0011] Figure 1 This is a diagram of an architecture of a recurrent neural network transcriber (RNN-T) that can be applied to speech recognition according to an embodiment of the present invention;

[0012] Figure 2 This is a block diagram / flowchart illustrating an algorithm for training a recurrent neural network transcriber (RNN-T) for speech recognition according to an embodiment of the present invention;

[0013] Figure 3 This is a block diagram / flowchart illustrating an algorithm according to an embodiment of the present invention for synthesizing audio features from text, updating the encoder and predictor, and recovering the encoder;

[0014] Figure 4 This is a diagram of a posterior mesh for RNN-T according to an embodiment of the present invention;

[0015] Figure 5 This is an exemplary processing system to which the method and system can be applied according to embodiments of the present invention;

[0016] Figure 6 This is an exemplary processing system configured to implement one or more neural networks for modeling road layouts according to an embodiment of the present invention;

[0017] Figure 7 This is a block diagram schematically depicting an exemplary neural network according to another embodiment of the present invention;

[0018] Figure 8 This is a block diagram illustrating an illustrative cloud computing environment with one or more cloud computing nodes according to an embodiment, wherein a local computing device used by a cloud consumer communicates with the cloud computing nodes; and

[0019] Figure 9 This is a block diagram illustrating a set of functional abstraction layers provided by a cloud computing environment according to an embodiment. Detailed Implementation

[0020] Embodiments of the present invention provide systems and methods for customizing language models from sufficiently strong basic RNN-T models to specific target domains. The RNN-T can be pre-trained from scratch. The trained RNN-T can then be customized by the user to a specific domain.

[0021] Beam search can be performed on a neural network for inference, resulting in lower computational cost, smaller memory footprint, and a simpler inference engine.

[0022] In various embodiments, the method can solve RNN-T-based modeling, in which a “language model” and “vocabulary” are integrated into a neural network along with other modules and cannot be directly manipulated from the outside after the network has been trained and deployed.

[0023] Exemplary applications / uses to which this invention can be applied include, but are not limited to, automatic speech recognition (ASR), natural language translation, etc.

[0024] It should be understood that aspects of the invention will be described according to the given illustrative framework; however, other frameworks, structures, and process features and steps may be modified within the scope of the aspects of the invention.

[0025] Now referring to the accompanying drawings, where the same reference numerals denote the same or similar elements, first refer to... Figure 1 The diagram illustrates the architecture of a recurrent neural network transcriber (RNN-T) that can be applied to speech recognition according to an embodiment of the present invention.

[0026] Automatic speech recognition (ASR) using deep neural networks (DNNs) can employ a hybrid framework that simultaneously implements several models. Models used in a hybrid system can include an acoustic model (AM) and a language model (LM). An RNN-T model can comprise three parts: a predictor network that encodes the labeled sequence into text embeddings, an encoder network that encodes the observation vector sequence into acoustic embeddings, and a neural network (e.g., a combiner) with a softmax output layer that combines the text and acoustic embeddings.

[0027] Language model customization enables developers and users to add application-specific and use-case-specific words, phrases, and sentences to a trained language model for automatic speech recognition (ASR). In other words, it allows for the customization of a previously trained and deployed language model (LM), providing flexible and efficient customization capabilities. In various embodiments, in hybrid systems, the acoustic model, language model, and vocabulary are explicitly modularized. Therefore, language model customization can be easily implemented because the language model and vocabulary can be directly manipulated and modified even after training and deployment.

[0028] In one or more embodiments, the RNN-T 100 for ASR may have three components, including an encoder 130 for audio, a predictor 140 for text, and a combiner 150 for combining the outputs of the encoder 130 and the predictor 140. The encoder 130 can encode audio frames into acoustic embeddings at time t. t The embedding can compress the input feature space into a smaller feature space. An acoustic model (AM) can be incorporated into encoder 130. Predictor 140 can encode the text history up to index h into a text embedding t. h The language model can be incorporated into predictor 140. One or more text embedding vectors and one or more encoder outputs can be fixed-dimensional real-valued vectors. These embeddings can be fed into merger 150, which combines them to obtain the language model. t,h The probability distribution generated at the output unit.

[0029] In various embodiments, frame-level alignment between audio symbols and output symbols is not used for training or customizing the RNN-T. A pair of text and synthesized audio from that text can be used to train or customize the RNN-T, wherein an end-to-end model can be trained based on the audio and transcript pair without pre-computed alignment. Computer-implemented methods may include updating the encoder network by using synthesized first-domain audio data and first-domain text data.

[0030] In various embodiments, the input feature sequence x 110 can be fed into the encoder 130 of a recurrent neural network transcriber (RNN-T) 100, wherein the input feature sequence x can be a time-ordered sequence of acoustic features represented as vectors. In various embodiments, x = (x1, x2, ..., x...). T The input sequence can be of any length T. In one or more embodiments, audio data can be synthesized from text available in the source domain. The text used to synthesize the audio data can be the same text used in the training of the original RNN-T ASR model. The acoustic model of encoder 130 can convert one or more acoustic features x t Convert to high-level representation Where t is the time index. in, It can be an embedding vector sequence of length T, where the embedding can compress the input feature space into a smaller output vector (dense vector) of fixed size and length T. In various embodiments, encoder 130 can be a unidirectional encoder network or a bidirectional encoder network.

[0031] Predictor 140 can be used as an RNN language model, which learns from the previously non-blank target y predicted by the RNN-T model.u-1 To generate high-level representations based on conditions Where u is the output label index. in, It is an embedding vector.

[0032] Transforming audio signals into word sequences requires the ability to recognize speech sounds (such as phonemes or syllables) regardless of the obvious distortions caused by different speech sounds. RNNs can be applied to the problem of mapping input and output sequences. When the RNN output is a probability, a distribution over an output sequence of the same length as the input sequence can be obtained. Speech recognition involves determining the most probable word sequence W = w1,...,w n Given an acoustic input sequence x = x1,...,x T , where T can represent the number of frames in the utterance.

[0033] In various embodiments, an output symbol sequence y 120 of length u-1 can be fed into a predictive neural network 140 that can serve as a language model, wherein the output symbol sequence y is generated by a recurrent neural network transcriber (RNN-T). y in 120 u-1 Indicates previous predictions (e.g., symbols). RNN-T predicts the next symbol y based on the previous symbol sequence up to u-1. u In various embodiments, y = (y1, y2, ..., y...). U ) can be a target output symbol sequence of length U belonging to set Y.

[0034] In RNN-T modeling, additional whitespace symbols can be introduced. The sequence y of length U is expanded into a set of sequences Φ(y) of length -(T+U). The length of the symbol or tag sequence (U) and the length of the acoustic feature frame (T) may not be the same.

[0035] Since the output of the prediction neural network 140 is merged with the output of the encoder 130 at the combiner 150 before the output symbols are generated, it is not possible to directly manipulate the prediction network 140 for customization (adding words, phrases, and sentences).

[0036] In various embodiments, the input vector x t and output vector y u It can be represented as a fixed-length real-valued vector; for example, for character-based speech recognition, each x... t It can be a vector of Mel frequency cepstral coefficients (MFCCs), and each y t It can be a one-hot vector encoded for a specific character, where the Merr frequency cepstral coefficients (MFCCs) represent the short-term power spectrum of the sound. Input vector x tand the output vector y u can be mapped, wherein the input vector x t and the output vector y u can have different lengths T and U.

[0037] In various embodiments, the encoder 130 can initially be pre-trained. Before training the RNN-T, the encoder network 130 and the prediction network 140 can be initialized.

[0038] In various embodiments, an acoustic model and a cross-entropy language model are obtained in advance, wherein the acoustic model can be a phoneme acoustic model. In various embodiments, the cross-entropy language model can be a character cross-entropy language model, a sub-word cross-entropy language model, or a word cross-entropy language model.

[0039] In various embodiments, the encoder 130 of the end-to-end speech recognition model can be initialized based on the acoustic model. The encoder network 130 can also be updated using a pair of synthetic audio and associated text from the source domain. The "source" data is the training data used to train the original RNN-T before customization.

[0040] In various embodiments, the predictor network 140 acts as a language modeler that determines / predicts the text associated with the input audio.

[0041] In various embodiments, the prediction network 140 can be a recurrent neural network, wherein the prediction network 140 can have an input layer, an output layer, and one or more hidden layers. The size of the input layer can be the same as the length of the input vector, wherein there can be tokens K = {k1, k2,... k K}, and y u can be equal to k K , where n is the index 1 ≤ k ≤ K. The input can be encoded as a one-hot vector.

[0042] In various embodiments, before emitting the output symbol, the output of the prediction network 140 is merged with the output from the encoder network 130. In a character-based system, the (one or more) output symbols from the RNN-T (after softmax calculation) are a set of characters and <BLANK> symbols. In various embodiments, the history of the prediction network is not used <blank>Symbols. Directly manipulating the prediction network 140 for customization (adding words, phrases, and sentences) is not possible. The outputs from the encoder network 130 and the prediction network 140 are used to update the prediction network 140 or the entire RNN-T 100. However, if the RNN-T 100 is customized using only text data, the output from the encoder network 130 cannot be obtained when audio data is unavailable.

[0043] In various embodiments, the output feature sequence y180 can be an output sequence generated by RNN-T100 based on the input feature sequence x110 by searching on the output probability grid defined by P(y|t,u), where y=(y1,y2,...y U-1 ,y U A sequence Y can be the set of all sequences belonging to a certain output space Y. * The output sequence is of length U, and P(y|t,u) is the posterior probability of y given "t" and "u", where "t" is the time index in the time-ordered sequence of acoustic features represented as vectors, and u is the index on the output sequence of length U. In various embodiments, y = (y1, y2, ..., y...). u-1 It is fed into the prediction network 140.

[0044] In various embodiments, a connective temporal classification (CTC) model is trained using acoustic features x, represented as vectors, as input and phonemes as output to obtain a phoneme acoustic model. The neural network trained using this CTC model can be used to initialize the encoder network 130 of the RNN-T 100. The phoneme acoustic model can be used as the initial acoustic model for the encoder network 130 of the RNN-T 100. Acoustic features can be used as input, and modeling units can be set as outputs to train the initial acoustic model to obtain the target acoustic model.

[0045] In various embodiments, cross-entropy (CE) can be used to train the language model (LM). The CTC acoustic model and the CE language model can be used to initialize the encoder 130 (encoder) and predictor 140 (decoder) of the end-to-end system, respectively. After initialization, the end-to-end system can have a suitable initial state.

[0046] RNN-T training can start with a pre-trained model, or, if no pre-training is needed, training can start with random initialization.

[0047] In various embodiments, the language model may include an LSTM layer and a first input embedding layer. LSTM can be used for both the encoder network 130 and the prediction network 140; however, other types of neural networks (such as Transformer neural networks) may also be used for either or both of the encoder and prediction networks.

[0048] In various embodiments, the outputs generated by encoder 130 and predictor 140 can be combined by combiner 150, wherein combiner 150 can produce a high-level representation. The weighted sum of (embedding vectors). The combiner network 150 can combine the outputs from the encoder network. and the output from the prediction network To output embedded z t,u (Logarithm (logit)). In various embodiments, the combiner 150 is a feedforward network that takes the encoder network output. and predict network output The combination is the sum of two embedded linear transformations:

[0049]

[0050] Among them W enc and W pre It is the weight matrix, b z ψ is the deviation vector, and ψ is a nonlinear function, such as Tanh or ReLU.

[0051] Using linear transformation to z t,u Connect to the output layer:

[0052] h t,u =(W y z t,u +b y )

[0053] Where W is the weight matrix, b y It is the deviation vector.

[0054] In various embodiments, a softmax function 160 is applied to the output of the merger 150, wherein the softmax function 160 normalizes the output of the merger neural network 150 to produce a probability (posterior) distribution 170P(y) on the predicted output class. t+u The activation function for |t,u). P(y) t+u |t,u) defines the posterior grid, where each node represents the posterior distribution. The predicted output class can be a character or sub-character (part of a character) from the text training corpus / dictionary. The softmax function 160 can be the output layer of an RNN-T 100.

[0055] In various embodiments, probability distribution 170 can be used to generate output feature sequence y 180, which can be generated by searching on the output probability grid defined by P(y|t,u). Output symbol sequence y 120 can be used to update predictor 140. When training the model, beam search can be omitted because the pair of symbol sequences and input audio features is given, so the posterior probability grid can be computed. Parameters are updated by minimizing the RNN-T loss. The RNN-T loss is defined as the sum of symbol posterior probabilities over all possible RNN-T alignments:

[0056]

[0057] Each sequence It is one of the RNN-T alignments between x and y, where The elements belong to the symbol set.

[0058] In various embodiments, end-to-end training based solely on the transcript and audio can be achieved, eliminating the need for the iterations and long training strides required in hybrid modeling. Furthermore, in various embodiments, a dictionary linking spelling and pronunciation is not required. The RNN-T can be trained based on an associated pair of audio data and its transcription.

[0059] In one or more embodiments, audio data can be synthesized from text available in the source domain. The text used to synthesize the audio data can be the same text used during the training of the original RNN-T model. When updating the encoder network, the synthesized audio is used as the text used during the training of the original RNN-T model. Therefore, the encoder network can be updated conditionally based on the appropriate output from the prediction network.

[0060] In various embodiments, a pair of synthesized audio and text from the source domain can be used to update the encoder network.

[0061] It can also synthesize audio data from text within the target domain.

[0062] In various embodiments, the prediction network can be updated using a pair of synthesized audio and text from the target domain. When the prediction network is updated, the encoder network has already adapted to the synthesized audio from the source domain. Therefore, the prediction network can be updated conditionally with the appropriate output from the encoder network.

[0063] In various embodiments, the encoder network can be restored to its original conditions. Although the quality of synthesized audio has recently improved, updating the encoder network for speech recognition using synthesized audio is not always valuable. The final encoder network can be identical to the original encoder network, uncontaminated by synthesized audio from a customized target domain. Once the customized features are no longer needed, the encoder can be reset to its initial state.

[0064] Figure 2 This is a block diagram / flowchart illustrating an algorithm for training a recurrent neural network transcript (RNN-T) for speech recognition according to an embodiment of the present invention.

[0065] In one or more embodiments, an algorithm 200 for training a recurrent neural network transcript (RNN-T) may include an input feature sequence x prepared as a vector of acoustic features, wherein the input feature sequence x may be audio data synthesized from text used for initial training of the (original) RNN-T. The encoder network may be initially trained. The audio data may be synthesized from text in a source domain, wherein the source domain may be represented by a standard corpus.

[0066] In box 210, the feature sequence x of the acoustic features can be synthesized by reading the text from the source domain.

[0067] In 215, the feature sequence x of the acoustic features can be fed into the encoder 130 of the RNN-T, where the acoustic features can be a vector of Merr frequency cepstral coefficients (MFCC).

[0068] At box 220, the encoder can be trained using an associated pair of synthesized audio and text from the source domain. The encoder can generate a sequence of hidden vectors (h0, h1, ..., h...). T The input audio features can have a length T. A hidden vector h can be computed for each audio feature. t .

[0069] In 225, the hidden vector sequence is (h0, h1, ..., h...). T It can be fed into the connector 230.

[0070] In box 230, the combiner will hide the vector sequence (h0, h1, ..., h t ) and the hidden vector sequence (h0, h1, ..., h) from predictor 270 u-1 ) combined to produce with h T and h u The index t and u related to the induced local field z t,u .

[0071] At 235, the local field z will be sensed. t,u It is fed into the softmax function 240.

[0072] At 240, the softmax function generates the posterior probability P(y|t,u)250.

[0073] In 245, the posterior probability P(y|t,u) is output by the softmax function.

[0074] At 250, the output from 240 is P(y|t,u).

[0075] At 255, P(y|t,u) is fed to the output sequence generator 260, which produces an output symbol sequence y of length U.

[0076] In box 260, the output sequence generator 260 produces an output symbol sequence y of length U.

[0077] At 265, an output symbol sequence y of length U-1 is fed into predictor neural network 270 to update predictor neural network 270, where y in 120 u-1 Indicates previous predictions (e.g., one or more letters). Based on the previous word sequence up to u-1, RNN-T predicts the next symbol y. u .

[0078] In box 270, the predictor neural network 270 is updated, and a sequence of hidden vectors (h0, h1, ..., h) is generated. U ).

[0079] Figure 3 This is a block diagram / flowchart illustrating an algorithm according to an embodiment of the present invention for synthesizing audio features from text, updating the encoder and predictor, and recovering the encoder.

[0080] In box 310, identify the source domain text to be used for training.

[0081] In box 320, audio data for source domain training of the encoder is synthesized from text from the source domain.

[0082] In box 330, update the encoder network while keeping the weights of the predictor neural network and the combiner neural network constant (i.e., fixed).

[0083] In box 340, identify the target domain text to be used for training.

[0084] In box 350, audio for target domain training of the predictor is synthesized from text in the target domain. The synthesized audio for the target domain is fed into the encoder. In various embodiments, only the prediction network is updated. The encoder network is not used for updating the synthesized audio in the target domain.

[0085] In box 360, update the predictor network while keeping the weights of the encoder neural network and the combiner neural network constant.

[0086] In box 370, during the actual deployment of the custom model, the encoder network is restored to its weights before being fed target domain audio. Restoring the weights on the encoder network to the state trained on the source domain resets the encoder to its pre-defined state.

[0087] Figure 4 This is a diagram of a posterior mesh for RNN-T according to an embodiment of the present invention.

[0088] y = (y1, y2, ..., y U ) can represent a target output symbol sequence with a vertical translation length of U.

[0089] x = (x1, x2, ..., x T ) can represent the acoustic feature vector at T time steps.

[0090] Each node 400 represents P(y) t+u The posterior distribution P(y|t,u) is defined by |t,u).

[0091] Figure 5 This is an exemplary processing system 500 according to an embodiment of the present invention, to which the method and system can be applied.

[0092] In various embodiments, the processing system 500 may include at least one processor (CPU) 504 and may have a graphics processing unit (GPU) 505 capable of performing vector computations / manipulations operatively coupled to other components via a system bus 502. A cache 506, a read-only memory (ROM) 508, a random access memory (RAM) 510, an input / output (I / O) adapter 520, a voice adapter 530, a network adapter 540, a user interface adapter 550, and a display adapter 560 may be operatively coupled to the system bus 502.

[0093] First storage device 522 and second storage device 524 are operatively coupled to system bus 502 via I / O adapter 520. Storage devices 522 and 524 can be any of disk storage devices (e.g., magnetic disk or optical disk storage devices), solid-state devices, magnetic storage devices, etc. Storage devices 522 and 524 can be the same type of storage devices or different types of storage devices.

[0094] Speaker 532 is operatively coupled to system bus 502 via sound adapter 530. Transceiver 542 is operatively coupled to system bus 502 via network adapter 540. Display device 562 is operatively coupled to system bus 502 via display adapter 560.

[0095] First user input device 552, second user input device 554, and third user input device 556 are operatively coupled to system bus 502 via user interface adapter 550. User input devices 552, 554, and 556 can be any device such as a keyboard, mouse, keypad, image capture device, motion sensing device, microphone, or a device combining the functions of at least two of the aforementioned devices. Of course, other types of input devices can also be used, while maintaining the spirit of this principle. User input devices 552, 554, and 556 can be user input devices of the same type or different types. User input devices 552, 554, and 556 can be used to input information to and output information to system 500.

[0096] In various embodiments, the processing system 500 may also include other elements (not shown) readily apparent to those skilled in the art, as well as certain elements omitted. For example, as readily understood by those skilled in the art, various other input and / or output devices may be included in the processing system 500 depending on the specific implementation. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, as readily understood by those skilled in the art, additional processors, controllers, memories, etc., may be utilized in various configurations. Given the teachings of the principles provided herein, these and other variations of the processing system 500 will readily occur to those skilled in the art.

[0097] Furthermore, it should be understood that system 500 is a computer system for implementing corresponding embodiments of this method / system. Part or all of processing system 500 can be performed within... Figures 1-4 Implemented in one or more components, and further, it should be understood that the processing system 500 can perform at least a portion of the methods described herein, including, for example... Figures 1-4 At least a part of the method.

[0098] Figure 6 An exemplary processing system 600, according to an embodiment of the present invention, is configured to implement one or more neural networks for modeling road layouts.

[0099] In one or more embodiments, the processing system 600 may be a computer system 500 configured to perform a computer-implemented method for a custom recurrent neural network transcriptome for speech recognition.

[0100] In one or more embodiments, the processing system 600 may be a computer system 500 having a memory component 670, including but not limited to the computer system's random access memory (RAM) 510, hard disk drive 522, and / or cloud storage device, to store and implement computer-implemented methods for understanding road layouts from video images. The memory component 670 may also utilize a database to organize the memory storage.

[0101] In various embodiments, memory component 670 may include encoder neural network 610, which may be configured to implement multiple acoustic models configured to model acoustic inputs and perform automatic speech recognition (ASR). In various embodiments, encoder neural network 610 may be implemented as a long short-term memory (LSTM) or bidirectional LSTM (BLSTM). Encoder neural network 610 may also be configured to receive acoustic signals as input. The input may be a sequential set of audio data received by microphone 556. Encoder neural network 610 may also be configured to generate output values ​​as embeddings.

[0102] In various embodiments, memory component 670 may include predictor neural network 620, which may be configured to learn one or more acoustic models and to generate encoder embeddings to perform automatic speech recognition (ASR). In various embodiments, predictor neural network 620 may be implemented as a long short-term memory (LSTM). Predictor neural network 620 may also be configured to generate output values ​​as embeddings.

[0103] In various embodiments, memory component 670 may include a combiner neural network 630 that can be configured to combine two separate sets of input data from an encoder and a predictor, wherein the data may be feature / vector h. t and h u The combiner neural network 630 can be configured to produce an output, which can be a letter, a subword, or a word.

[0104] In various embodiments, memory component 670 may include output generator 640 configured to generate an output symbol sequence y 120 having a length of u-1. Output generator 640 may be configured to receive output from combiner neural network 630.

[0105] In various embodiments, memory component 670 may include a Softmax function 650 configured to generate predictions from the output values ​​of combiner neural network 630.

[0106] In various embodiments, memory component 670 may include synthesizer 660 configured to synthesize first-domain audio data from first-domain text data and / or synthesize second-domain audio data from second-domain text data.

[0107] Figure 7 This is an exemplary block diagram depicting an exemplary neural network according to another embodiment of the present invention.

[0108] The neural network 700 may include multiple neurons / nodes, and output nodes may communicate using one or more connections among multiple connections 708. The neural network 700 may include multiple layers, including, for example, one or more input layers 702, one or more hidden layers 704, and one or more output layers 706. In one embodiment, nodes in each layer may be used to apply any function (e.g., input program, input data, etc.) to any previous layer to produce an output, and hidden layers 704 may be used to transform inputs from input layers (or any other layers) into outputs for nodes in different layers.

[0109] Figure 8 This is a block diagram illustrating an illustrative cloud computing environment with one or more cloud computing nodes according to an embodiment, wherein a local computing device used by a cloud consumer communicates with the cloud computing nodes.

[0110] It should be understood that although this disclosure includes a detailed description of cloud computing, the implementation of the teachings set forth herein is not limited to a cloud computing environment. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0111] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing power, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management costs or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0112] The characteristics are as follows:

[0113] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power (such as server time and network storage) on demand without human interaction with the service provider.

[0114] Wide network access: Capabilities are available on the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0115] Resource pooling: A provider's computing resources are grouped into resource pools to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. Typically, consumers cannot control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center), thus exhibiting location independence.

[0116] Rapid flexibility: Capabilities can be rapidly and flexibly (in some cases automatically) provided to expand outward quickly and be rapidly released to shrink back down. For consumers, the available capacity often appears unlimited and can be purchased at any time and in any quantity.

[0117] Measurable services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0118] The service model is as follows:

[0119] Software as a Service (SaaS): The capability offered to consumers is the ability to use applications running on a provider's cloud infrastructure. These applications can be accessed from various client devices via thin client interfaces such as web browsers (e.g., web-based email). Aside from limited user-specific application configuration settings, consumers neither manage nor control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities.

[0120] Platform as a Service (PaaS): This provides consumers with the ability to deploy consumer-created or acquired applications on cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers neither manage nor control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the applications they deploy and may also have control over the configuration of the application hosting environment.

[0121] Infrastructure as a Service (IaaS): This provides consumers with the capability to deploy and run any software, including operating systems and applications, on the cloud, providing them with processing, storage, networking, and other basic computing resources. Consumers neither manage nor control the underlying cloud infrastructure, but they have control over the operating system, storage, and deployed applications, and may have limited control over chosen network components (e.g., host firewalls).

[0122] The deployment model is as follows:

[0123] Private cloud: A cloud infrastructure that operates exclusively for a single organization. It can be managed by that organization or a third party, and can exist inside or outside the organization.

[0124] Community cloud: A cloud infrastructure shared by several organizations that supports a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist inside or outside the organization.

[0125] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud services.

[0126] Hybrid cloud: A cloud infrastructure consisting of two or more clouds (private, community, or public) that remain distinct entities but are bound together by standardized or proprietary technologies that enable data and applications to be ported together (e.g., cloud bursts for load balancing between clouds).

[0127] Cloud computing environments are service-oriented, characterized by statelessness, loose coupling, modularity, and semantic interoperability. The core of computing is the infrastructure comprising a network of interconnected nodes.

[0128] refer to Figure 8 The diagram illustrates an illustrative cloud computing environment 950. As shown, the cloud computing environment 950 includes one or more cloud computing nodes 910 with which local computing devices used by cloud consumers can communicate. Examples of these local computing devices include, but are not limited to, personal digital assistants (PDAs) or cellular phones 951, desktop computers 952, laptop computers 953, and / or automotive computer systems 954. Nodes 910 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 950 to provide Infrastructure as a Service, Platform as a Service, and / or Software as a Service without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that... Figure 8 The types of computing devices 951, 952, 953, and 954 shown are merely illustrative, and computing node 910 and cloud computing environment 950 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0129] Figure 9 This is a block diagram illustrating a set of functional abstraction layers provided by a cloud computing environment according to an embodiment.

[0130] refer to Figure 9 This demonstrates the 950 (cloud computing environment) Figure 7 This provides a set of functional abstractions. It should be understood beforehand that... Figure 8 The components, layers, and functions shown are merely illustrative, and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0131] The hardware and software layer 1060 includes hardware and software components. Examples of hardware components include: a mainframe 61; a RISC (Reduced Instruction Set Computer) based server 62; a server 63; a blade server 64; a storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0132] The virtualization layer 1070 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0133] In one example, management layer 1080 can provide the example functionalities described below. Resource Provisioning 81 provides dynamic acquisition of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 82 provides cost tracking for the use of resources in the cloud computing environment and provides bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security Functions provide authentication for cloud consumers and tasks and protection for data and other resources. User Portal 83 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides cloud resource allocation and management to meet required service levels. Service Level Agreement (SLA) Planning and Fulfillment 85 provides pre-scheduling and procurement of cloud resources according to the SLA for its projected future needs.

[0134] Workload layer 1090 provides examples of functionalities that can leverage a cloud computing environment. Examples of workloads and functionalities that can be provided in this layer include, but are not limited to: map creation and navigation 91; software development and lifecycle management 92; instruction delivery in virtual classrooms 93; data analysis and processing 94; and a recurrent neural network transcriber (RNN-T) 96 that implements automatic speech recognition (ASR), wherein the teacher neural network can be a recurrent neural network configured to learn automatic speech recognition and prepare the student neural network.

[0135] As used herein, the terms "hardware processor subsystem" or "hardware processor" can refer to a processor, memory, software, or a combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuitry, processing circuitry, instruction execution device, etc.). One or more data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or computing element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more on-board memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories that may be on-board or off-board, or may be dedicated to use by the hardware processor subsystem (e.g., ROM, RAM, basic input / output system (BIOS), etc.).

[0136] In some embodiments, the hardware processor subsystem may include and execute one or more software elements. The one or more software elements may include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0137] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application-specific integrated circuits (ASICs), FPGAs, and / or PLAs.

[0138] According to embodiments of the present invention, these and other variations of the hardware processor subsystem are also contemplated.

[0139] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0140] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0141] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0142] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including local area network (LAN) or wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing the status information of the computer-readable program instructions.

[0143] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0144] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0145] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0146] In the specification, references to "one embodiment" or "embodiment" and other variations of the invention mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment of the invention. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing in various places throughout the specification, as well as any other variations, do not necessarily refer to the same embodiment.

[0147] It should be understood that, for example, in the cases of "A / B", "A and / or B", and "at least one of A and B", the use of any of the following " / ", "and / or", and "at least one" is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). This can be extended to many of the listed items, as will be apparent to those skilled in the art and related fields.

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the blocks may occur in a different order than indicated in the figures. For example, two blocks shown consecutively may actually be implemented as a single step, executed simultaneously, substantially simultaneously, with partial or complete time overlap, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0149] Preferred embodiments of the systems and methods have been described (these are intended to be illustrative and not limiting). It should be noted that modifications and variations can be made by those skilled in the art based on the foregoing teachings. Therefore, it should be understood that changes can be made to the specific embodiments disclosed, and these changes are within the scope of the invention as summarized by the appended claims. Thus, aspects of the invention have been described in the details and features required by patent law, and the claimed and patentable aspects are set forth in the appended claims.< / blank>

Claims

1. A method for computer-based implementation of a custom recurrent neural network transcriptome (RNN-T), comprising: Synthesize first-domain audio data from first-domain text data derived from the first-domain data; The synthesized first-domain audio data is fed into the trained encoder of the recurrent neural network transcriber (RNN-T) with initial conditions, and the encoder is updated using both the synthesized first-domain audio data and the first-domain text data. Synthesize second-domain audio data from second-domain text data derived from the second-domain data; The synthesized second-domain audio data is fed into the updated encoder of the recurrent neural network transcriber (RNN-T), and the prediction network of the recurrent neural network transcriber (RNN-T) is updated using both the synthesized second-domain audio data and the second-domain text data. The encoder output is combined with the prediction network output using the combiner of the recurrent neural network transcriber (RNN-T); as well as The updated encoder is reset to a predetermined state by restoring the weights on the updated encoder to the initial conditions, wherein the initial conditions include the state trained on the first domain.

2. The method according to claim 1, wherein, The combiner generates an induced local field z that is fed into the softmax function. t,u As output.

3. The method according to claim 2, wherein, The softmax function generates the posterior probability P(y|t,u).

4. The method according to claim 3, wherein, The output sequence generator of the recurrent neural network transcriptome (RNN-T) is used to generate an output based on the input feature sequence x, wherein the output is an output sequence y = (y1, y2, ... y...). U-1 ,y U The output sequence is an output sequence of length U, and the input feature sequence x is a time-ordered sequence of acoustic features represented as vectors.

5. The method according to claim 4, wherein, The input feature sequence x is derived from the synthesized first-domain audio data.

6. A system for customizing recurrent neural network transcriptomers (RNN-T), comprising: One or more processor devices; A memory that communicates with at least one of the one or more processor devices; as well as Display screen; The memory includes: An encoder is configured to receive first-domain audio data synthesized from first-domain text data from a first domain, wherein the encoder is a trained encoder of the recurrent neural network transcriber (RNN-T) with initial conditions, and the encoder is further configured to update from the initial conditions using both the synthesized first-domain audio data and the first-domain text data, and to receive second-domain audio data synthesized from second-domain text data from a second domain. The predictor is configured to be updated using both the synthesized second-domain audio data and the second-domain text data; A combiner is configured to combine the output of the encoder with the output of the predictor; and An output sequence generator is configured to generate an output sequence y based on an input feature sequence x, wherein the input feature sequence x is a time-ordered sequence of acoustic features represented as vectors; Specifically, by restoring the weights on the updated encoder to the initial conditions, the updated encoder is reset to a predetermined state, wherein the initial conditions include the state trained on the first domain.

7. The system according to claim 6, wherein, The connector generates a local induced field z. t,u As output.

8. The system according to claim 7, wherein, The memory also includes a softmax function, which is configured to receive the sensed local field z. t,u And generate output.

9. The system according to claim 8, wherein, The output sequence y = (y1, y2, ... y U-1 , y U The output sequence is based on the input feature sequence x and has a length of U.

10. The system according to claim 9, wherein, The memory also includes a synthesizer configured to synthesize first-domain audio data from first-domain text data and second-domain audio data from second-domain text data.

11. A computer program product for customizing a recurrent neural network transcriber (RNN-T), the computer program product comprising program instructions executable by a computer to cause the computer to: Synthesize first-domain audio data from first-domain text data derived from the first-domain data; The synthesized first-domain audio data is fed into the trained encoder of the recurrent neural network transcriber (RNN-T) with initial conditions, and the encoder is updated using both the synthesized first-domain audio data and the first-domain text data. Synthesize second-domain audio data from second-domain text data derived from the second-domain data; The synthesized second-domain audio data is fed into the updated encoder of the recurrent neural network transcriber (RNN-T), and both the synthesized second-domain audio data and the second-domain text data are used to update the predictor of the recurrent neural network transcriber (RNN-T). The encoder's output is combined with the predictor's output using the combiner of the recurrent neural network transcriber (RNN-T); and The updated encoder is reset to a predetermined state by restoring the weights on the updated encoder to the initial conditions, wherein, The initial conditions include the state of training on the first domain.

12. The computer program product according to claim 11, wherein, The combiner generates an induced local field z that is fed into the softmax function. t,u As output.

13. The computer program product according to claim 12, wherein, The softmax function generates the posterior probability P(y|t,u).

14. The computer program product according to claim 13, wherein, The output sequence generator of the recurrent neural network transcriptome (RNN-T) is used to generate an output based on the input feature sequence x, wherein the output is an output sequence y = (y1, y2, ... y...). U-1 , y U The output sequence is an output sequence of length U, and the input feature sequence x is a time-ordered sequence of acoustic features represented as vectors.

15. The computer program product according to claim 14, wherein, The first domain audio data is the input feature sequence x.

16. A method for a computer-based implementation of a custom recurrent neural network transcriptome (RNN-T), comprising: Synthesize first-domain audio data from first-domain text data derived from the first-domain data; The synthesized first-domain audio data is fed into the trained encoder of the recurrent neural network transcriber (RNN-T) with initial conditions, and the encoder is updated using both the synthesized first-domain audio data and the first-domain text data, wherein the encoder encodes the synthesized first-domain audio data into an acoustic embedding a. t , wherein the acoustic embedding a t The synthesized first-domain audio data is compressed into a smaller feature space; The acoustic embedding a t The binder is fed into the recurrent neural network transcriber (RNN-T); Synthesize second-domain audio data from second-domain text data derived from the second-domain data; The synthesized second-domain audio data is fed into an updated encoder, wherein the updated encoder encodes the synthesized second-domain audio data into an acoustic embedding b. t , wherein the acoustic embedding b t The synthesized second-domain audio data is compressed into a smaller feature space; The output sequence from the combiner is fed into the predictor of the recurrent neural network transcriptome (RNN-T), and the predictor is updated using both the synthesized second-domain audio data and the second-domain text data; and The updated encoder is reset to a predetermined state by restoring the weights on the updated encoder to the initial conditions, wherein the initial conditions include the state trained on the first domain.

17. The method according to claim 16, wherein, The combiner embeds the acoustic material into a through a weighted summation. t Combined with the embedding from the predictor.

18. The method according to claim 17, wherein, The combiner generates an induced local field z that is fed into the softmax function. t,u As output.

19. The method according to claim 18, wherein, The softmax function generates the posterior probability P(y|t,u).

Images