Training a model to process sequence data

Abstract

A technique for training a model is disclosed. Training samples are obtained that include an input sequence of observations and a target sequence of symbols having a different length than the input sequence of observations. The input sequence of observations is fed to the model to obtain a sequence of predictions. The sequence of predictions is shifted by an amount relative to the input sequence of observations. The model is updated based on a loss using the shifted sequence of predictions and the target sequence of symbols.

Description

[Technical field]

[0001] The present disclosure relates generally to machine learning, and more particularly, to techniques for training models used to process sequence data. [Background technology]

[0002] End-to-end automatic speech recognition (ASR) systems using CTC (Connectionist Temporal Classification) loss function have attracted attention due to the ease of training and decoding efficiency of the system. End-to-end ASR systems use CTC models to predict sub-word or word sequences with or without a subsequent language model. CTC-based ASR can run faster compared to related NN (Neural Network) / HMM (Hidden Markov Model) hybrid systems. Hence, it is expected to significantly reduce power consumption and computational resource costs.

[0003] The combination of unidirectional LSTM model and CTC loss function is one of the promising ways to build streaming ASR. However, such combination usually incurs a time delay between acoustic features and output symbols during decoding, which increases the latency of streaming ASR. The related NN / HMM hybrid system trained from a frame-level forced alignment between acoustic features and output symbols does not incur a time delay. In contrast to the hybrid model, the CTC model is usually trained on training samples with different lengths of acoustic features and output symbols. This means that there is no time alignment supervision. The CTC model trained without frame-level alignment results in output symbols after the model has consumed enough information about the output symbols, which results in a time delay between acoustic features and output symbols.

[0004] A method is proposed to apply constraints on the CTC alignment to reduce the time delay between acoustic features and output symbols (Andrew Senior, et al., "Acoustic modelling with CD-CTC-sMBR LSTM RNNs," in Proc. ASRU, 2015, pp. 604-609.). It is investigated that the delay can be limited by restricting the set of search paths used in the forward-backward algorithm to those where the delay between the CTC labels and the "ground truth" alignment does not exceed some threshold. However, the method disclosed in this paper requires an iterative step to prepare a frame-level constrained alignment prior to CTC model training.

[0005] US Patent Application No. 20170148431A1 discloses an end-to-end deep learning system and method for recognizing speech in widely different languages, such as English or Mandarin Chinese. An entire pipeline of hand-crafted components is replaced with a neural network, allowing end-to-end learning to handle a wide variety of speech, including noisy environments, accents, and different languages. However, the technology disclosed in this patent document attempts to modify the neural network topology.

[0006] US Patent Application Publication No. 20180130474A1 discloses a method, system, and apparatus, including a computer program, for learning pronunciation from an acoustic sequence encoded on a computer storage medium. The method includes stacking one or more frames of acoustic data to generate a sequence of transformed frames of acoustic data, and processing the sequence of transformed frames of acoustic data through an acoustic modeling neural network that includes one or more recurrent neural network (RNN) layers and a final CTC output layer to generate a neural network output. The technique disclosed in this patent document merely adjusts the input to the encoder to reduce the frame rate.

[0007] Therefore, there is a need for novel training techniques that can efficiently reduce the time delay between the input and output of a model trained on training samples in which the input observations and output symbols have different lengths. Summary of the Invention

[0008] According to an embodiment of the present invention, a computer-implemented method for training a model is provided. The method includes obtaining training samples including an input sequence of observations and a target sequence of symbols having a different length than the input sequence of observations. The method also includes feeding the input sequence of observations to the model to obtain a sequence of predictions. The method further includes shifting the sequence of predictions by an amount relative to the input sequence of observations. The method further includes updating a loss-based model using the shifted sequence of predictions and the target sequence of symbols.

[0009] Methods according to embodiments of the present invention enable a trained model to output predictions at the right time relative to the input reducing the latency of the prediction process.

[0010] In a preferred embodiment, a sequence of predictions can be shifted forward relative to an input sequence of observations to generate a sequence of shifted predictions, and the model is unidirectional. The method enables the trained model to output predictions earlier to reduce the latency of the prediction process relative to the input. Models trained by the method are suitable for streaming applications.

[0011] In certain embodiments, the model can be a recurrent neural network based model. In certain embodiments, the loss can be a connectionist time series classification (CTC) loss.

[0012] In certain embodiments, shifting the sequence of predictions includes adjusting the length of the shifted sequence of predictions and the length of the input sequence of observations to be the same.

[0013] In a preferred embodiment, shifting the sequence of predictions and updating the model using the shifted sequence of predictions may be performed at a predetermined rate, allowing the method to balance the accuracy and latency of the trained model prediction process.

[0014] In certain embodiments, the model can be a neural network-based model having a plurality of parameters, and feeding the input sequence includes performing forward propagation through the neural network-based model, and updating the model includes performing back propagation through the neural network-based model to update the plurality of parameters.

[0015] In a further preferred embodiment, the model can be an end-to-end speech recognition model. Each observation in the input sequence of training samples can represent an acoustic feature, and each symbol in the target sequence of training samples can represent a phone, a context-dependent phone, a character, a word-piece, or a word. This allows the method to enable the speech recognition model to output the recognized results in a timely manner that reduces the overall latency of the speech recognition process, or provides more time for subsequent processes to improve the recognition accuracy.

[0016] Also described and claimed herein are computer systems and computer program products relating to one or more aspects of the invention.

[0017] According to another embodiment of the present invention, a computer program product for decoding using a model is provided. The computer program product includes a computer-readable storage medium having program instructions embodied therein. The program instructions are executable by a computer to cause the computer to execute a method including feeding an input to the model to obtain an output. The model is trained by obtaining training samples including an input sequence of observations and a target sequence of symbols having a different length than the input sequence of observations. The model can be further trained by feeding the input sequence of observations to the model to obtain a sequence of predictions, shifting the sequence of predictions by an amount relative to the input sequence of observations, and updating the model based on a loss using the shifted sequence of predictions and the target sequence of symbols.

[0018] The computer program product according to an embodiment of the present invention may output a prediction in a timely manner that reduces the latency of the prediction process relative to the input.

[0019] Additional features and advantages are realized through the techniques of the present invention.Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention.

[0020] The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of this specification. These and other features and advantages of the invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a block diagram illustrating a speech recognition system including a shift-forward CTC (Connectionist Time Series Classification) training system for training a CTC model used for speech recognition according to an exemplary embodiment of the present invention. [Diagram 2] FIG. 1 is a schematic diagram illustrating a unidirectional LSTM CTC model as an example of a CTC model to be trained according to an embodiment of the present invention. [Diagram 3] FIG. 1 illustrates a speech signal for an example sentence, "this is true," and the resulting phone probabilities calculated for the example speech signal from bidirectional and unidirectional LSTM phone CTC models trained by a standard training process. [Figure 4] FIG. 1 illustrates a method for forward-shifted CTC training with one frame-shift according to an exemplary embodiment of the present invention. [Diagram 5] 4 is a flowchart illustrating a novel forward-shifting CTC training process for training a CTC model used for speech recognition according to an exemplary embodiment of the present invention. [Figure 6] FIG. 13 shows the posterior probabilities of monophone CTC models trained by forward-shifting CTC training, where the maximum number of frames to shift was set to 1 and the rate of samples to shift was in the range of 0.1 to 0.3. [Figure 7]FIG. 13 shows the posterior probabilities of monophone CTC models trained by forward-shifting CTC training, where the rate of samples to shift was set to 0.1 and the maximum number of frames to shift was in the range from 1 to 3. [Figure 8] FIG. 13 shows the posterior probability of a word CTC model trained by forward shifting CTC training, with the maximum number of frames to shift set to 1 and the rate of samples to shift set to 0.1. [Figure 9] Figure 1 shows the time delays due to the phone CTC model and the word CTC model for the hybrid model. [Figure 10] 1 is a schematic diagram illustrating a computer system in accordance with one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Although the present invention will be described below in conjunction with specific embodiments, those skilled in the art will understand that the embodiments described below are provided by way of example only and are not intended to limit the scope of the present invention.

[0023] One or more embodiments according to the invention are directed to computer-implemented methods, computer systems, and computer program products for training a model used to process sequence data, in which a sequence of predictions obtained from the model to be trained is shifted by an amount with respect to an input sequence of observations, and the shifted sequence of predictions is used to update the model based on a calculated loss.

[0024] In the following, first with reference to Fig. 1 to Fig. 4, a computer system for training a model according to an exemplary embodiment of the invention is described, where the model to be trained is a CTC (Connectionist Time Series Classification) model for speech recognition and the sequence data to be processed are sequences of acoustic features. Then, with reference to Fig. 5, a computer-implemented method for training a model according to an exemplary embodiment of the invention is described, where the model to be trained by the method is a CTC model for speech recognition and the sequence data to be processed are sequences of acoustic features. Then, an experimental study on a novel CTC training for speech recognition according to an exemplary embodiment of the invention is described with reference to Fig. 6 to Fig. 9. Finally, with reference to Fig. 10, a hardware configuration of a computer system according to one or more embodiments of the invention is described.

[0025] Below, with reference to FIG. 1, a block diagram of a speech recognition process system 100 including a shift-forward CTC training system 110 according to an exemplary embodiment of the present invention will be described.

[0026] As shown in FIG. 1, a speech recognition system 100 may include a feature extraction module 104 for extracting acoustic features from an input, and a speech recognition module 106 for performing speech recognition on the input.

[0027] The speech recognition system 100 according to an exemplary embodiment of the present invention further includes a forward-shifting CTC training system 110 for performing new CTC training to obtain a trained CTC model 170 that constitutes the speech recognition module 106, and a training data store 120 for storing a collection of training data used in the new CTC training performed by the forward-shifting CTC training system 110.

[0028] The feature extraction module 104 may receive as input audio signal data 102 digitized by sampling the audio signal at a predefined sampling frequency and a predefined bit depth. The audio signal may be input from a microphone, for example. The feature extraction module 104 may also receive the audio signal data 102 from a remote client device via a network, such as the Internet. The feature extraction module 104 is configured to extract acoustic features from the received audio signal data 102 by any known acoustic feature analysis to generate a sequence of extracted acoustic features.

[0029] The acoustic features may include, but are not limited to, MFCC (Mel Frequency Cepstral Coefficients), LPC (Linear Predictive Coding) coefficients, PLP (Perceptual Linear Prediction) Cepstral Coefficients, log Mel Spectrum, or any combination thereof. The acoustic features may further include "dynamic" acoustic features, such as, for example, delta features and double delta features of the aforementioned acoustic features that are static.

[0030] It should be noted that while the elements of the acoustic feature sequence are referred to as "frames," the audio signal data 102 includes a series of sampled values ​​of the audio signal at a given frequency. Typically, the audio signal data 102 is sampled at 8,000 Hz for narrowband audio and 16,000 Hz for wideband audio. The temporal duration of each frame in the acoustic feature sequence can be, but is not limited to, approximately 10-40 milliseconds.

[0031] The speech recognition module 106 is configured to convert the input sequence of extracted acoustic features into an output sequence of words. The speech recognition module 106 predicts the most likely speech content for the input sequence of extracted acoustic features using the CTC model 170 and outputs the result 108.

[0032] The speech recognition module 106 according to the exemplary embodiment of the present invention uses the CTC model 170 and can be an end-to-end model. In certain embodiments, the speech recognition module 106 can include a subword (e.g., phone, character) unit end-to-end model. In other embodiments, the speech recognition module 106 can include a word unit end-to-end model. Examples of units of the end-to-end model can include phone, character, context-dependent phone such as triphone and quinphone, word part, word, etc. The speech recognition module 106 includes at least the CTC model 170. The CTC model 170 is the target of the novel CTC training performed by the forward shifting CTC training system 110. The CTC model 170 is defined as a model trained by using a CTC loss function, and the architecture of the CTC model 170 is not limited.

[0033] When the speech recognition module 106 is configured as a subword (e.g., phone) end-to-end model, the speech recognition module 106 includes an appropriate language model, such as an n-gram model and a neural network-based model (e.g., a recurrent neural network (RNN)), and a dictionary, in addition to the CTC model 170 that outputs a sequence of subwords. When the speech recognition module 106 is configured as a word-based end-to-end model, the speech recognition module 106 may include only the CTC model 170 that directly outputs a sequence of words, and a language model and a dictionary are not required.

[0034] Also, the speech recognition module 106 can complete the speech recognition using only a neural network, and does not require a complex speech recognition decoder. However, in other embodiments, a language model can be further applied to the result of the word-by-word end-to-end model to improve the accuracy of the speech recognition. Also, in the described embodiment, the speech recognition module 106 also receives an input sequence of acoustic features. However, in another embodiment, the raw waveform of the audio signal data 102 can also be received by the speech recognition module 106. Therefore, the raw audio signal data 102 can be treated as a kind of acoustic features.

[0035] The speech recognition module 106 finds the word sequence that has the greatest likelihood based on the input sequence of acoustic features and outputs the word sequence as a result 108 .

[0036] The shift forward CTC training system 110 shown in FIG. 1 is configured to perform novel CTC training to obtain a CTC model 170 that at least in part constitutes the speech recognition module 106 .

[0037] In the described embodiment, training data store 120 stores a collection of training data, each of which includes speech data and a corresponding transcription.

[0038] It should be noted that the speech data stored in the training data store 120 can be provided in the form of a sequence of acoustic features after feature extraction, which can be identical to that performed by the feature extraction module 104 in the front-end process for estimation. If the speech data is provided in the form of audio signal data that is identical to the audio signal data 102 for estimation, the speech data can be subjected to feature extraction before training to obtain a sequence of acoustic features. Also, the transcription can be provided in the form of a sequence of phones, context-dependent phones, characters, word parts, or words depending on the units that the CTC model 170 is targeting.

[0039] In the described embodiment, each training sample is given as a pair of an input sequence of observations and a target sequence of symbols, where the observations are acoustic features and the symbols are subwords (e.g., phones) or words. The training data may be stored in an internal storage device operatively coupled to the processing circuit or in an external storage device.

[0040] The forward-shifting CTC training system 110 performs a novel CTC training process to obtain a CTC model 170. During the novel CTC training process, the forward-shifting CTC training system 110 performs predetermined operations on a sequence of predictions obtained from a trained CTC model prior to the CTC calculation and parameter updates of the CTC model.

[0041] Before describing the novel CTC training, an exemplary architecture of the CTC model is first described.

[0042] Referring to FIG. 2, a schematic diagram of the LSTM CTC model is shown as an example of the CTC model. To train the CTC model, pairs of subwords (e.g., monophones) / word sequences and audio signal data are fed without alignment. The LSTM CTC model 200 may include an input portion 202 for receiving an input sequence of acoustic features obtained from the given audio signal data via feature extraction, an LSTM encoder 204, a softmax function 206, and a CTC loss function 208. Frame stacking is also contemplated, in which consecutive frames are stacked together as a super-frame as input for the CTC model.

[0043] The LSTM encoder 204 converts the input sequence of acoustic features into high-level features. The LSTM encoder 204 shown in FIG. 2 is unidirectional. Note that the term "unidirectional" means that the network only gets information from past states and not future states, as opposed to bidirectional models where the network gets information from past and future states simultaneously. Using a unidirectional model is preferable for streaming (and possibly real-time) ASR, since a unidirectional model does not require the entire frame sequence prior to decoding. A unidirectional model can output predictions in sequence as acoustic features arrive.

[0044] The softmax function 206 computes a probability distribution by normalization based on the output high-level features obtained from the LSTM encoder 204. The CTC loss function 208 is a specific type of loss function designed for sequence labeling tasks.

[0045] Note that the related NN / HMM hybrid system training requires frame-level alignment, and the length of the target sequence of phones must be equal to the length of the input sequence of acoustic features. This frame-level alignment can generally be achieved by constrained alignment techniques. However, this frame-level alignment makes the training process complicated and time-consuming.

[0046] In contrast to related NN / HMM systems, the target sequence of subwords or words required to train a CTC model can have a different length from the input sequence of acoustic features. In general, the length of the input sequence of acoustic features is much longer than the target sequence of subwords or words. That is, no frame-level alignment is required at all and there is no time alignment supervision for training a CTC model.

[0047] Due to the aforementioned properties, a CTC model with a unidirectional LSTM encoder trained without frame-level alignment will yield output symbols after the CTC model has consumed enough information about the output symbols, which results in a time delay between the acoustic features and the output symbols (subwords or words), which is not the type of time delay that can be reduced by investing a large amount of resources.

[0048] Figure 3 shows at the top the waveform of a speech signal for the example sentence "this is true," and also shows at the middle and bottom the resulting phone probabilities calculated for the example speech signal from bidirectional and unidirectional LSTM phone CTC models trained by a standard CTC training process.

[0049] The CTC model emits a spiked sparse posterior distribution over the target output symbols (subwords or words) where most frames emit a null symbol with high probability and some frames emit the target output symbol of interest. Note that the symbol with the highest posterior probability at each time index, at least excluding the blank, is referred to herein as a "spike". In Figure 3, the probability of the blank symbol is omitted for convenience. The spike timing emitted by a trained CTC model is generally not controlled.

[0050] As shown at the bottom of Figure 3, the spike timings from the unidirectional LSTM model are delayed from the actual acoustic features and speech signal shown at the top of Figure 3. Spikes corresponding to the detected single phones, “DH”, “IH”, “S”, “IH”, “Z”, “T”, “R”, and “UW”, are output with a time delay relative to the input acoustic features.

[0051] Note that the bidirectional LSTM model outputs posterior probabilities aligned to the acoustic features shown in the middle of Figure 3. This means that the bidirectional LSTM model gives spikes in a more timely manner compared to the unidirectional model, because it summarizes the entire input sequence of acoustic features prior to decoding and leverages information from both past and future states. Therefore, using the unidirectional model is preferable for streaming ASR.

[0052] To reduce the time delay between spike timing and acoustic features, a forward-shifting CTC training system 110 according to an exemplary embodiment of the present invention performs a forward shift on a sequence of predictions (posterior probability distributions) obtained from a CTC model that is trained prior to CTC computation and parameter updates via backpropagation.

[0053] 1 in conjunction with FIG 4, a detailed block diagram of the forward-shifting CTC training system 110 will be further described. FIG 4 illustrates an aspect of forward-shifting CTC training with one frame shift according to an exemplary embodiment of the present invention.

[0054] As shown in FIG. 1, a forward-shifting CTC training system 110 may include an input feed module 112 for feeding an input sequence of acoustic features to a trained CTC model to obtain a sequence of predictions, along with a forward-shifting module 114 for forward shifting the obtained sequence of predictions, and an update module 116 for updating parameters of the trained CTC model in a manner based on the shifted sequence of predictions.

[0055] The input feed module 112 is configured to first obtain training samples including an input sequence of acoustic features and a target sequence of subwords or words as correct labels, and is further configured to feed the input sequence of acoustic features included in each training sample to the trained CTC model to obtain a sequence of predictions.

[0056] Let X denote a sequence of acoustic feature vectors over T time steps, and x t Let be the acoustic feature vector at time index t (t = 1, ..., T) in sequence X. We use the CTC model to obtain the sequence of acoustic feature vectors X = {x1, ..., x T}, we obtain a sequence of predictions O={o1,...,o T} is obtained, where o t (t=1,...,T) represents the prediction for each time index t, and each prediction o t is the posterior probability distribution over the target output symbols (subwords or words).

[0057] A forward shift module 114 is configured to shift the sequence of obtained predictions O with respect to the input sequence of acoustic feature vectors X by a predetermined amount to obtain a shifted sequence of predictions O'. In a preferred embodiment, the sequence of obtained predictions O is shifted forward with respect to the input sequence of acoustic features X.

[0058] The forward shifting module 114 is further configured to adjust the length of the shifted sequence of predictions O′ and the length of the input sequence of acoustic feature vectors X so that they are identical. In a particular embodiment, the adjustment is performed by shifting the end of the sequence of predictions O′, e.g., the last element of the prediction, o T This can be done by padding the sequence of predictions O' with one or more copies of T. Thus, the length of the sequence of predictions O' is kept at T.T The use of a copy of the last element of is one example. In the described embodiment, padding may be used as a method of adjustment. However, the method of adjustment is not limited to padding. In another particular embodiment, the adjustment can be performed by truncating the sequence of acoustic feature vectors X from the end by a predetermined amount to be shifted.

[0059] If the given amount to shift (the number of frames to shift) is one frame, the sequence of shifted predictions O' holds the remaining predictions except for the beginning, and the sequence of shifted predictions O' is divided into a set {o2,...,o T ,o T}. The last element o of the prediction in the shifted sequence of predictions O' is T Note that, ,is doubled. Also note that only the predictions (posterior probability distributions) are shifted, nothing else, including the input acoustic feature vectors, is shifted.

[0060] In a preferred embodiment, shifting the sequence O of predictions is not performed for all training samples, but only for a portion of the training samples, which can be determined by a predetermined rate. The predetermined rate of training samples to be shifted can be in the range of about 5% to 40%, more preferably in the range of about 8% to 35%. Also, the unit amount of training samples to be shifted can be one training sample, or a group of training samples (e.g., a mini-batch).

[0061] Furthermore, the amount to shift (or the number of frames to shift) may be fixed to an appropriate value in a particular embodiment. The fixed value may depend on the goal of reducing the delay time and the time duration of each frame. A reduction in delay commensurate with the time duration of each frame and the number of frames to shift is obtained.

[0062] In another embodiment, the amount to shift can be determined probabilistically within a predetermined range. The term "probabilistically" means relying on a predetermined distribution, such as a uniform distribution. The predetermined range, or the upper limit of the predetermined range, can be determined in a manner that depends on the goal of reducing the delay time and the temporal duration of each frame. A reduction in delay commensurate with the temporal duration of each frame and the average number of frames to shift is obtained, as will be shown experimentally later.

[0063] The update module 116 is configured to update the model based on a loss function using the shifted sequence of predictions O' contained in the training sample and the target sequence of symbols (subwords or words). As shown at the bottom of Figure 4, the parameters of the CTC model are updated by calculating the CTC loss based on the shifted sequence of predictions O' and backpropagating through the CTC model. Also, the parameter update can be performed every time one training sample (online) or a group of training samples (e.g., mini-batch) is processed.

[0064] The CTC computation includes the process of CTC alignment estimation. Let y denote a sequence of target output symbols having length L, and y i Let (i=1,...,L) be the ith subword or word in a target sequence y. In contrast to related alignment-based NN / HMM hybrid system training, which requires L to be equal to T, CTC enables alignment-free training by introducing an extra null symbol φ that expands a length-L sequence y into a set of length-T sequences Φ(y). The set of length-T sequences, y^(y^) is the element of Φ(y) and is denoted by {y1^,y2^,y3^,...,y T-1 ^,y T Each sequence in {\displaystyle \mathbb {X}} is one of the CTC alignments between the sequence of acoustic feature vectors X and the sequence of target output symbols y.

[0065] For example, assume that a given output phone sequence is "ABC" and the length of the input sequence is 4. In this case, the possible phone sequences are {AABC, ABBC, ABCC, ABC_, AB_C, A_BC, _ABC}, where "_" represents the blank symbol (φ).

[0066] The CTC loss is defined as the sum of the symbol posterior probabilities over all possible CTC alignments:

[0067]

number

[0068] CTC training maximizes the sum of all possible output sequences, or The update module 116 minimizes the negative sum while allowing a null output for any frame probability. CTC Update the parameters of the CTC model to minimize L based on the sequence of shifted predictions O'. Note that minimizing the loss (CTC loss) involves maximizing the negative of the loss, which can be called reward, utility, or fitness. The update module 116 updates the parameters of the CTC model to minimize L based on the sequence of shifted predictions O'. CTC and to update the parameters of the CTC model every time a training sample (online) or a group of training samples (e.g., a mini-batch) is processed. CTC Backpropagation can be performed through the network based on

[0069] The background idea for forward shifted CTC training is as follows: CTC alignment (y1^,y2^,y3^,...,y T-1 ^,y T If the forward shift of the sequence of predictions is such that the forward shifted CTC alignment (y2^, y3^, ..., y T-1 ^,y T ^,yT Due to the high probability of forward-shifted CTC alignment, the entire network of CTC models is trained to promote forward-shifted CTC alignment via backpropagation, which results in a reduced time delay after the entire training.

[0070] In the described embodiment, the CTC model is described as being a unidirectional LSTM model. However, the architecture of the CTC that is the target of the novel forward shifted CTC training is not limited and can be any RNN type model, including basic RNN, LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), Elman network, Jordan network, Hopfield network, etc. The RNN type model can also include more complex architectures such as any one of the aforementioned RNN type models used in combination with other architectures such as CNN (Convolutional Neural Network), VGG, ResNet, and Transformer.

[0071] Please note that the new forward shifted CTC training is only used in training. The trained CTC model can be used in the same way as the model trained by normal CTC training. The topology (e.g., the manner in which neurons are connected) and configuration (e.g., the number of hidden layers and units) of the CTC model, and the method of decoding using the CTC model, remain unchanged.

[0072] In certain embodiments, each of modules 104, 106 and forward shifted CTC training system 110 illustrated in FIG. 1, as well as each of modules 112, 114, and 116 of forward shifted CTC training system 110, may be implemented as, but is not limited to, software modules including program instructions or data structures, or both, in conjunction with hardware components such as a processor, memory, etc., as hardware modules including electronic circuitry, or as a combination of the above.

[0073] These modules and systems may be implemented on a single computing device, such as a personal computer and server machine, or across multiple devices in a distributed manner, such as a computer cluster of computing devices, client-server systems, cloud computing systems, edge computing systems, and the like.

[0074] The training data store 120 and storage for parameters of the CTC model 170 may be provided by using any internal or external storage device or storage medium operatively coupled to the processing circuitry of the computer system implementing the forward shifted CTC training system 110.

[0075] Also in certain embodiments, the feature extraction module 104 and the speech recognition module 106 including the CTC model 170 trained by the forward-shifted CTC training system 110 are implemented on a user's computer system, while the forward-shifted CTC training system 110 is implemented on a speech recognition system provider's computer system.

[0076] In a further variant embodiment, only the feature extraction module 104 is implemented at the user side and the speech recognition module 106 is implemented at the provider side. In this embodiment, the client side computer system only transmits the sequence of acoustic features to the provider side computer system and receives the decoded result 108 from the provider side. In another variant embodiment, both the feature extraction module 104 and the speech recognition module 106 are implemented at the provider side and the client side computer system only transmits the audio signal data 102 to the provider side computer system and receives the decoded result 108 from the provider side.

[0077] In the following, a novel forward-shifted CTC training process for training a CTC model used for speech recognition according to an exemplary embodiment of the present invention will be described with reference to Fig. 5. Fig. 5 is a flow chart illustrating the novel forward-shifted CTC training process. It should be noted that the process illustrated in Fig. 5 may be performed by a processing circuit such as the forward-shifted CTC training system 110 illustrated in Fig. 1, as well as a processing unit of a computer system implementing modules 112, 114, and 116 of the system 110.

[0078] The process shown in FIG. 5 may begin at step S100, for example, in response to receiving a request from an operator for a new forward shifted CTC training.

[0079] In step S101, the processing unit can set training parameters including the maximum number of frames to be shifted (the amount to be shifted) and the rate of samples to be shifted in the new forward shifted CTC training.

[0080] In step S102, a processing unit may prepare a collection of training samples from the training data store 120. Each training sample may include an input sequence of acoustic feature vectors X having length T and a target sequence of symbols (subwords (e.g., phones) or words) y having length L.

[0081] In step S103, the processing unit can initialize the CTC model. The initial values ​​of the parameters of the CTC model are set appropriately.

[0082] In step S104, the processing unit may select one or more training samples in the prepared collection. A mini-batch of training samples may be selected.

[0083] In step S105, the processing unit can perform forward propagation through the CTC model by feeding the input sequence of acoustic feature vectors X to obtain a sequence O of predictions having length T for each selected training sample.

[0084] In step S106, the processing unit may determine whether to perform forward shifting based on the rate of samples to be shifted given in step S101. Mini-batches may be randomly selected at a predetermined rate as targets for forward shifting.

[0085] In step S107, the processing unit may branch the process in a manner that depends on the decision made in step S106. If in step S107 the processing unit determines that the selected training sample is a target for forward shifting (YES), the process may proceed to S108.

[0086] In step S108, the processing unit can determine the number of frames to shift based on the maximum number of frames to shift given in step S101. The number of frames to shift can be determined probabilistically based on a particular distribution. In a particular embodiment, the number of frames to shift can be determined from an integer uniform distribution up to an upper bound (the maximum number of frames to shift) for the selected mini-batch.

[0087] In step S109, a processing unit may perform a forward shift on the sequence of predictions O obtained in step S105 to generate a sequence of shifted predictions O'.

[0088] On the other hand, in response to determining in step S107 that the selected training sample is not a target for forward shifting (No), the process may proceed directly to S110.

[0089] In step S110, a processing unit can use the sequence of shifted predictions O' or the sequence of original predictions O to compute the CTC loss and back-propagate through the CTC model to update the parameters of the CTC model. The forward shifting can be performed on the selected mini-batch. For the remaining mini-batches, CTC training can be performed as usual.

[0090] In step S111, the processing unit can determine whether the process is to be terminated. If a predetermined convergence or termination condition is met, the processing unit can determine that the process should be terminated.

[0091] In response to determining in step S111 that the process is not to end (NO), the process may loop back to S104 for subsequent training samples. Alternatively, in response to determining in step S111 that the process is to end (YES), the process may proceed to S112. In step S112, the processing unit may store the currently obtained parameters of the CTC model in a suitable storage device, and the process ends in step S113.

[0092] According to the above-described embodiments, a novel training technique is provided that can efficiently reduce the time delay between the input and output of a model trained on training samples in which the input observations and the output symbols have different lengths.

[0093] The novel CTC training enables the trained model to output predictions at the right time to reduce the latency of the prediction process to the input. Preferably, the trained model can output predictions earlier to reduce the latency of the prediction process to the input. The trained model is suitable for streaming ASR applications. As demonstrated in the experimental results described later, latency and speech recognition accuracy can be balanced by adjusting the training parameters of the novel CTC training.

[0094] While the actual overall latency of streaming end-to-end ASR is affected by other factors, reducing the time delay between acoustic features and symbols results in lower latency of streaming ASR, or allows more time for subsequent (post)processing to improve accuracy. Compared to regular CTC training, the novel forward-shifted CTC training does not require any additional information, such as forced frame-level alignment. Moreover, during decoding, the CTC model trained with this forward-shifted training can be used in the same way as the model trained with regular CTC training.

[0095] Furthermore, as demonstrated in the experimental results described below, there is almost no adverse effect on the accuracy of the CTC model with respect to speech recognition.

[0096] In the above-described embodiment, the CTC model trained by the forward-shifted CTC training system 110 is described as being directly used as the CTC model 170 constituting the speech recognition module 106. However, in other embodiments, the CTC model trained by the forward-shifted CTC training system 110 may not be directly used as the CTC model 170. In certain embodiments, the CTC model trained by the forward-shifted CTC training system 110 may be used in a knowledge distillation framework. For example, the unidirectional LSTM trained by the forward-shifted CTC training system 110 may be used as an induced CTC model, and the bidirectional LSTM model trained under the guidance of the induced CTC model may be used as a teacher model for knowledge distillation for the student unidirectional LSTM model used as the CTC model 170. For example, the bidirectional LSTM model trained under the guidance of the induced CTC model may be used as a teacher model for knowledge distillation for the student unidirectional LSTM model, where the student unidirectional LSTM model is trained based on the forward shift.

[0097] It should also be noted that in still other embodiments, the CTC model trained by the forward-shifted CTC training system 110 need not be used solely as the CTC model 170. In certain other embodiments, post-fusion involving the CTC model trained by the forward-shifted CTC training system 110 is also contemplated.

[0098] It should be noted that the languages ​​to which the novel training for speech recognition according to the exemplary embodiment of the present invention may be applicable are not limited, and such languages ​​may include, but are in no way limited to, Arabic, Chinese, English, French, German, Japanese, Korean, Portuguese, Russian, Swedish, and Spanish. Since the novel training has an alignment-free nature, the GMM / HMM system for forced alignment may be omitted. Also, when a word-wise end-to-end model is used, no dictionary or language model is required at all. For this reason, the novel training is suitable for some languages ​​for which it is difficult to prepare a GMM / HMM system or a dictionary, or both.

[0099] Furthermore, in the above-mentioned embodiment, the novel forward-shifted CTC training has been described as being applied to speech recognition. However, the applications to which the CTC model is applicable are not limited to speech recognition. The CTC model may be used in various sequence recognition tasks other than speech recognition. Also, the problem of latency arises not only in speech recognition but also in other sequence recognition tasks. Such sequence recognition tasks may include handwritten text recognition from images or sequences of pen strokes, optical character recognition, gesture recognition, machine translation, etc. Therefore, it is expected that the novel forward-shifted CTC training will be applied to such other sequence recognition tasks.

[0100] Although advantages gained in connection with one or more particular embodiments of the present invention have been described and will be explained below, it should be understood that some embodiments may not have these potential advantages, and that these potential advantages are not necessarily required for all embodiments.

[0101] Experimental research A program implementing the forward-shifted CTC training system 110 shown in FIG. 1 and the forward-shifted CTC training process described in FIG. 5 according to an exemplary embodiment was coded and executed on a given set of speech data. ASR experiments were conducted using a standard English conversational telephone speech data set to verify the performance of the novel forward-shifted CTC training. The novel forward-shifted CTC training was applied to train a unidirectional LSTM phone CTC model and a unidirectional LSTM word CTC model. The time delay was measured from a sufficiently powerful offline hybrid model trained from frame-level forced alignment.

[0102] Experimental setup 262 hours of segmented speech from standard 300 hours Switchboard-1 audio with transcription were used. For acoustic features, 40-dimensional logMel filterbank energies over 25 ms frames were extracted every 10 ms. The static features and the delta and double delta coefficients of the features were used with frame stacking with a decimation rate of 2. For evaluation, the Switchboard (SWB) and CallHome (CH) subsets of the NIST Hub5 2000 evaluation data set were used. Considering the training data consisting of SWB-like data, testing on the CH test set is an unfit scenario for the model.

[0103] For the unidirectional LSTM phone CTC model, 44 phones from the Switchboard pronunciation lexicon and a blank symbol were used. For decoding, a 4-gram language model with 24M words was trained from the Switchboard and Fisher transcriptions with a vocabulary size of 30K. A CTC decoding graph was constructed. For the neural network architecture, six unidirectional LSTM layers with 640 units (unidirectional LSTM encoder) and a 640 × 45 fully connected linear layer were stacked, followed by a softmax activation function. All neural network parameters were initialized to samples of a uniform distribution over (-ε, ε), where ε is the inverse square root of the input vector size.

[0104] For the word CTC model, words with at least five occurrences in the training data were selected. This resulted in an output layer with 10,175 words and a blank symbol. The same six unidirectional LSTM layers were selected, and one fully connected linear layer with 256 units was added to reduce computation. A 256 × 10,176 fully connected linear layer was followed by a softmax activation function. For better convergence, the unidirectional LSTM encoder part was initialized with the trained monophone CTC model. Other parameters were initialized in a similar manner to the monophone CTC model. For decoding, a simple peak selection was performed over the output word posterior distribution, and repeated and blank symbols were removed.

[0105] All models were trained for 20 epochs, with Nesterov accelerated stochastic gradient descent starting at 0.01 and increasing to (0.5) per epoch after epoch 10. 1 / 2 A learning rate of annealing was used with a batch size of 128.

[0106] For the novel forward shifted CTC training, two parameters are shifted including "shift max" which indicates the maximum number of frames to be shifted and "shift rate" which is the rate of mini-batches whose outputs are posterior probabilities. Training mini-batches were randomly selected with "shift rate" and forward shifting was performed on the selected mini-batches. For the shift size, the shift size was selected from an integer uniform distribution ranging from 0 to an upper bound given by the training parameter, "shift max", for each selected mini-batch.

[0107] Post-spike As mentioned before, the CTC model emits a posterior distribution that is highly spiky. The posterior probability for the utterance "this(DH IH S)is(IH Z) true(TR UW)" from the SWB test set was examined.

[0108] Figure 6 shows the posterior probabilities of single-phone CTC models trained with the novel forward-shifted CTC training where the maximum number of frames to shift (shift max) is set to 1 and the rate of samples to shift (shift rate) ranges from 0.1 to 0.3 (Examples 1-3). Compared with the posterior spikes from normal CTC training at the top (Comparative Example 1), the posterior spikes from forward-shifted CTC training (Examples 1-3) appear earlier, which is the expected behavior of the novel forward-shifted CTC training.

[0109] Figure 7 shows the posterior probabilities (Examples 1, 4, and 7) of the single-phone CTC model with forward-shifted CTC training where the rate of samples to shift (shift rate) was set to 0.1 and the maximum number of frames to shift (shift max) ranged from 1 to 3. While some earlier spikes also appeared in this case, some spikes were not forward-shifted, especially in the cases with the larger maximum number of frames to shift (shift max).

[0110] Finally, the posterior probability of the word CTC model with forward-shifted CTC training was investigated. Figure 8 shows the posterior probability of the word CTC model with forward-shifted CTC training where the maximum number of frames to shift (shift max) was set to 1 and the rate of samples to shift (shift rate) was set to 0.1 (Example 10). As shown in Figure 8, it was confirmed that earlier spike timing was obtained from the CTC model trained with forward-shifted training. Note that for all word CTC models with normal CTC training and forward-shifted CTC training, the unidirectional LSTM monophone CTC model trained with normal CTC training was used for initialization.

[0111] Time Delay from Hybrid Model Next, the time delay of each word after decoding for the SWB and CH test sets was investigated. Figure 9 shows the definition of time delay by the phone and word CTC models for the hybrid model. In Figure 9, each box represents a unit of output symbol prediction for each model. Note that the unit size is different between the hybrid and CTC models due to the lower frame rate achieved by input frame stacking and decimation in the CTC model. For the hybrid model, "-b", "-m", and "-e" represent the three states of the HMM, and the context-dependent variant identifiers for each state are omitted from this figure for simplicity.

[0112] To set the ground truth of word timing, a sufficiently powerful offline hybrid model was used, trained on the Switchboard+Fisher dataset for 2000 hours through an iterative and elaborate forced alignment step. More specifically, a combination of two bidirectional LSTMs and one Residual Network (ResNet) acoustic model and an n-gram language model was used for decoding and timestamps were obtained for each word. The WER on the SWB and CH test sets using this hybrid model was 6.7% and 12.1%, respectively, which was much better than that using the following CTC model. This hybrid model was used to obtain proper alignment for reference, not for streaming ASR. Furthermore, it is noted that this hybrid model was trained with much more training data.

[0113] For the output from the monophone CTC model, a timestamp was also obtained after decoding. For both hybrid CTC and monophone CTC decoding, the same graph-based static decoder was used, while the blank symbols were properly handled. For the word CTC model, the first spike of each word occurrence was assumed to be the beginning time of the word occurrence. To measure the delay, the recognition results from the hybrid and CTC models were first aligned with dynamic programming-based string matching, as shown in Figure 9, and the average delay at the beginning of the aligned words was calculated. For the unidirectional LSTM monophone CTC model, the shift maximum value was varied from 1 to 3, and shift rates from 0.1 to 0.3 were investigated (Examples 1-9). The conditions and evaluated results of the example and comparative example for the unidirectional LSTM monophone CTC model are summarized in Table 1.

[0114] [Table 1]

[0115] Although there was some variation in WER and time delay, it was demonstrated that a steady reduction in delay was obtained by using the novel forward-shifted CTC training. This trend was common in both matched SWB and unmatched CH test sets. For example, as written in bold in Table 1, the time delay was reduced by 25 ms without observing a detrimental effect on WER. By setting a larger shift rate, a further reduction in time delay was obtained, at the expense of WER, which may provide streaming application developers with an option to adjust the time delay. As in the previous study of spike timing, no further time delay reduction was observed by setting a larger shift maximum.

[0116] For the unidirectional LSTM word CTC model, the same settings were used as in the previous study of spike timing (Example 10). The conditions and evaluated results of Example 10 and Comparative Example 2 for the unidirectional LSTM word CTC model are summarized in Table 2.

[0117] [Table 2]

[0118] While a slight WER degradation on the SWB test set was observed, it was determined that the time delay was reduced by approximately 25 ms with the novel forward-shifted training.

[0119] It was demonstrated that the novel forward-shifted CTC training can reduce the time delay between acoustic features and output symbols in the unidirectional LSTM phone model and the word CTC model. It was also investigated that the novel forward-shifted CTC training allows the trained models to generate spikes earlier. It was also confirmed that in most cases the time delay can be reduced by about 25 ms without any adverse effect on the WER. It is worth noting that the CTC models trained with the novel forward-shifted training simply generate output symbols earlier and can be used without any changes to existing decoders for models trained with normal CTC training. It is desirable to reduce the latency to below 200 ms, which is known as the acceptable limit in human-machine interaction, and it is noted that various efforts have been made and combined to achieve this. The 25 ms obtained by only a simple change in the CTC training pipeline is preferable and can be combined with other efforts.

[0120] Computer Hardware Components Referring now to FIG. 10, a schematic diagram of an example of a computer system 10 that can be used for the speech recognition system 100 is shown. The computer system 10 shown in FIG. 10 is implemented as a computer system. The computer system 10 is merely one example of a suitable processing device and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the invention described herein. In any event, the computer system 10 can be implemented as and / or perform any of the functions described above.

[0121] Computer system 10 is operable with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system 10 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, in-vehicle devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, distributed cloud computing environments that include any of the above systems or devices, and the like.

[0122] Computer system 10 may be described in the general context of computer system executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types.

[0123] As shown in Figure 10, computer system 10 is shown in the form of a general purpose computing device. Components of computer system 10 may include, but are not limited to, a processor (or processing unit) 12 and memory 16 coupled to processor 12 by a bus, including a memory bus or memory controller, and a processor bus or local bus using any of a variety of bus architectures.

[0124] Computer system 10 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by computer system 10, and includes both volatile and nonvolatile media, removable and non-removable media.

[0125] Memory 16 may include computer system readable media in the form of volatile memory, such as random access memory (RAM). Computer system 10 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 18 may be provided for reading from and writing to non-removable, non-volatile magnetic media. As further shown and described below, storage system 18 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0126] By way of example, and not by way of limitation, a program / utility having a set of program modules (at least one), as well as an operating system, one or more application programs, other program modules, and program data, may be stored in storage system 18. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. The program modules generally perform the functions or methods, or combinations thereof, of the embodiments of the present invention described herein.

[0127] The computer system 10 may also communicate with one or more peripheral devices 24, such as a keyboard, a pointing device, a car navigation system, an audio system, and the like, a display 26, one or more devices that allow a user to interact with the computer system 10, or any device that allows the computer system 10 to communicate with one or more other computing devices (e.g., a network card, a modem, and the like), or a combination thereof. Such communication may occur through an input / output (I / O) interface 22. Additionally, the computer system 10 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, through a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system 10 through a bus. It should be understood that other hardware or software components, or combinations thereof, not shown, may also be used in conjunction with the computer system 10. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems, among others.

[0128] Computer Program Implementation The invention may be a computer system, method, and / or computer program product, which may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to perform aspects of the invention.

[0129] A computer readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves on which instructions are recorded, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as a transitory signal itself, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electrical signal transmitted over wires.

[0130] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, transmission optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the respective computing / processing device.

[0131] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk®, C++, or the like, and traditional procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the scenario where the computer is executed entirely on a remote computer or server, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) is capable of executing computer readable program instructions by utilizing state information of the computer readable program instructions to customize the electronic circuitry to perform aspects of the invention.

[0132] Aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer readable program instructions.

[0133] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams, and can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.

[0134] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0135] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may occur out of the order described in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that executes the specified functions or operations, or executes a combination of dedicated hardware instructions and computer instructions.

[0136] The terms used herein are intended to describe particular embodiments and are not intended to limit the invention. As used herein, the singular forms "a" and "the" are intended to include the plural, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising", when used herein, specify the presence of the stated features, steps, layers, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, steps, layers, elements, components, or groups thereof, or combinations thereof.

[0137] Corresponding structure, materials, acts, and equivalents of all means or step and functional elements in the appended claims are intended to include, when present, any structure, material, or act for performing that function in combination with claimed elements other than those expressly claimed. The description of one or more aspects of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed.

[0138] Many variations and modifications will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications, or technical improvements over techniques found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for training a unidirectional model, comprising: obtaining training samples including an input sequence of observations and a target sequence of symbols having a different length than the input sequence of observations; performing forward propagation by feeding the input sequence of observations into the unidirectional model to obtain a sequence of predictions, which are posterior probability distributions; shifting the obtained sequence of predictions forward by an amount relative to the input sequence of observations to obtain a sequence of forward shifted predictions; performing backpropagation based on losses using the forward shifted sequence of predictions and the target sequence of symbols to update the unidirectional model; Including, the one-way model comprises an end-to-end speech recognition model; each observation in the input sequence of the training samples represents an acoustic feature; each symbol in the target sequence of training samples represents a phone, a context-dependent phone, a letter, a word part, or a word; The forward shifting reduces the time delay between spike timing emitted by a trained unidirectional model and the acoustic features. The method.

2. The method described in claim 1, wherein the unidirectional model is a unidirectional LSTM model.

3. The method of claim 1 or 2, wherein the unidirectional model is a recurrent neural network based model.

4. The method of claim 1 , wherein the loss is a connectionist temporal classification (CTC) loss.

5. 5. The method of claim 1, wherein shifting the sequence of predictions comprises adjusting a length of the shifted sequence of predictions and a length of the input sequence of observations to be identical.

6. 6. The method of claim 1, wherein shifting the sequence of predictions and updating the unidirectional model using the shifted sequence of predictions is performed at a predetermined rate.

7. The method of claim 6 , wherein the predetermined rate ranges from approximately 5% to 40%.

8. 8. The method of claim 1, wherein the amount to shift is fixed.

9. 9. The method of claim 1, wherein the amount to shift is determined probabilistically within a predetermined range.

10. 10. The method of claim 1, wherein the unidirectional model is a neural network-based model having a plurality of parameters, and wherein feeding the input sequence comprises forward propagation through the neural network-based model, and updating the unidirectional model comprises performing back propagation through the neural network-based model to update the plurality of parameters.

11. 1. A computer system for training a unidirectional model by executing program instructions, comprising: a memory for storing said program instructions; a processing circuit in communication with the memory for executing the program instructions, obtaining training samples including an input sequence of observations and a target sequence of symbols having a different length than the input sequence of observations; performing forward propagation by feeding the input sequence of observations into the unidirectional model to obtain a sequence of predictions that are posterior probability distributions; shifting the obtained sequence of predictions forward by an amount relative to the input sequence of observations to obtain a sequence of forward shifted predictions; and a processing circuit configured to back-propagate using the sequence of forward shifted predictions and the target sequence of symbols to update the unidirectional model based on a loss; Equipped with the one-way model comprises an end-to-end speech recognition model; each observation in the input sequence of the training samples represents an acoustic feature; each symbol in the target sequence of training samples represents a phone, a context-dependent phone, a letter, a word part, or a word; The forward shifting reduces the time delay between spike timing emitted by a trained unidirectional model and the acoustic features. Computer system.

12. The computer system of claim 11, wherein the unidirectional model is a unidirectional LSTM model.

13. 13. The computer system of claim 11 or 12, wherein the unidirectional model is a recurrent neural network based model.

14. A computer system described in any one of claims 11 to 13, wherein the loss is a CTC (Connectionist Time Series Classification) loss.

15. 15. The computer system of claim 11, wherein the processing circuitry is further configured to adjust the shifted sequence of predictions such that a length of the shifted sequence of predictions and a length of the input sequence of observations are the same.

16. 16. A computer system according to claim 11, wherein shifting the sequence of predictions and updating the unidirectional model using the shifted sequence of predictions is performed at a predetermined rate.

17. A computer program product for causing a computer to carry out the steps of the method according to any one of claims 1 to 10.

18. 20. A computer readable storage medium having recorded thereon the computer program of claim 17.

Images