Soft forgetting for automatic speech recognition based on temporal classification of connectionist mechanisms

Abstract

The present disclosure relates to the field of machine learning, and provides a method, a computer system, and a computer program product for soft forgetting of automatic speech recognition based on connection mechanism time classification. In the method of soft forgetting training, one or more computer processors train a first model using one or more training batches, wherein each of the one or more training batches includes one or more information blocks. In response to completing the training of the first model, one or more computer processors initiate training of a second model using one or more training batches. One or more computer processors jitter the random block size of each information block of each of the one or more training batches for the second model. One or more computer processors expand the second model on one or more non-overlapping continuous jittered information blocks. In response to the expansion of the second model, one or more computer processors reduce overfitting for the second model by applying twin regularization.

Description

Technical Field

[0001] The present invention relates generally to the field of machine learning and, more particularly, to automatic speech recognition. Background Art

[0002] End-to-end (E2E) automatic speech recognition (ASR) systems have been the subject of significant research. Such systems aim to simplify the complex training and inference pipelines of conventional hybrid ASR systems. Hybrid systems combine Gaussian mixture models, hidden Markov models (HMMs), different neural networks, and involve multiple stages of model building and alignment between sequences of speech features and HMM context-dependent states. In contrast, E2E systems use recurrent neural networks and train the acoustic model in one go by summing all alignments via a concatenated mechanism temporal classification loss or learning the best alignment via an attention mechanism.

[0003] Recurrent neural networks (RNNs) are a class of artificial neural networks in which the connections between nodes form a directed graph along a sequence, allowing the network to exhibit temporal dynamic behavior of time series. Unlike feedforward neural networks, RNNs can use internal states (memory) to process input sequences, allowing RNNs to be suitable for tasks such as unsegmented handwriting recognition or speech recognition. Long short-term memory (LSTM) cells are alternative layer cells for recurrent neural networks (RNNs). RNNs composed of LSTM cells are called LSTM networks. A common LSTM cell consists of a cell, an input gate, an output gate, and a forget gate. The cell memorizes values ​​over arbitrary time intervals, and each gate regulates the flow of information into and out of the cell. Gated recurrent unit (GRU) is a gating mechanism in recurrent neural networks. GRU performance on polyphonic music modeling and speech signal modeling is found to be similar to LSTM. However, GRU exhibits better performance on smaller data sets. LSTM can be a bidirectional (BLSTM) consisting of two LSTM networks, where each of the contained layers is expanded forward and backward in time. For E2E ASR systems, the BLSTM can be spread over the entire length of the speech utterance, allowing the BLSTM to better capture long-term context, which is especially useful in the absence of alignment.

[0004] Connectionist temporal classification (CTC) is a type of neural network output and associated scoring function, which is often used to train RNNs such as LSTM networks to solve sequence problems where the temporal alignment between the input sequence and the output sequence is not known a priori (e.g., online handwriting recognition or recognition of phonemes in speech audio). CTC is independent of the underlying neural network structure. The output tags of the CTC network include additional "blanks" or useless symbols that consume points in the input sequence that are not aligned with the output sequence. The CTC network has a continuous output (e.g., softmax) that is fitted by training to model the probability of the tag. The CTC score can be used with backpropagation to update the neural network weights. Summary of the invention

[0005] The embodiment of the present invention discloses a computer-implemented method, computer program product and system for soft forgetting training. The computer-implemented method includes one or more computer processors training a first model using one or more training batches, wherein each of the one or more training batches includes one or more information blocks. In response to the completion of the training of the first model, the one or more computer processors initiate the training of a second model using the one or more training batches. The one or more computer processors jitter the random block size of each information block of each of the one or more training batches for the second model. The one or more computer processors expand the second model on one or more non-overlapping continuous jittered information blocks. In response to the expansion of the second model, the one or more computer processors reduce the overfitting of the second model by applying twin regularization.

[0006] According to one aspect, a computer-implemented method is provided, comprising: training a first model using one or more training batches by one or more computer processors, wherein each of the one or more training batches includes one or more information blocks; in response to completion of the training of the first model, initiating training of a second model using the one or more training batches by one or more computer processors; jittering, by one or more computer processors, a random block size of each information block of each of the one or more training batches used for the second model; expanding, by one or more computer processors, the second model on one or more non-overlapping consecutive jittered information blocks; and reducing overfitting of the second model by applying double regularization, by one or more computer processors, in response to the expansion of the second model.

[0007] According to another aspect, a computer program product is provided, comprising: one or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, the stored program instructions comprising: program instructions for training a first model using one or more training batches, wherein each of the one or more training batches comprises one or more information blocks; program instructions for initiating training of a second model using the one or more training batches in response to completion of the training of the first model; program instructions for jittering a random block size of each information block of each of the one or more training batches for the second model; program instructions for expanding the second model over one or more non-overlapping consecutive jittered information blocks; and program instructions for reducing overfitting of the second model by applying twin regularization in response to the expansion of the second model.

[0008] According to another aspect, a computer system is provided, comprising: one or more computer processors; one or more computer-readable storage media; and program instructions stored on the computer-readable storage media for execution by at least one of the one or more processors, the stored program instructions comprising: program instructions for training a first model using one or more training batches, wherein each of the one or more training batches comprises one or more information blocks; program instructions for initiating training of a second model using the one or more training batches in response to completion of the training of the first model; program instructions for jittering a random block size of each information block of each of the one or more training batches for the second model; program instructions for expanding the second model over one or more non-overlapping consecutive jittered information blocks; and program instructions for reducing overfitting of the second model by applying twin regularization in response to the expansion of the second model. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings:

[0010] Figure 1 is a functional block diagram illustrating a computing environment according to an embodiment of the present invention;

[0011] Figure 2 is a diagram illustrating an embodiment of the present invention. Figure 1 A flowchart of the operating steps of a soft forgetting training program on a server computer within a computing environment;

[0012] Figure 3A An example diagram showing an example 300 according to an embodiment of the present invention is shown;

[0013] Figure 3BAn example 320 is shown, according to an embodiment of the present invention, where an example table depicts word error rates for a baseline model and a soft forgetting model;

[0014] Figure 3C An example 330 is shown, according to an embodiment of the present invention, where an example table depicts word error rates for a baseline model and a soft forgetting model;

[0015] Figure 3D Showing example 340, an example table depicting word error rates for a plurality of models, according to an embodiment of the present invention;

[0016] Figure 3E An example 350 is shown, according to an embodiment of the present invention, an example graph depicting word error rates for multiple models;

[0017] Figure 3F An example 360 ​​is shown, the example graph depicting CTC loss for a full-utterance BLSTM and a soft-forget BLSTM according to an embodiment of the present invention; and

[0018] Figure 4 is a block diagram of components of a server computer according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] When the machine translates the acoustic sequence into the corresponding phenomenon, an automatic speech recognition system (ASR) based on connectionist temporal classification (CTC) is usually used. When using a bidirectional long short-term memory (BLSTM) network deployed on the entire speech utterance, the automatic speech recognition system based on CTC performs well. However, the unidirectional long short-term memory (ULSTM) network traditionally lags behind the BLSTM network because the ULSTM only incorporates the forward-in-time context in time, while the BLSTM network additionally incorporates the backward-in-time context in time. Although the BLSTM network usually performs well for the described task, the strict BLSTM network leads to significant overfitting problems, and subsequently leads to increased error rates and reduced generalization applicability.

[0020] Embodiments of the present invention allow the use of soft forgetting training to significantly increase classification accuracy and reduce word error rate without significantly changing the underlying network. In an embodiment of the present invention, the BLSTM is expanded only on small non-overlapping information blocks or blocks of the input speech or acoustic sequence, thereby reducing overfitting. In another embodiment of the present invention, the block size of each batch is randomly assigned instead of a fixed global block value, thereby improving generalization performance. In yet another embodiment of the present invention, twin regularization is applied using the mean squared error between the trained full-speech BLSTM and the soft forgetting BLSTM, thereby retaining speech-level context information. The embodiment significantly improves both offline / non-streaming and streaming word error rates of the CTC ASR system. Implementations of embodiments of the present invention may take a variety of forms, and exemplary implementation details are discussed subsequently with reference to the accompanying drawings.

[0021] The present invention according to preferred embodiments will now be described in detail with reference to the accompanying drawings.

[0022] Figure 1 is a functional block diagram illustrating a computing environment generally designated 100 according to one embodiment of the present invention. The term "computing" as used in this specification describes a computer system that includes multiple physically distinct devices that operate together as a single computer system. Figure 1 This merely provides an illustration of one implementation and does not imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the described environment may be made by those skilled in the art without departing from the scope of the invention as described in the claims.

[0023] The computing environment 100 includes a server computer 120 connected to a network 102. The network 102 may be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN) (such as the Internet), or a combination of the three, and may include wired, wireless, or fiber optic connections. The network 102 may include one or more wired and / or wireless networks capable of receiving and transmitting data, voice, and / or video signals, including multimedia signals containing voice, data, and video information. In general, the network 102 may be any combination of connections and protocols that will support communication between the server computer 120 and a computing device (not shown) within the computing environment 100. In various embodiments, the network 102 operates locally via a wired, wireless, or optical connection, and may be any combination of connections and protocols (e.g., personal area network (PAN), near field communication (NFC), laser, infrared, ultrasound, etc.).

[0024] The server computer 120 may be a stand-alone computing device, a management server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, sending, and processing data. In other embodiments, the server computer 120 may represent a server computing system such as a server system utilizing multiple computers in a cloud computing environment. In another embodiment, the server computer 120 may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, or any programmable electronic device capable of communicating with a computing device (not shown) within the computing environment 100 via the network 102. In another embodiment, the server computer 120 represents a computing system utilizing clustered computers and components (e.g., database server computers, application server computers, etc.) that act as a single seamless resource pool when accessed within the computing environment 100. In the depicted embodiment, the server computer 120 includes a database 122 and a program 150. In other embodiments, the server computer 120 may include other applications, databases, programs, etc. not described in the computing environment 100. The server computer 120 may include internal and external hardware components, such as those described with respect to Figure 4 Depicted and described in more detail.

[0025] Database 122 is a repository of data used by program 150. In the depicted embodiment, database 122 resides on server computer 120. In another embodiment, database 122 may reside on computing device 110 or elsewhere within computing environment 100, as long as program 150 can access database 122. A database is an organized collection of data. Database 122 can be implemented with any type of storage device capable of storing data and configuration files that can be accessed and utilized by program 150, such as a database server, hard drive, or flash memory. In an embodiment, database 122 stores data used by program 150, such as training sets, labeled distribution data and graphs, and historical model accuracy and performance statistics. In the depicted embodiment, database 122 contains training corpus 124 and cognitive model 126.

[0026] The training corpus 124 may contain one or more sets of one or more examples of classified (e.g., labeled) data, referred to hereinafter as training sentences. In one embodiment, each training set includes a tag (e.g., a string, a text sequence, a character, etc.) and an associated array or set of training sentences that can be used to train one or more models. In a further embodiment, the training corpus 124 contains vectorized (i.e., one-hot encoding, word embedding, dimensionality reduction, etc.) training sets and associated training sentences. In various embodiments, the training corpus 124 contains complete or partial utterances in auditory or written form. In this embodiment, the utterance is a continuous sequence of audio feature vectors.

[0027] Model 126 utilizes one or more cognitive models to classify one or more question examples (e.g., natural language statements, test statements, utterances, continuous time step data, etc.). In an embodiment, model 126 utilizes transferable neural network algorithms and models (e.g., long short-term memory (LSTM), bidirectional long short-term memory (BLSTM), deep stacked network (DSN), deep belief network (DBN), convolutional neural network (CNN), recurrent neural network (RNN), composite hierarchical deep model, etc.). In the depicted embodiment, model 126 utilizes one or more BLSTM. In a further embodiment, the BLSTM is trained using a supervised training method that feeds a vectorized training set of feature data to model 126. In various embodiments, features are labeled with associated classes so that model 126 can learn which features are associated with a particular class before use. Model 126 is trained to recognize differences between tags or classes. Model 126 utilizes a training set to generate a probability set representing the probability of a question example (e.g., utterance, etc.) belonging to a particular tag or class. In this embodiment, the model 126 classifies (e.g., labels) question examples (e.g., questions, training vectors, or sentences, etc.) by considering different features available as structured or unstructured data. In one embodiment, the model 126 inputs a sequence of audio features (e.g., vectors) and outputs a speech label for each sequence. The training of the model 126 is relative to Figure 2 Describe and describe in more detail.

[0028] Program 150 trains one or more models using soft forgetting. In various embodiments, program 150 may implement the following steps. Program 150 is initiated after receiving a training request for one or more models. In one embodiment, program 150 is initiated after receiving or retrieving one or more training sets. Program 150 retrieves a set of training data. Program 150 trains one or more full-utterance bidirectional long short-term memory networks. Program 150 begins training one or more soft-forgotten bidirectional long short-term memory networks. Program 150 continues the training process by expanding one or more soft-forgotten bidirectional long short-term memory networks. Program 150 jitters the block size of one or more soft-forgotten bidirectional long short-term memory networks. Program 150 utilizes the one or more trained full-utterance bidirectional long short-term memory networks in the twin regularization of one or more soft-forgotten bidirectional long short-term memory networks. In the depicted embodiment, program 150 is a stand-alone software program. In another embodiment, the functionality of program 150 or any combination thereof may be integrated into a single software program. In some embodiments, program 150 may be located on a separate computing device (not depicted), but may still communicate over network 102. In various embodiments, the client version of program 150 resides on a computing device (not depicted) within computing environment 100. Figure 2 Process 150 is depicted and described in greater detail.

[0029] According to a preferred embodiment, the present invention may include different accessible data sources, such as database 122, which may include personal data, content or information that the user wishes not to be processed. Personal data includes personally identifiable information or sensitive personal information and user information, such as tracking or geolocation information. Processing refers to any automated or unautomated operation or set of operations, such as collecting, recording, organizing, structuring, storing, adapting, changing, retrieving, consulting, using, making available, combining, restricting, erasing or destroying personal data in other ways. Program 150 enables authorized and secure processing of personal data. Program 150 provides informed consent, notifying the collection of personal data, allowing users to opt in or opt out of processing personal data. Consent can take several forms. Opt-in consent can impose on users to take affirmative action before personal data is processed. Alternatively, opt-out consent can impose on users to take affirmative action to prevent personal data from being processed before personal data is processed. Program 150 provides information about the nature of personal data and processing (e.g., type, scope, purpose, duration, etc.). Program 150 provides users with a copy of the stored personal data. The procedure 150 allows to correct or complete incorrect or incomplete personal data. The procedure 150 allows to delete personal data immediately.

[0030] Figure 2is a flow chart depicting the operational steps of a program 150 for a soft forgetting training model according to an embodiment of the present invention.

[0031] The program 150 retrieves a set of training data (step 202). In an embodiment, the program 150 retrieves a plurality of utterances, acoustic sequences, and / or sentences associated with the training set. In one example, the program 150 utilizes a 300-hour English switchboard (SWB) dataset as a training dataset, and the program 150 utilizes the training data to train one or more speaker-independent (SI) and speaker-adapted (SA) models. In this example embodiment, the SI model uses 40-dimensional logMel features with per-speaker cepstral mean subtraction (CMS) without any vocal tract normalization (VTLN). The SA model uses 40-dimensional logMel features with per-speaker CMS and VLTN, 40-dimensional feature space maximum likelihood linear regression (FMLLR) features, and 100-dimensional i-vectors. In another embodiment, the program 150 processes the retrieved training dataset (e.g., utterances, acoustic sequences, etc.) into a plurality of subsets (e.g., blocks, information blocks). In yet another embodiment, the program 150 segments the historical utterances into discrete sets containing different processed versions of the same utterance. In various embodiments, program 150 defines the boundaries of an utterance by utilizing predetermined utterance intervals. In this embodiment, program 150 segments the utterance into one or more sets defined by time constraints. In an embodiment, program 150 vectorizes the training data set and the segmented utterance / token sets. In various embodiments, program 150 segments the training data into one or more training batches containing subsets of the training data. In this embodiment, program 150 may randomly create batches, or program 150 may receive batch sizes from a user.

[0032] Program 150 trains a full-utterance bidirectional long short-term memory network (step 204). Program 150 trains one or more full-utterance BLSTMs (e.g., a first set of models) using the retrieved training data set detailed in step 202. In an embodiment, program 150 performs supervised training using labeled vectorized training data. For example, program 150 feeds utterance / label pairs into model 126, allowing program 150 to make inferences between training data (e.g., utterances) and classification data (i.e., labels). In an embodiment, program 150 determines whether sufficient accuracy is obtained by utilizing a test set. BLSTM processes sequential data (x1, x2, ..., xt) to calculate subsequent hidden states, while the parameters of the model remain the same. In an example, program 150 trains a 6-layer BLSTM network with 512 hidden neurons in each forward and backward direction. In a further example, the 6-layer BLSTM outputs are mapped to a 1024-dimensional hidden vector representing a 45-dimensional posterior probability vector of 44 phonemes and blank symbols through a linear layer and a softmax layer output. In various implementations, supervised training determines the difference (i.e., error) between the prediction and the target, and propagates the difference back through the layers so that the model / network "learns." In an embodiment, program 150 utilizes a stochastic gradient algorithm such as synchronous stochastic gradient descent (SGD) to implement back propagation. In an embodiment, program 150 utilizes the following as the loss function:

[0033] ( Regarding Equation (1), the twin regularization loss is the LSTM hidden representation h f and h b The mean square error between . and denote the parameters of the forward and backward ULSTM networks respectively, x and y are the input acoustic sequence and the output token sequence, and λ>0 is used as a scaling factor.

[0034] Program 150 can modify the learning rate in order to adjust the CTC or cross entropy cost, thereby allowing program 150 to increase or decrease the adaptability of relevant units and layers. For example, program 150 uses a learning rate of 0.04, a Nesterov momentum of 0.9, and a batch size of 128. Program 150 modifies the learning rate by √0.05 in each epoch after the first 10 epochs. In an embodiment, program 150 adds a sequence noise injection that performs a long-sum-exp on the logMel feature sequence of the current utterance and the randomly sampled utterance from the training data set contained in the training corpus 124. In another embodiment, program 150 utilizes two hyperparameter weights on the features of the randomly sampled utterance and the probability of injecting sequence noise. In the example, program 150 utilizes a 0.4 value of the hyperparameter of the SI model and a 0.2 / 0.4 value of the hyperparameter of the SA model based on the calculated retention CTC loss. In various embodiments, program 150 retains the weights of the full-utterance BLSTM fixed during training.

[0035] Program 150 expands a block-based bidirectional long short-term memory network (step 206). Program 150 initiates soft forgetting training of one or more block-based BLSTM networks (e.g., a second set of models). Program 150 concurrently trains and expands one or more block-based BLSTMs, taking the previous hidden state and input at this time step to generate the input of the next series of time steps, allowing information to flow through the network so that the last unit includes information derived from all previous units. Program 150 identifies one or more non-overlapping windows of C consecutive time steps of the input acoustic sequence (e.g., speech). In an embodiment, program 150 expands one or more block-based BLSTMs only on the identified non-overlapping windows of C consecutive time steps of the input acoustic sequence. Traditionally, full speech (e.g., no soft forgetting) BLSTM overfits the training data, especially when the training data is limited. In another embodiment, program 150 sets the hidden state and unit state to zero after each C time step. In this embodiment, C is empirically calculated based on the retention (e.g., test speech) CTC loss. In an example, the block size may range from 5 to 50 stacked frames (100 milliseconds to 1 second) in steps of five stacked frames.

[0036] The program 150 jitters the block size of a block-based bidirectional long short-term memory network (step 208). The program 150 utilizes the training data set (as described in step 202) and the training technique utilized in step 204 to train one or more block-based BLSTMs (as referenced in step 206). In an embodiment, the program 150 splits each training sequence into short blocks or chunks (e.g., duration) with additional context observations. In various embodiments, the program 150 perturbs C from one training batch to the next. The perturbation improves the generalization performance of the model. In an embodiment, the program 150 perturbs C by adding jitter to the block size of each batch during training. In different embodiments, the program 150 randomly assigns (e.g., perturbs, jitters) the block size. In this embodiment, C is randomly assigned between batches in training. In an example, the program 150 jitters the block size C of each batch as follows batch :C batch =C+u, where u~U(-A, A) is a uniform discrete random variable on [-A, A]. Continuing from the example, program 150 jitters across training batches between intervals U(-2, 2). In this example, program 150 selects a block size of 40 stacked frames. In a further embodiment, program 150 calculates A empirically based on a hold-out (e.g., test or validation utterance set) CTC loss.

[0037] The program 150 applies twin regularization to the block-based bidirectional long short-term memory network (step 210). Twin regularization adds a regularization term that forces the forward hidden state to be as close as possible to the simultaneous backward hidden state computed by the twin network (e.g., full-utterance BLSTM) running backward in time. In addition to the CTC loss, the program 150 adds the twin regularization loss to the training of one or more block-based BLSTMs. In an embodiment, as described in step 204, the twin regularization loss is the mean squared error between the hidden states of the trained full-utterance BLSTM network. The overall loss function for soft forgetting is as follows:

[0038] (2)

[0039] Regarding equation (2), L totCorresponds to the sum of the CTC loss and the twin regularization loss using a full-utterance BLSTM. Setting λ to 0 results in hard forgetting, where hidden or cell state information is not transferred across blocks. In an embodiment, program 150 sets λ to a non-zero value, which significantly improves the word error rate (WER) when compared to λ with a zero value (e.g., hard forgetting). In this embodiment, soft forgetting retains some of the whole-utterance context through twin regularization. The weights of the whole-utterance BLSTM remain fixed during the training process. In various embodiments, once the block-based BLSTM network has completed training, program 150 discards (e.g., disposes of, eliminates, deletes, etc.) the BLSTM network for the entire utterance and performs interference without any modification.

[0040] In various embodiments, once the soft forgetting BLSTM network completes training, program 150 discards the full-utterance BLSTM network and performs interference without any modification. In one embodiment, program 150 transmits and / or installs one or more soft forgetting BLSTMs to a production environment or server. In various embodiments, program 150 may select multiple soft forgetting BLSTMs and deploy the models to the same production environment or deploy the models to multiple corresponding production, test, or auxiliary environments. In one embodiment, program 150 hosts (e.g., allows access and utilization) one or more trained soft forgetting BLSTMs, thereby allowing applications and programs (not depicted) to access the model. In this embodiment, the applications and programs can feed inputs into the trained soft forgetting BLSTMs and can receive or request associated outputs.

[0041] Therefore, by executing Figure 2 In the operating steps, program 150 trains one or more soft forgetting BLSTMs. Program 150 trains (e.g., pre-trains) one or more full-utterance BLSTMs. Program 150 initiates training of one or more soft forgetting BLSTMs by initializing one or more block-based BLSTMs. In this embodiment, the soft forgetting BLSTM reduces the overfitting problem caused by the full-utterance BLSTM. Program 150 adds jitter (e.g., randomness) to the block size, thereby improving the generalization performance of the model. In this embodiment, the jittering block size prevents model memorization of fixed-size blocks. Program 150 applies twin regularization and CTC loss using the trained full-utterance BLSTM, thereby allowing utterance-level context information to be retained.

[0042] Figure 3AAn example 300 is shown, which is an example illustration of a soft forgetting training process. The example 300 includes a pre-trained full-utterance BLSTM network 302, which is an example BLSTM trained without soft forgetting, a block-based BLSTM network 304, which is an example BLSTM trained with soft forgetting, an acoustic feature sequence 306 (which is an utterance represented as a spectrogram), a forward LSTM network 308 (which is a component of the BLSTM network), a backward LSTM network 310 (which is a component of the BLSTM network), an LSTM output 312 (which is a layer containing the output of the previous network), a linear layer 314 (which is an adaptive layer modified to conform to changes between input and output), a twin regularization 316 (which illustrates the process described in step 210), and a true symbol sequence 318 (which is a symbol equivalent of the acoustic feature sequence 306).

[0043] Figure 3B Example 320 is shown, which is a table depicting word error rates (WER) of one or more full-utterance and soft-forget BLSTMs trained with the Hub5-2000 Switchboard (SWB) and Call Home (CH) training sets. Example 320 shows that the full-utterance training network overfits, while the soft-forget training (e.g., block-based) network does not overfit.

[0044] Figure 3C Example 330 is shown, which is a table depicting word error rates (WER) for a set of speaker-independent models and speaker-adaptive models, each model including a baseline model (e.g., full-utterance BLSTM) and a soft forget model (e.g., block-based BLSTM) trained with the Hub5-2000 Switchboard (SWB), Call Home (CH), RT02, RT03, and RT04 training sets. Example 330 demonstrates an absolute improvement of 0.9% and 1.6% for the soft forget model over the corresponding baseline model.

[0045] Figure 3D An example 340 is shown, which is an example graph depicting word error rates for multiple conventional models trained with a Hub5-2000 switch board (SWB) and call home (CH) training set. Models denoted with a "*" symbol utilize velocity perturbations and models denoted with a "+" symbol utilize byte pair encoding.

[0046] Figure 3E Illustrated is an example 350 , which is a graph depicting word error rates for multiple models, including a block-based BLSTM, a soft-forget BLSTM, a full-utterance BLSTM, and a ULSTM with twin regularization.

[0047] Figure 3FExample 360 ​​is shown, which is a graph showing the CTC loss for a full-utterance trained BLSTM and a block-based BLSTM trained with soft forgetting. Example 360 ​​includes a full-utterance BLSTM 362 and a soft-forget BLSTM 364.

[0048] Other comments and / or examples

[0049] Some embodiments of the present invention recognize the following facts, potential problems, and / or potential areas of improvement relative to the prior art.

[0050] Automatic speech recognition systems based on connectionist temporal classification (CTC) perform well when using a bidirectional long short-term memory (BLSTM) network spread over an entire speech utterance. Some embodiments of the present invention recognize that the full-utterance BLSTM better captures long-term context but leads to overfitting. Embodiments of the present invention propose soft forgetting as a solution. During training, some embodiments of the present invention spread the BLSTM only on small non-overlapping blocks of the input utterance. Some embodiments of the present invention randomly pick block sizes for each batch instead of a fixed global block size. In order to retain some utterance-level information, some embodiments of the present invention encourage the hidden states of the BLSTM to be similar to the hidden states of the preprocessed full-utterance BLSTM. Using a 300-hour English exchange board dataset, some embodiments of the present invention show that soft forgetting improves the word error rate (WER) to an average of 7%-9% above the competing whole-utterance phone CTC BLSTM. Some embodiments of the present invention obtain 9.1% / 17.4% WER using speaker-independent models and 8.7% / 16.8% WER using speaker-adaptive models on the Hub5-2000 switchboard / call-home test set, respectively. Some embodiments of the present invention recognize that soft forgetting improves WER when the model is used in the limited temporal context of flow recognition. Finally, some embodiments of the present invention present empirical insights into the effects of regularization and data augmentation of soft forgetting.

[0051] Some embodiments of the present invention recognize that end-to-end (E2E) automatic speech recognition (ASR) systems have become the subject of important recent research. Such systems are intended to simplify the complex training and inference pipelines of conventional hybrid ASR systems. Hybrid systems combine Gaussian mixture models, hidden Markov models (HMMs), and different neural networks, and involve multiple stages of model construction and alignment between speech feature sequences and HMM context-related states. In contrast, end-to-end systems use recursive neural networks and train acoustic models at one time by summing all alignments via a connection mechanism temporal classification (CTC) loss or learning the best alignment through an attention mechanism. Some embodiments of the present invention recognize that the word error rate (WER) gap between E2E and hybrid ASR systems has decreased over time.

[0052] Some embodiments of the invention recognize that recursive neural networks with long short-term memory (LSTM) hidden units are the neural network of choice for ASR systems. Bidirectional LSTM (BLSTM) networks are particularly popular and include two LSTM networks at each layer that are expanded forward and backward in time. For E2E ASR systems, the BLSTM network is expanded over the entire length of the speech utterance. Full utterance expansion enables the BLSTM to better capture long-term context, which is particularly useful in the absence of alignment. Control of memorizing long-term context is left to four trainable gates (input, forget, unit, and output) of the LSTM unit. Some embodiments of the invention identify architectural variations of the LSTM unit in order to control this information processing behavior.

[0053] Some embodiments of the present invention recognize that full-utterance unfolding of BLSTM leads to overfitting, even in the presence of well-known regularization techniques such as dropout. This is particularly harmful to the WER of E2E ASR systems given the limited training data (e.g., a few hundred hours of speech). Some embodiments of the present invention utilize soft forgetting to combat this overfitting. First, some embodiments of the present invention unfold the BLSTM only on small non-overlapping blocks of the input acoustic utterance rather than the entire utterance. The hidden and cell states of the forward and backward LSTM networks are reset to zero at the block boundaries. To prevent memorization of fixed-size blocks, some embodiments of the present invention randomly perturb the block size across batches during training. Finally, some embodiments of the present invention use twin regularization in order to retain some utterance-level context. Twin regularization adds the mean squared error between the hidden states of the chunked BLSTM model and the pre-trained full-utterance BLSTM model to the CTC loss. Some embodiments of the present invention recognize that twin regularization promotes some memorization of context across blocks.

[0054] Some embodiments of the present invention have considered chunked training of CTC ASR models primarily for streaming inference. However, soft forgetting additionally combines chunk jittering and twin regularization, and significantly improves both offline / non-streamed and streaming WER for CTC ASR systems. Some embodiments of the present invention show that, using a 300-hour English exchange board dataset, soft forgetting significantly improves WER by 7%-9% relative to a competitive phoneme CTC baseline across several test sets. Some embodiments of the present invention present empirical evidence for the regularization and data augmentation effects of soft forgetting.

[0055] Some embodiments of the present invention recognize that the ULSTM network lags behind the BLSTM network in terms of ASR WER because the ULSTM network only incorporates the temporal forward context, while the BLSTM network additionally incorporates the temporal backward context. In its original formulation, the twin regularization jointly trains two ULSTM networks that operate independently in temporal forward and temporal backward. The overall training loss is:

[0056] ()

[0057] Among them, the twin regularization loss is the LSTM hidden representation h f and h b The mean square error between:

[0058] (3) in, and denote the parameters of the forward and backward ULSTM networks respectively, x and y are the input acoustic and output token sequences, and λ>0 is the scaling factor.

[0059] Figure 3A Depicts a block diagram of soft forgetting that contains the following elements:

[0060] Block-based BLSTM network: The soft forgetting training BLSTM is only expanded on non-overlapping windows of C consecutive time steps of the input acoustic sequence. The intuition behind this choice is that the full-utterance BLSTM tends to overfit the training data, especially when it is restricted. Setting the hidden and cell states to 0 after every C time steps alleviates this overfitting. Some embodiments of the present invention choose C empirically based on the leave-out CTC loss.

[0061] Block size jittering: Some embodiments of the invention recognize that perturbing C from one batch to the next improves the generalization performance of the model. Some embodiments of the invention recognize that adding jittering has a data augmentation effect. Some embodiments of the invention jitter the block size C of each batch as follows batch :C batch =C+u, where u＝U(A, A) is a discrete random variable uniformly distributed on [A, A]. Some embodiments of the present invention select A empirically based on the leave-out CTC loss.

[0062] · Twin Regularization: To incorporate some utterance-level context, some embodiments of the present invention add a twin regularization loss in addition to the CTC loss. The loss is the mean squared error between the hidden states of the pre-trained full-utterance BLSTM network and the currently trained block-based BLSTM network. Compared to the original formulation of twin regularization where the two models are jointly trained, some embodiments of the present invention keep the weights of the full-utterance BLSTM weights fixed during training.

[0063] Therefore, the overall loss function of soft forgetting is:

[0064] (2) Setting λ=0 results in hard forgetting, where hidden or cell state information is not transferred across blocks. Some embodiments of the invention recognize that non-zero values ​​of λ produce significantly better WERs than λ=0, indicating the value of preserving some full-utterance context through twin regularization.

[0065] Some embodiments of the present invention recognize that block-based training of BLSTM does not require any block-by-block alignment and only affects the forward pass through the BLSTM network. Some embodiments of the present invention recognize a batch of B utterances with N blocks as a batch of BN small utterances. Some embodiments of the present invention reshape the resulting block-by-block posterior vectors to construct a posterior sequence of the original full utterance and use it to calculate the CTC loss. Once the block-based BLSTM network is trained using soft forgetting, some embodiments of the present invention discard the full-utterance BLSTM network and perform inference without any modification.

[0066] Some embodiments of the present invention use a 300-hour English exchange board task to train a speaker-independent (SI) model and a speaker-adapted (SA) model. The SI model uses 40-dimensional logMel features with per-speaker cepstral mean subtraction (CMS) and without any vocal tract length normalization (VTLN). The SA model uses 40-dimensional logMel features with per-speaker CMS and VTLN, 40-dimensional feature space maximum likelihood linear regression (FMLLR) features, and 100-dimensional i-vectors, similar to a feature fusion system.

[0067] Some embodiments of the present invention train multiple 6-layer BLSTM networks in PyTorch, each with 512 hidden neurons in the forward and backward directions. The output 1024-dimensional hidden vector is mapped to a 45-dimensional posterior probability vector representing 44 phonemes and blank symbols through a linear layer and softmax. Some embodiments of the present invention include models that utilize increments and double increments on logMel features, and frame stacking and skipping at a rate of 2 for logMel and FMLLR features. This produces 240-dimensional input acoustic features for the SI model and 420-dimensional input acoustic features for the SA model. The 4-gm LM is trained on all texts from the SWB+Fisher corpus with a vocabulary of 30k words.

[0068] Some embodiments of the present invention use synchronous stochastic gradient descent (SGD) to train all models with a learning rate of 0.04, a Nesterov momentum of 0.9, and a batch size of 128 on 4 V100 GPUs. After the first 10 epochs, the learning rate is multiplied by 0.5 each epoch. Some embodiments of the present invention also utilize sequence noise injection on logMel features for both SI and SA models. Sequence noise injection performs log-sum-exp on the sequence of logMel features of the current utterance and randomly sampled utterances from the training set, which regularizes the model and can also be considered as instant data augmentation because it creates a new sequence of input features. There are two hyperparameters: the weights on the features of the randomly sampled utterances and the probability of injecting sequence noise. Some embodiments of the present invention use values ​​of (0.4, 0.4) for the hyperparameters of all SI models based on the leave-out CTC loss, and use values ​​of (0.2, 0.4) for the hyperparameters of all SA models.

[0069] Some embodiments of the invention characterize different elements of soft forgetting in steps to gain an understanding of the impact they have on WER. First, some embodiments of the invention train a block-based BLSTM model with a fixed block size. Some embodiments of the invention vary the block size from 5 to 50 stacked frames (100 milliseconds to 1 second) in steps of 5 stacked frames. Some embodiments of the invention show that based on the leave-out CTC loss, the optimal block size is 40 stacked frames, which is Figure 3B The Hub5-2000SWB / CH shown on the test set shows a WER of 12.7% / 22.5%.

[0070] Next, some embodiments of the present invention introduce block size jittering of U(2,2) across training batches, which leads to further absolute reductions in WER of 0.6% on SWB and 1.0% on CH. Some embodiments of the present invention recognize that the block-based BLSTM model with hard forgetting has a WER comparable to the full-utterance BLSTM model without sequence noise injection. Some embodiments of the present invention introduce twin regularization on the last three BLSTM layers and vary λ on 0.001, 0.01, 0.1, 1.0. Some embodiments of the present invention recognize that the best leave-out CTC loss is at λ=0.01, which leads to a WER of 11.1% on SWB and a WER of 19.7% on CH, which is only 0.2% worse than the sequence noise injection baseline on SWB and 1.6% absolutely better on CH. Finally, some embodiments of the present invention introduce sequence noise injection, resulting in a WER of 10.6% on SWB and a WER of 19.5% on CH. The benefits of sequence noise injection are mitigated in the presence of soft forgetting, as both techniques have the same effect of regularizing the model and performing on-the-fly data augmentation.

[0071] Some embodiments of the present invention perform a grid search by jointly varying the block size over 5, 10, ..., 50 and the block jitter over 1, 2, ..., 10 on the best model including twin regularization and sequence noise. Some embodiments of the present invention show that most values ​​of block size 20 and block jitter 6 produce almost equally low holdout losses. In response to picking the best block size of 40 and block jitter of 10, some embodiments of the present invention only realize a minor improvement of 0.2% in CH WER and no improvement in SWB.

[0072] Next, some embodiments of the present invention apply the soft forgetting formulation to the SA model. Figure 3C Results for a baseline CTC model and a model with soft forgetting across five test sets are shown. Some embodiments of the present invention recognize that soft forgetting achieves 0.9% and 1.6% absolute improvements over the baseline SA model for the SWB and CH test sets. For reference, some embodiments of the present invention also show the results from Figure 3BSoft forgetting improves the average WER of the SI model by 1.3% absolute and the average WER of the SA model by 1.4% absolute. Some embodiments of the present invention then perform state-level minimum Bayesian risk (sMBR) sequence training of these models and perform a rescan of the resulting lattice using an LSTM language model (NNLM) trained on the Fisher+SWB corpus. The embedding layer of the NNLM has 512 nodes, followed by 2 LSTM layers, each with 2048 nodes. Before the softmax-based estimation of the 30k-dimensional word posterior vector, the feature space is reduced to 128 by a linear bottleneck layer. Some embodiments of the present invention use a combination of DropConnect and Dropout to regularize the model. Some embodiments of the present invention utilize SGD with an initial learning rate of 0.01 and a Nesterov momentum of 0.9 for training. After 20 training epochs, the learning rate is annealed by a factor of 0.5 in 15 steps. Some embodiments of the present invention obtain final WERs of 9.6% / 17.7% for SI and 8.7% / 16.8% for the SA phoneme CTC model with soft forgetting.

[0073] To measure the performance of a phoneme CTC model with soft forgetting, some embodiments of the present invention show Figure 3D The WER of various models in the literature on the Hub5-2000 SWB and CH test sets. Some embodiments of the present invention perform data augmentation using speed perturbations (0.9x, 1x, and 1.1x) to enable our SI system to compare it with other systems using data augmentation. Speed ​​perturbations further improve the SI system to 9.1% / 17.4%. Some embodiments of the present invention recognize that the model compares favorably with all end-to-end systems. The WER improvement is particularly large on the more challenging CH test set. Soft forgetting also significantly helps to reduce the WER gap between the phoneme CTC and SA hybrid BLSTM models with feature fusion using a complex multi-stage training pipeline and 32k CD states as output. Specifically, some embodiments of the present invention reduce the WER gap between the SA hybrid BLSTM and phoneme CTC by 71% for SWB and 74% for CH compared to models using only sequence noise injection.

[0074] Some embodiments of the present invention utilize SI and SA models before sMBR and NNLM rescanning for decoding with bounded latency, and similarly can be drawn using SA models. These embodiments unroll the BLSTM model on non-overlapping blocks during inference. The hidden and cell states of the forward LSTM network are copied from one block to the next as it improves the WER. The backward LSTM hidden and cell states are reset to zero.

[0075] Figure 3E The WER of a chunked BLSTM with and without soft forgetting versus decoding block size is shown. For each choice of decoding block size, some embodiments of the present invention report the best WER across all training block sizes. As a reference, some embodiments of the present invention also show the WER of a full-utterance BLSTM and a competitive full-utterance ULSTM model trained using a twin regularization loss for the full-utterance BLSTM. Some embodiments of the present invention recognize that a chunked BLSTM trained with soft forgetting significantly improves the chunked BLSTM across all decoding block sizes.

[0076] Some embodiments of the invention recognize that full-utterance BLSTM models tend to overfit the data, and soft forgetting is one way to mitigate this. Figure 3F The training and hold-out CTC losses are shown as training proceeds for a baseline full-utterance SI BLSTM and a chunked SI BLSTM with soft forgetting. Some embodiments of the invention recognize that the converged training and hold-out losses for the chunk-based BLSTM with soft forgetting are significantly closer when compared to the full-utterance BLSTM, indicating that soft forgetting does regularize the model.

[0077] Some embodiments of the present invention recognize that the hidden and cell states of the BLSTM are reset after the random block size effectively creates data augmentation. This random forgetting causes the BLSTM to output different hidden representations for the same input acoustic feature sequence and creates different training samples. Some embodiments of the present invention utilize the best SABLSTM model with soft forgetting and forward Hub5-2000SWB test set. For each utterance, some embodiments of the present invention expand the BLSTM on the entire utterance and on blocks of size 30, 31, ..., 50 representing the block size used in training. Some embodiments of the present invention reset the hidden and cell states to zero after each block. Some embodiments of the present invention then calculate a 1024-dimensional representation by averaging the output of the final BLSTM layer and projecting it down to 2 dimensions using t-distributed random neighbor embedding (t-SNE). Some embodiments of the present invention recognize that the embedding shows acoustic clustering across utterances, for example, all utterances with hesitation are clustered in one region of the space. More importantly, some embodiments of the invention recognize that, due to the twin regularization loss used during training, forgetting the hidden and cell states after blocks of different sizes perturbs the embeddings in the local neighborhood of each utterance. The nature of this perturbation is different for each utterance and is not simply independent and identically distributed noise due to the highly nonlinear nature of the underlying representation. Some embodiments of the invention recognize that this perturbation is an effective form of data augmentation.

[0078] As follows, embodiments of the present invention utilize soft forgetting to train a better CTC ASR model consisting of three elements. First, embodiments of the present invention only expand the BLSTM on non-overlapping blocks of the input acoustic frames instead of the full utterance. Therefore, the hidden and cell states are forgotten from one block to the next. Second, embodiments of the present invention perturb the block durations across training batches. Finally, embodiments of the present invention recognize that the CTC loss is regularized by promoting the hidden representations of the BLSTM to approach those from a pre-trained full-utterance BLSTM model. Some embodiments of the present invention show that soft forgetting improves WER by 7%-9% compared to a competitive phoneme CTC baseline, and also helps to bring the WER gap with the prior art hybrid BLSTM close to about 70%.

[0079] Figure 4 1 shows a block diagram of the components of the server computer 120 according to an illustrative embodiment of the present invention. It should be understood that Figure 4 This merely provides an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications may be made to the depicted environments.

[0080] The server computers 120 each include a communication fabric 404 that provides communications between cache 403, memory 402, permanent storage 405, communication units 407, and input / output (I / O) interfaces 406. The communication fabric 404 may be implemented with any architecture designed to pass data and / or control information between processors (such as microprocessors, communication and network processors, etc.), system memory, peripheral devices, and any other hardware components within the system. For example, the communication fabric 404 may be implemented with one or more buses or crossbar switches.

[0081] Memory 402 and permanent storage 405 are computer-readable storage media. In this embodiment, memory 402 includes random access memory (RAM). In general, memory 402 may include any suitable volatile or non-volatile computer-readable storage media. Cache 403 is a fast memory that enhances the performance of computer processor 401 by saving recently accessed data from memory 402 and data that is close to being accessed.

[0082] Program 150 may be stored in permanent storage 405 and memory 402 for execution by one or more corresponding computer processors 401 via cache 403. In an embodiment, permanent storage 405 includes a magnetic hard drive. As an alternative or in addition to a magnetic hard drive, permanent storage 405 may include a solid-state hard drive, a semiconductor memory device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0083] The media used by permanent storage 405 also may be removable. For example, a removable hard drive may be used for permanent storage 405. Other examples include optical and magnetic disks, thumb drives, and smart cards, which are inserted into a drive for transfer to another computer-readable storage medium that is also part of permanent storage 405.

[0084] In these examples, communications unit 407 provides communications with other data processing systems or devices. In these examples, communications unit 407 includes one or more network interface cards. Communications unit 407 may provide communications using physical and / or wireless communications links. Program 150 may be downloaded to permanent storage 405 via communications unit 407.

[0085] The I / O interface 406 allows for input and output of data with other devices that may be connected to the server computer 120. For example, the I / O interface(s) 406 may provide a connection to an external device(s) 408, such as a keyboard, a keypad, a touch screen, and / or some other suitable input device. The external device 408 may also include a portable computer-readable storage medium, such as, for example, a thumb drive, a portable optical or magnetic disk, and a memory card. Software and data (e.g., program 150) for implementing embodiments of the present invention may be stored on such a portable computer-readable storage medium and may be loaded onto the permanent storage device 405 via the I / O interface(s) 406. The I / O interface(s) 406 are also connected to a display 409.

[0086] Display 409 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0087] The programs described herein are identified based on the applications implemented in the specific embodiments of the invention. However, it should be understood that any specific program terminology herein is used only for convenience, and thus the present invention should not be limited to use only in any specific application identified and / or implied by such terminology.

[0088] The present invention may be a system, method, and / or computer program product. The computer program product may include computer-readable storage medium(s) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0089] Computer readable storage medium can be a tangible device that can retain and store instructions for use by instruction execution devices. Computer readable storage medium can be, for example but not limited to, electronic storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage medium includes the following: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device such as punch card or a protruding structure in a groove with instructions recorded thereon, and any suitable combination of the above. Computer readable storage medium as used herein should not be interpreted as a temporary signal itself, such as radio waves or other free propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (e.g., light pulses passing through fiber optic cables) or electrical signals emitted by wires.

[0090] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or downloaded to an external computer or external storage device. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0091] The computer readable program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., conventional procedural programming languages ​​such as "C" programming language or similar programming languages, and quantum programming languages ​​such as "Q" programming language, Q#, quantum computing language (QCL) or similar programming languages, low-level programming languages ​​such as assembly language or similar programming languages. The computer readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a stand-alone software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuits so as to perform aspects of the present invention.

[0092] The present invention will be described below with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention. It should be understood that each box of the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer readable program instructions.

[0093] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which causes the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable storage medium in which the instructions are stored includes a manufactured product containing instructions for implementing aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0094] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in or in multiple boxes in the flowchart and / or block diagram.

[0095] The flow charts and block diagrams in the accompanying drawings show the architecture, functions and operations of the possible implementations of the system, method and computer program product according to different embodiments of the present invention. To this end, each box in the flow chart or block diagram may represent a module, segment or part of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may not occur in the order marked in the figure. For example, depending on the functions involved, the two blocks shown in succession can actually be executed substantially simultaneously, or these blocks can sometimes be executed in reverse order. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0096] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present invention. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found in the marketplace, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for soft forgetting training, comprising: training, by one or more computer processors, a first model using one or more training batches, wherein each of the one or more training batches comprises one or more information chunks; In response to completion of training of the first model, initiating, by one or more computer processors, training of a second model using the one or more training batches; jittering, by one or more computer processors, a random block size of each information block of each of the one or more training batches for the second model; unfolding the second model over one or more non-overlapping consecutive blocks of jitter information by one or more computer processors; as well as In response to the unfolding of the second model, overfitting of the second model is reduced by one or more computer processors by applying twin regularization.

2. The method according to claim 1, wherein: The first model is a full-utterance bidirectional long short-term memory network.

3. The method according to claim 1, wherein: The second model is a block-based bidirectional long short-term memory network.

4. The method according to claim 1, wherein: The one or more non-overlapping consecutive jitter information blocks are calculated from a connection mechanism time classification loss.

5. The method according to claim 1, wherein: The information blocks are acoustic sequences.

6. The method according to claim 1, wherein: The training batch includes one or more acoustic sequences, wherein each acoustic sequence has an associated textual tag.

7. The method according to claim 1, further comprising: responsive to completion of the training of the second model, deploying, by one or more computer processors, the first model; as well as In response to the completion of the training of the second model, the second model is deployed to one or more production environments by one or more computer processors.

8. The method according to claim 1, wherein: The twin regularization includes a loss value of the second model and a connection mechanism temporal classification loss, wherein the loss value is the mean squared error between the first model and the second model and the hidden state of the first model.

9. A computer program product for soft forgetting training, comprising: One or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, the stored program instructions comprising: Program instructions for training a first model using one or more training batches, wherein each of the one or more training batches includes one or more information blocks; program instructions for initiating training of a second model using the one or more training batches in response to completion of the training of the first model; program instructions for jittering a random block size of each information block of each of the one or more training batches for the second model; program instructions for spreading the second model over one or more non-overlapping consecutive blocks of dither information; and Program instructions for reducing overfitting of the second model by applying twin regularization in response to the unfolding of the second model.

10. The computer program product according to claim 9, wherein: The first model is a full-utterance bidirectional long short-term memory network.

11. The computer program product of claim 9, wherein: The second model is a block-based bidirectional long short-term memory network.

12. The computer program product of claim 9, wherein: The one or more non-overlapping consecutive jitter information blocks are calculated from a connection mechanism time classification loss.

13. The computer program product of claim 9, wherein: The program instructions stored on the one or more computer-readable storage media include: program instructions for deploying, by one or more computer processors, the first model in response to completion of training of the second model; and In response to completion of training of the second model, program instructions for deploying the second model to one or more production environments by one or more computer processors.

14. The computer program product of claim 9, wherein: The twin regularization includes a loss value of the second model and a connection mechanism temporal classification loss, wherein the loss value is a mean squared error between the first model and the second model and a hidden state of the first model.

15. A computer system for soft forgetting training, comprising: one or more computer processors; one or more computer-readable storage media; as well as Program instructions stored on a computer-readable storage medium for execution by at least one of the one or more processors, the stored program instructions comprising: Program instructions for training a first model using one or more training batches, wherein each of the one or more training batches includes one or more information blocks; program instructions for initiating training of a second model using the one or more training batches in response to completion of the training of the first model; program instructions for jittering a random block size of each information block of each of the one or more training batches for the second model; program instructions for spreading the second model over one or more non-overlapping consecutive blocks of dither information; and Program instructions for reducing overfitting of the second model by applying twin regularization in response to the unfolding of the second model.

16. The computer system of claim 15, wherein: The first model is a full-utterance bidirectional long short-term memory network.

17. The computer system according to claim 15, wherein: The second model is a block-based bidirectional long short-term memory network.

18. The computer system according to claim 15, wherein: The one or more non-overlapping consecutive jitter information blocks are calculated from a connection mechanism time classification loss.

19. The computer system of claim 15, wherein: The information blocks are acoustic sequences.

20. The computer system of claim 15, wherein: The training batch includes one or more acoustic sequences, wherein each acoustic sequence has an associated textual tag.

21. The computer system of claim 15, wherein: The program instructions stored on the one or more computer-readable storage media include: program instructions for deploying, by one or more computer processors, the first model in response to completion of training of the second model; and In response to completion of training of the second model, program instructions for deploying the second model to one or more production environments by one or more computer processors.

22. The computer system of claim 15, wherein: The twin regularization includes a loss value of the second model and a connection mechanism temporal classification loss, wherein the loss value is a mean squared error between the first model and the second model and a hidden state of the first model.

Images