Method and computer system for training a model

Abstract

A computer-implemented method for training a model is disclosed. The model is capable of retaining a history of one or more previous elements and having a predicted direction. The method includes obtaining a training sequence of elements. The method also includes segmenting the training sequence into a plurality of parts. The method also includes selecting one of the plurality of parts according to the direction of the model to generate modified training data. The method also includes using the modified training data to further train the model.

Description

Background Art

[0001] The present disclosure generally relates to techniques for training models for prediction and techniques for rescoring identification hypotheses using models.

[0002] In speech recognition, RNN (Recurrent Neural Network) language models are used for decoding and lattice rescoring, which applies the RNN language model to lattices generated by weaker language models such as n-gram models. With the help of the stronger modeling ability of the RNN language model, the error rate of the rescored lattices is reduced compared to the original lattices generated by the weaker language model.

[0003] RNN language models have hidden states that represent an infinite history of previous words, which is the source of their strong modeling power. RNN language models are usually trained with written text and / or transcribed utterances that may be error-free. In the case of domain adaptation, automatically transcribed text can also be used as training data. However, error trends in automatically transcribed training text are usually different from those observed in lattices generated by weaker language models.

[0004] In the assumptions given as a primary result of automatic speech recognition, recognition errors are inevitably included at any word position. However, RNN language models are not trained to handle such errors, so when errors are encountered in the assumptions, the hidden state of the RNN will collapse, which has a negative impact when calculating the language model probability of subsequent words. This negative impact on the re-scoring process due to errors in the assumptions may also appear in other recognition tasks, such as machine translation tasks, image or movie subtitling tasks, to name a few.

[0005] Therefore, there is a need for a technique that can mitigate the negative impact on forecasts due to errors in assumptions. Summary of the invention

[0006] According to an embodiment of the present invention, a computer-implemented method for training a model is provided. The method includes obtaining a training sequence of elements for a model, wherein the model is capable of retaining a history of one or more previous elements and having a prediction direction. The method also includes segmenting the training sequence into a plurality of parts. The method also includes selecting one of the plurality of parts according to the prediction direction to generate modified training data. The method also includes training the model using the modified training data.

[0007] According to the method of an embodiment of the present invention, the model can be trained to estimate the probability of starting or ending from the middle of a sequence, which will actually appear in the input of the model. This allows the model to have improved accuracy.

[0008] In a preferred embodiment, the trained model is used to re-score one or more input sequences that are obtained as hypotheses for the recognition process. Thus, the error rate of the re-scored results is expected to be reduced compared to the original results of the recognition process. Since the model is trained so that the probability of starting or ending in the middle of the sequence is estimated, it allows the model to appropriately perform re-scoring of the hypothesis even if the hypothesis includes errors. Negative effects caused by errors in the hypothesis can be mitigated.

[0009] In a further preferred embodiment, when rescoring each input sequence with the model, the history is discarded at least in part in response to satisfying a condition related to the confidence of the recognition process for an input element in the input sequence, thereby further reducing the error rate of the rescoring result compared to a rescoring result without discarding the history based on the confidence of the recognition process.

[0010] In a further preferred embodiment, the model has a hidden state that represents the history. The history is discarded by resetting the hidden state of the model to a state that has accepted a predetermined symbol. Rescoring is restarted in response to a word that meets the condition. If an element with questions about confidence is included in the hypothesis, the reset prevents such an element from collapsing the hidden state of the model. Such an uncertain element in the hypothesis can propagate a reasonable hidden state for subsequent elements, which suppresses the occurrence of sudden errors after the uncertain element in the hypothesis. Thus, the score of the hypothesis after the uncertain element can be calculated with a reasonable hidden state.

[0011] In a particular embodiment, the method further comprises adding a predetermined symbol at a boundary of a portion. The modified training data comprises the portion and the predetermined symbol. Depending on the prediction direction, the predetermined symbol may represent the beginning of a sequence or the end of a sequence. Alternatively, the predetermined symbol may represent a recognition error, and the model has an additional token corresponding to the predetermined symbol.

[0012] In a specific embodiment, the prediction direction is a forward direction, a later part of the plurality of parts is selected as a part, and a boundary at which a predetermined symbol is added is the beginning of the later part. Alternatively, the prediction direction is a backward direction, a previous part of the plurality of parts is selected as a part, and a boundary at which a predetermined symbol is added is the end of the previous part.

[0013] In a preferred embodiment, the model is a recursive neural network language model, the training sequence of elements is a training sentence including a plurality of words, the recognition process is speech recognition, and the model is used to re-score the results output from the speech recognition. Thus, it allows speech recognition to have improved performance.

[0014] According to other embodiments of the present invention, a computer-implemented method for rescoring an input sequence obtained as a hypothesis for a recognition process is provided. The method includes preparing a model for rescoring the hypothesis. The model is capable of retaining a history of one or more previous elements and has a rescoring direction. The method also includes feeding the input sequence into the model according to the direction. The method also includes at least partially discarding the history in response to satisfying a condition related to the confidence of the recognition process regarding an input element in the input sequence. The method also includes outputting a result of rescoring the input sequence.

[0015] According to the method of other embodiments of the present invention, the error rate of the result is expected to be reduced compared to the original result of the recognition process and the re-scoring process without discarding the history based on the confidence of the recognition process. Since the negative impact caused by errors in the hypothesis can be mitigated, it allows the model to appropriately perform re-scoring of the hypothesis even when the hypothesis includes errors.

[0016] In a preferred embodiment, the model has a hidden state that represents the history, and the history is discarded by resetting the hidden state of the model to a state where a predetermined symbol has been accepted. Rescoring is restarted in response to a word that satisfies a condition. If an element with questions about confidence is included in a hypothesis, the reset prevents such an element from collapsing the hidden state of the model. Such an uncertain element in a hypothesis can propagate a reasonable hidden state for subsequent elements, which suppresses the occurrence of sudden errors after the uncertain element in the hypothesis. Thus, the score of the hypothesis after the uncertain element can be calculated with a reasonable hidden state.

[0017] In a preferred embodiment, the model is constructed by the following steps: obtaining a training sequence of elements; segmenting the training sequence into a plurality of parts; selecting one of the plurality of parts to generate modified training data depending on the rescoring direction; and training the model using the modified training data. Since the model is trained so that the probability of estimating the start or end of the sequence is further mitigated, the negative impact caused by errors in the hypothesis can be further mitigated. When the hypothesis is rescored, the model can be made resistant to errors in the hypothesis.

[0018] Computer systems and computer program products related to one or more aspects of the present invention are also described and claimed herein.

[0019] Additional features and advantages are realized through the techniques of the present invention.Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features and advantages of the invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0021] Figure 1 A block diagram of a speech recognition system including a re-scoring module and a training data modification module according to an exemplary embodiment of the present invention is shown;

[0022] Figure 2 A schematic diagram of training data modification for training an RNN (Recurrent Neural Network) language model according to an exemplary embodiment of the present invention is described;

[0023] Figure 3 is a flow chart depicting a process for training an RNN language model for hypothesis rescoring according to an exemplary embodiment of the present invention;

[0024] Figure 4 A schematic diagram of a rescoring process with a reset operation according to an exemplary embodiment of the present invention is described;

[0025] Figure 5 A schematic diagram depicting normal operation in a re-scoring process when there are no uncertain terms in a given hypothesis according to an exemplary embodiment of the present invention;

[0026] Figure 6 A schematic diagram depicting a reset operation in a re-scoring process when there are uncertain words in a given hypothesis according to an exemplary embodiment of the present invention;

[0027] Figure 7 is a flowchart depicting a process for training a backward RNN language model for hypothesis rescoring according to other exemplary embodiments of the present invention;

[0028] Figure 8 A schematic diagram depicting a reset operation in a re-scoring process using a backward RNN language model according to other exemplary embodiments of the present invention; and

[0029] Fig. 9 A computer system in accordance with one or more embodiments of the invention is depicted. DETAILED DESCRIPTION

[0030] The present invention will be described below with reference to specific embodiments, but those skilled in the art will appreciate that the embodiments described below are mentioned only by way of example and are not intended to limit the scope of the present invention.

[0031] One or more embodiments according to the present invention relate to computer-implemented methods, computer systems, and computer program products for training a model, wherein the model is capable of retaining a history of previous elements in a sequence and can be used to re-score a primary hypothesis generated by a speech recognition process. One or more other embodiments according to the present invention relate to computer-implemented methods, computer systems, and computer program products for re-scoring hypotheses using a model, wherein the model for hypothesis re-scoring is capable of retaining a history of previous elements in a sequence and obtaining hypotheses to be re-scored from a speech recognition process.

[0032] In the following, we first refer to Figure 1 and Figure 2 , a computer system for training an RNN (recurrent neural network) language model and for rescoring primary hypotheses generated by a speech recognition process according to an exemplary embodiment of the present invention will be described. Then, with reference to Figure 3 , a computer-implemented method for training an RNN language model using training data modification according to an exemplary embodiment of the present invention will be described. Then, with reference to Figure 4 as well as Figure 5 and Figure 6 , a computer-implemented method for rescoring primary hypotheses using an RNN language model according to an exemplary embodiment of the present invention will be described. Then, further reference is made to Figure 7 and Figure 8 , other exemplary embodiments according to the present invention will be described, in which the model to be trained is a backward RNN language model.

[0033] In addition, an experimental study of speech recognition with novel training data modification and rescoring capabilities according to an exemplary embodiment of the present invention will be described. Fig. 9 , the hardware configuration of a computer system according to one or more embodiments of the present invention will be described.

[0034] refer to Figure 1 , a block diagram of a speech recognition system 100 according to an exemplary embodiment of the present invention is described. Figure 1 As shown, the speech recognition system 100 includes a decoder 110 that receives an input 102 and outputs a primary recognition hypothesis in the form of a lattice 104; an acoustic model 112 used by the decoder 110; and a weaker language model 114 used by the decoder 110; a rescoring module 120 that receives the lattice 104 and outputs a rescored lattice 106; and an RNN language model 130 used by the rescoring module 120.

[0035] The input 102 may be a speech signal digitized by sampling an analog audio input at a predetermined sampling frequency and a predetermined bit depth. Before entering the decoder 110, the speech signal may be converted into an acoustic feature sequence via an appropriate front end using any known acoustic feature analysis.

[0036] Decoder 110 uses acoustic model 112 and weaker language model 114 to predict plausible speech content of input 102 and outputs a preliminary recognition hypothesis as a lattice 104 .

[0037] The weaker language model 114 is a model for estimating the probability distribution over a sequence of words or sentences. The weaker language model 114 can predict subsequent words in a sentence based on previous words that have been observed in the sentence. The weaker language model 114 can be, but is not limited to, a unigram model, an n-gram model, a maximum entropy language model, or a skip-gram language model.

[0038] If resources are available without any restrictions, any state-of-the-art language model will be adopted. However, available resources are actually limited, and it is not efficient to apply stronger language models in the early stages of speech recognition. In a sense, the weaker language model 114 is a weaker model that includes fewer computing resources to generate hypotheses than a more complex language model (such as an RNN), and its modeling power is sufficient for the computing resources included.

[0039] The acoustic model 112 is a model for estimating the probability to generate an acoustic feature sequence when a sequence of words is given. The acoustic model 112 can be based on a GMM (Gaussian mixture model) / HMM (Hidden Markov Model) or NN (Neural Network) / HMM system, which uses NN instead of GMM in the GMM / HMM system, where HMM is used to process the temporal variability of speech and GMM is used to determine the degree of match of each state of each HMM with an input frame or a short window of an input frame. The neural network used in the NN / HMM system can be a DNN (Deep Neural Network), a CNN (Convolutional Neural Network) or an RNN-based architecture.

[0040] Decoder 110 outputs primary recognition hypothesis as lattice 104. Lattice 104 represents multiple decoded sentences. Lattice 104 may include a node list and an edge list. Each node in lattice 104 may represent a time point. Each edge connected between a pair of start nodes and end nodes may represent a segmented word that starts at the time of the start node and ends at the time of the end node. In the described embodiment, decoder 110 is configured to output multiple hypotheses in the form of a lattice. However, in other embodiments, decoder 110 may output hypotheses in the form of a prefix tree structure or a list of N best hypotheses. The representation of the hypothesis is not limited.

[0041] The lattice 104 may include confidence information calculated when generating the primary recognition hypothesis. Based on the acoustic model 112, the decoder 110 may calculate the acoustic score for each unit (such as a frame) and statistics (such as the average of the acoustic scores on multiple units corresponding to each word). Based on the weaker language model 114, the decoder 110 may calculate the language score for each word. In addition, the decoder 110 may calculate the total confidence score by integrating the acoustic score and the language score. A confidence score may be given to each word in the lattice 104, including an acoustic score, an average acoustic score, a language score, a total confidence score, or a combination thereof, and the confidence of the recognition of each word is represented by these confidence scores.

[0042] The rescoring module 120 is configured to rescore each sentence in the lattice 104 using the RNN language model 130 and output the rescored lattice 106. The RNN (Recurrent Neural Network) language model 130 can have any recursive architecture with hidden states, including basic recursive neural networks, LSTM (Long Short Term Memory), Elman networks, Jordan networks, Hopfield networks, GRU (Gated Recurrent Units), etc. RNN is a type of neural network that includes connections within a layer when compared to a feedforward network that includes connections between subsequent layers. The RNN language model 130 has a hidden state that represents an infinite history of previous words, and is able to retain the history of previous words in the hidden state. Note that the terms "previous" and "subsequent" represent the previous and subsequent according to the direction in which the sequence of words is fed to the RNN language model 130.

[0043] The RNN language model 130 has stronger modeling capabilities than the weaker language model 114. Since the hypotheses obtained from the primary speech recognition are subjected to hypothesis rescoring, a more complex and resource-consuming language model can be employed even if the available resources are actually limited.

[0044] The rescoring module 120 rescores each hypothesis in the original lattice 104 using the RNN language model 130 in order to find the best path through the lattice 104 or to reorder the paths in the original lattice 104. New language scores based on the RNN language model 130 are calculated to find the best path or to reorder the paths in conjunction with the acoustic scores given in the original lattice 104. The rescored lattice 106 represents a plurality of sentences with corresponding new scores. Using the rescored lattice 106 to select the best hypothesis or to reorder the hypotheses is optional and can be done if desired.

[0045] In the depicted embodiment, the raw lattice 104 is directly input into the rescoring module 120. However, in one or more embodiments, the raw lattice 104 may be subjected to other processing, such as lattice expansion, to obtain an expanded lattice, before or during rescoring.

[0046] like Figure 1 As shown, the speech recognition system 100 further includes a training module 140 , a training data modification module 150 and a training data storage 160 .

[0047] The training module 140 performs a training process using the given training data to build the RNN language model 130. In the described embodiment, the training data store 160 provides a collection of transcriptions. The transcriptions are transcribed by human experts. In specific cases such as domain adaptation, automatically transcribed texts may also be considered.

[0048] The RNN language model 130 can be trained by using an appropriate algorithm. The RNN can be trained by gradient descent methods, such as full or truncated BPTT (back propagation through time) and RTRL (real-time recursive learning) algorithms.

[0049] As described above, the training sentences stored in the training data store 160 are typically transcribed by human experts. There are few errors in human-transcribed text. On the other hand, the lattice 104 generated by the decoder 110 typically includes one or more errors at various word positions. Even the ASR systems in the prior art can make incorrect identifications. Such errors in the assumptions cause the hidden states of the RNN language model 130 to collapse during the rescoring process. Even the error trends in the automatically transcribed training texts are typically different from the error trends observed in the lattice 104. Therefore, the RNN language model 130 needs to be trained to allow for such errors in the assumptions.

[0050] To this end, the training data modification module 150 according to an exemplary embodiment is configured to modify the original training data to generate modified training data so as to allow the RNN language model 130 trained based on the modified training data to handle such errors in the lattice 104 .

[0051] More specifically, the training data modification module 150 is configured to obtain each training sentence from the training data storage 160 and segment the training sentence into a plurality of parts, each of the plurality of parts including one or more words.

[0052] Figure 2 A schematic diagram of training data modification for training the RNN language model 130 is described. Figure 2 As shown, the training sentence 200 is segmented into two parts 202 and 204 at a segmentation position 206 which may be determined randomly.

[0053] The training data modification module 150 is also configured to select one of the plurality of parts according to the rescoring direction of the RNN language model 130. In the described embodiment, the RNN language model 130 is trained so that it predicts the next word in a sentence based on the previous words given from the beginning of the sentence. Therefore, the rescoring direction of the RNN language model 130 is forward. In this case, the latter part 204 is selected from the plurality of parts 202, 204. Note that the terms "next" and "previous" mean the next and previous according to the sentence direction.

[0054] The training data modification module 150 is also configured to add a predetermined symbol at the boundary (or edge) of the selected portion. In the described embodiment, the boundary at which the predetermined symbol is added is the beginning of the latter portion 204, and the predetermined symbol represents the beginning of a sequence (BOS) or the beginning of a sentence (SOS). Therefore, the modified training data 210 includes the selected latter portion 204 and the predetermined symbol 208 added at its beginning. Note that the unselected former portion 202 can be discarded from the training data.

[0055] In the described embodiment, the rescoring direction for the RNN language model 130 is forward. However, in other embodiments, the RNN language model 130 may be trained so that the previous word in the sentence is predicted based on the next word given from the end of the sentence. In this case, the rescoring direction of the RNN language model 130 is backward. If the rescoring direction of the RNN language model 130 is backward, the previous part 202 is selected from the plurality of parts 202, 204, and the boundary at which the predetermined symbol is added is the end of the previous part 202. Furthermore, in this case, the predetermined symbol represents the end of a sequence (EOS).

[0056] In other embodiments, instead of adding a sentence start symbol or a sentence end symbol that is the same as the symbol indicating the correct start of a sentence and the correct end of a sentence, the predetermined symbol may be a special symbol (e.g., ASR_ERROR) indicating the occurrence of an error in speech recognition. In this embodiment, the RNN language model 130 is trained to have an additional mark corresponding to the special symbol (ASR_ERROR), which makes it possible to distinguish between the natural start or end of a sentence and the start or end of a sentence that is intentionally created.

[0057] As described above, the segmentation position is determined randomly. However, in other embodiments, the segmentation position may be determined statistically so that the training sentence is segmented at a position where a recognition error may occur, or the training sentence is not segmented at a position where a recognition error is unlikely to occur.

[0058] Note that the training data modification module 150 does not modify all the training sentences stored in the training data storage 160. The training sentences to be divided may be randomly determined at a predetermined ratio from the set of training sentences in the training data storage 160. For example, only one sentence among a predetermined number of sentences (ratio in the range of about 10% to 50%) may be subjected to training data modification.

[0059] In the described embodiment, it is described that one training sentence is selected to generate one modified training sentence. However, in other embodiments, the same training sentence may be selected multiple times to generate multiple modified training sentences with different segmentation positions. In this case, it can be said that the training data is being enhanced. In addition, the modified training sentence 210, which is half of the original sentence and obtained through the previous training data modification, can be further subjected to subsequent training data modification.

[0060] The training module 140 is configured to train the RNN language model 130 using the modified training data, typically together with the original training sentences without data modification stored in the training data store 160. In the described embodiment, the original training data that has undergone the training data modification is no longer used for training. However, in other embodiments, the original training data that has undergone the training data modification can still be used as training data.

[0061] As described above, the RNN language model 130 trained by the training module 140 can be used for the hypothesis rescoring performed by the rescoring module 120. Due to the training data modification, the RNN language model 130 can be trained so that the probability of starting from the middle of a sentence or ending in the middle of a sentence is estimated. Unexpectedly, this helps to mitigate the negative effects caused by errors observed in the hypothesis. It should also be noted that in other embodiments, the RNN language model 130 trained by the training module 140 can be used in the decoding process performed by the decoder 110.

[0062] The aforementioned training data modifications contribute to improvements in the performance of the speech recognition system 100. Furthermore, by changing the logic of the re-scoring process, further improvements in the performance of the speech recognition system 100 can be expected.

[0063] The rescoring module 120 rescores each input sentence in the original lattice 104. In a preferred embodiment, the rescoring module 120 is also configured to at least partially discard the history retained in the RNN language model 130 when a predetermined condition is met. The predetermined condition is a condition related to the confidence of the speech recognition of a word in the input sentence, and is used to capture uncertain words that may have errors. In the described embodiment, the predetermined condition includes a threshold value (e.g., average acoustic score, language score, total score) of the confidence information given for the words in the input sentence. The threshold value can be determined empirically or statistically. In other embodiments, more complex conditions can be considered. For example, other neural network models trained to determine the timing of discarding history can be used to determine whether the predetermined condition is met.

[0064] Since the history is represented by the hidden state of the RNN language model 130, the history can be discarded by resetting the hidden state of the RNN language model 130 to a state that has accepted the predetermined symbol, which is the beginning of the sentence if the direction of the model is forward ( <bos>), if the direction of the model is backward, the predetermined symbol is the end of the sentence ( <eos>), or when used, the predetermined symbol is a special symbol (<ASR_ERROR> ). After the word meets the predetermined conditions, the re-scoring can be restarted.

[0065] In certain embodiments, Figure 1 Each of the modules 110, 120, 140 and 150 described in the drawings may be implemented as, but not limited to, a software module including instructions and / or data structures in conjunction with hardware components such as a processor, memory, etc.; a hardware module including electronic circuitry; or a combination thereof. Figure 1 These modules 110, 120, 140 and 150 described in the can be implemented on a single computer system such as a personal computer, a server machine, or implemented in a distributed manner on multiple devices such as a client server system, an edge computing system, a computer cluster.

[0066] References below Figure 3 The flowchart shown describes the process of training the RNN language model using training data modification according to an exemplary embodiment of the present invention.

[0067] Notice, Figure 3 The process shown can be performed by the training module 140 using the training data modification module 150, that is, implementing Figure 1 The processing units of the training module 140 and the training data modification module 150 are shown. In addition, Figure 3 The process shown is for training the RNN language model 130 so that it predicts forward. Therefore, the direction of rescoring the RNN language model 130 to be trained is forward.

[0068] Figure 3 The process shown may start at step S100 in response to receiving a request from an operator to train the RNN language model 130. Note that a new RNN model (eg, randomly initialized) or a pre-trained model may be given.

[0069] In step S101, the processing unit may read training sentences from the training data storage 160. A set of training sentences to be used for training may be specified in the request.

[0070] In step S102, the processing unit may determine whether the training sentence read from the training data storage 160 is subject to training data modification. For example, only one of the several sentences may be subject to training data modification. The predetermined rate may be specified in the request. If, in step S102, the processing unit determines that the current training sentence is subject to training data modification, processing may proceed to step S103.

[0071] In step S103 , the processing unit may randomly determine the segmentation position in the training sentence.

[0072] In step S104, the processing unit may segment the training sequence into multiple parts at the segmentation position determined in step S103, and select one of the multiple parts according to the direction of the RNN language model 130. Figure 3 In the process shown, the direction of the RNN language model 130 is forward, so the latter part is selected.

[0073] In step S105, the processing unit may add a sentence start symbol 208 at the beginning of the latter part 204 to generate a sentence as shown in the reference. Figure 2 Modified training data 210 is depicted. The modified training data 210 includes the selected rear portion 204 and the predetermined symbol 208.

[0074] If the processing unit determines at step S102 that the current training sentence has not been subjected to training data modification, the process may proceed directly to step S106.

[0075] In step S106, the processing unit may perform an iteration of training the RNN language model 130 using the modified training data or the original training data. Note that the iteration of training may be performed for each training sentence (e.g., online training) or a predetermined set of training sentences (e.g., batch training, mini-batch training).

[0076] At step S107, the processing unit may determine whether an exit condition is satisfied. For example, when a predetermined number of epochs have been completed, the exit condition is determined to be satisfied. If at step S107 the processing unit determines that the exit condition has not been satisfied, the process may loop back to step S101 for further iterations. On the other hand, if at step S107 the processing unit determines that the exit condition has been satisfied, the process may proceed to step S108, and the process may end at step S108.

[0077] according to Figure 3 The process of training the RNN language model using training data modification shown can train the RNN language model 130 so that the probabilities are estimated starting from the middle of the sequence. Unexpectedly, it helps to mitigate the negative impact caused by the observed errors in the hypothesis, making the model more resistant to the errors in the hypothesis when the hypothesis is rescored. Therefore, the error rate of the rescored lattice 106 is expected to be reduced compared to the original lattice 104.

[0078] The following will combine Figure 4 and Figure 5 , Figure 6 The rescoring process with a reset operation according to an exemplary embodiment of the present invention is further described. Note that referring to a series of Figure 4-Figure 6 The described rescoring process may be performed by the rescoring module 120 having the RNN language model 130, that is, implementing Figure 1 The processing units of the rescoring module 120 and the RNN language model 130 are shown. Also note that the rescoring direction of the RNN language model 130 is the forward direction.

[0079] First, for example, the rescoring module 120 may prepare the RNN language model 130 to rescore the hypotheses on the memory. Then, the lattice 104 is read by the rescoring module 120. Figure 4 As shown, the dot matrix 104 includes a plurality of nodes 104a and a plurality of edges 104b connected between pairs of nodes (in the figure, only one node and one edge number are representatively given). Each node 104a in the dot matrix 104 can represent a time point. Each edge 104b connected between a pair of start nodes and end nodes can represent a word spanning a segment starting at the time of the start node and ending at the time of the end node. Although multiple paths can be drawn in the dot matrix 104, here we focus on an input sentence 104c to describe the operation of the re-scoring process.

[0080] The rescoring module 120 continuously feeds each word in the input sentence 104c into the RNN language model 130 according to the forward direction in the described embodiment. <bos>,V,W,X,Y,Z, <eos>} are fed to the RNN language model 130 in this order. The rescoring module 120 receives the language model probabilities from the RNN language model 130 to obtain new language scores.

[0081] Figure 5 A schematic diagram depicting normal operation during the rescoring process. When there are no uncertain words in the hypothesis, each time a word is fed into the RNN language model 130, the history of the previous (previous) word is accumulated in the hidden state 130a of the RNN language model 130. The hidden state can affect the calculated probability of the subsequent (next) word.

[0082] Return to reference Figure 4 , when rescoring the input sentence, in response to satisfying a predetermined condition, the rescoring module 120 according to an exemplary embodiment can reset the hidden state 130a of the RNN language model 130 to a state in which a predetermined symbol (in the embodiment, the beginning of the sentence) has been accepted. The predetermined condition is a condition related to the confidence of the recognition of an input word in the input sentence 104c. The predetermined condition is used to capture uncertain words that may have errors. In the described embodiment, the predetermined condition includes a threshold value of the confidence score (e.g., average acoustic score, language score, total score) given to the words in the input sentence. When a word with a confidence score below the threshold is found, the rescoring module 120 resets the hidden state 130a of the RNN language model 130.

[0083] Figure 6 A schematic diagram of a reset operation in a rescoring process is described. When an uncertain word (X) is assumed, the history of the previous (previous) word accumulated in the hidden state of the RNN language model 130 is discarded at a timing after the uncertain word (X) is fed. Rescoring is restarted after the word that satisfies the predetermined condition (X), just as if the latter part of the input sentence starts from the position of the uncertain word (X). The former part of the input sentence including the uncertain word (X) can be retained. Therefore, the input sequence { <bos>,V,W,X,Y,Z, <eos>} is calculated as the score of the previous part { <bos>, V, W, X} and the latter part { <bos>,Y,Z, <eos>As a result, the occurrence of sudden errors can be prevented.

[0084] according to Figure 4-Figure 6 The illustrated process of rescoring the input sentence using the reset operation is expected to reduce the error rate of the rescored lattice 106 compared to the original lattice 104 of speech recognition and the rescored lattice without discarding the history based on the confidence of speech recognition. Since the negative impact caused by errors in the hypothesis can be mitigated, it allows the RNN language model 130 to appropriately perform rescoring of the hypothesis even if the hypothesis includes errors.

[0085] If a hypothesis includes words with questions about confidence, resetting prevents such words from corrupting the hidden state of the RNN language model 130. Such uncertain elements in the hypothesis can propagate reasonable hidden states of subsequent words, which suppresses the occurrence of sudden errors after the uncertain words in the hypothesis. Thus, the score of the hypothesis after the uncertain element can be calculated with a reasonable hidden state.

[0086] In addition, when the RNN language model 130 trained by the training module 140 and the training data modification module 150 is used, the RNN language model 130 is trained to estimate the probability of starting or ending from the middle of the sequence. The negative impact caused by errors in the assumptions can be further mitigated. When the assumptions are rescored, the RNN language model 130 can be made more resistant to errors in the assumptions. Combined with the characteristics of the RNN language model 130, the error rate of the rescored lattice 104 is further reduced.

[0087] In the above embodiment, the direction of the RNN language model 130 is described as the forward direction. Figure 7 The flowchart shown describes a process for training a backward RNN language model using training data modification according to other exemplary embodiments of the present invention.

[0088] Notice, Figure 7 The process shown can be performed by the training module 140 and the training data modification module 150, that is, implementing Figure 1 The processing units of the training module 140 and the training data modification module 150 are shown. Figure 7 The illustrated process may begin at step S200 in response to receiving a request from an operator to train the RNN language model 130 .

[0089] In step S201, the processing unit may read a training sentence from the training data storage 160. In step S202, the processing unit may randomly determine a segmentation position in the training sentence. In step S203, the processing unit may segment the training sequence into a plurality of parts at the segmentation position determined in step S202, and select one of the plurality of parts according to the direction of the RNN language model 130. Figure 7 In the process shown, the direction of the RNN language model 130 is the backward direction, so the first part is selected.

[0090] In step S204, the processing unit may add a sentence end symbol at the end of the previous part. <eos>, to generate modified training data. The modified training data includes the selected previous part and a predetermined symbol added at the end thereof <eos>Note that the unselected latter part may be discarded from the training data. In step S205, the processing unit may store the former part having the predetermined symbol in the training data storage 160 as modified training data.

[0091] At step S206, the processing unit may determine whether an exit condition is satisfied. For example, if a predetermined amount of training data has been processed, then the exit condition is determined to be satisfied. If at step S206 the processing unit determines that the exit condition is not satisfied, then the processing may loop back to step S201 for further training data modification. On the other hand, if the processing unit determines at step S206 that the exit condition has been satisfied, then the processing may proceed to step S207.

[0092] At step S207, the processing unit may perform iterations of training the RNN language model 130 using the modified training data and the original training data stored in the training data store 160. Note that iterations of training may be performed for each training sentence or a set of training sentences. For example, a predetermined number of patterns may be repeatedly performed. And the process may end at step S208.

[0093] Please refer to the following Figure 4 and Figure 8 , further describing the reset process with reset operation according to other embodiments of the present invention. Note that the reset process can be implemented by Figure 1 The processing units of the re-scoring module 120 and the RNN language model 130 are used to perform the combined Figure 4 and Figure 8 It is also noted that the rescoring direction of the RNN language model 130 according to other exemplary embodiments is the backward direction.

[0094] First, for example, the rescoring module 120 may prepare the backward RNN language model 130 on a memory. The lattice 104 is then read by the rescoring module 120. The rescoring module 120 feeds each word in the input sentence 104c into the backward RNN language model 130 in succession according to the backward direction in the described embodiment. Thus, the word sequence { <bos>,V,W,X,Y,Z, <eos>} are fed to the RNN language model 130 in reverse order, i.e., { <eos>,Z,Y,X,W,V, <bos>}.

[0095] When there is no uncertain word in the hypothesis, each time a word is input into the backward RNN language model 130, the history of the previous (next) word is accumulated in the hidden state 130a of the RNN language model 130. The hidden state can affect the calculated probability of the subsequent (previous) word.

[0096] When rescoring the input sentence, in response to a predetermined condition being met, the rescoring module 120 may reset the hidden state 130a of the RNN language model 130 to a state that has accepted a predetermined symbol (the end of the sentence in the described embodiment). When a word with a confidence score below a threshold is found, the rescoring module 120 resets the hidden state 130a of the RNN language model 130.

[0097] Figure 8 A schematic diagram of a reset operation in a rescoring process using a reverse RNN language model is described. When there is an uncertain word (X) in the hypothesis, the history of the previous (next) word accumulated in the hidden state of the RNN language model 130 is discarded at a timing after the uncertain word (X) is fed. Rescoring is restarted after the word that satisfies the predetermined condition (X), just as the previous part of the input sentence ends at the position of the uncertain word (X). The latter part of the input sentence before the position of the uncertain word (X) can be retained.

[0098] According to the foregoing embodiments of the present invention, a technique is provided that can train a model for hypothesis rescoring and mitigate the negative impact caused by errors in hypotheses. Also according to the foregoing embodiments of the present invention, a technique is provided that can rescore hypotheses with improved performance.

[0099] Languages ​​to which the novel training data modifications and / or novel rescoring procedures may be applied include, but are by no means limited to, for example, Arabic, Chinese, English, French, German, Japanese, Korean, Portuguese, Russian, and Spanish.

[0100] Furthermore, in the aforementioned embodiment, the RNN language model 130 is used for hypothesis rescoring as a model capable of retaining a history of one or more previous elements or words. RNN is preferred because the hidden state of the RNN language model can represent an infinite history of previous elements or words. However, other models such as n-gram language models that are also capable of retaining a history of a predetermined number of previous elements can also be envisioned.

[0101] Furthermore, in the aforementioned embodiments, novel training data modifications and novel rescoring processes are applied to speech recognition. However, such negative effects caused by rescoring hypotheses due to errors in the hypotheses may also occur in various recognition tasks other than speech recognition. In one or more embodiments, the recognition processes to which the novel training data modifications and / or novel scoring processes are applied may include machine translation, part-of-speech tagging, parsing, optical character recognition, handwriting recognition, image captioning, or video captioning.

[0102] Experimental studies

[0103] Implementation of the exemplary embodiment Figure 1 The system shown, Figure 3 The training process and Figure 4 The program for the rescoring process shown was coded and executed.

[0104] The LSTM acoustic model and the larger n-gram language model are used as the acoustic model 112 and the weak language model 114 for primary decoding, respectively. An ASR system including the LSTM acoustic model and the larger n-gram language model is used as Comparative Example 1 without lattice rescoring.

[0105] The LSTM language model is used as the RNN language model 130 for hypothesis rescoring. The telephone speech corpus SWITCHBOARD used for research and development was used, including 262 hours of Switchboard 1 audio with transcriptions provided by Mississippi State University and 1698 hours in the Fisher data set, and 15 hours of CallHome audio. Almost 2000 hours of audio data transcriptions were used to train the LSMT language model. For Examples 1 and 3, the LSTM language model was trained with a training data modification function. Only 30% of the training sentences were subjected to training data modification. On the other hand, for Example 2 and Comparative Example 2, the LSTM language model was trained without training data modification.

[0106] For Example 1 and Comparative Example 2, standard rescoring without resetting was performed in speech recognition. On the other hand, for Examples 2 and 3, novel rescoring with resetting was performed in speech recognition. The average acoustic score calculated for the duration corresponding to each word was used as a confidence score, and three standard deviations on the lower side of the average (mean-3σ (sigma)) were used as a threshold for the confidence score. The average and standard deviation were estimated by 100 best hypotheses for each lattice 104.

[0107] Then, the ASR accuracy of the constructed speech recognition systems of Examples 1-3 and Comparative Examples 1 and 2 was evaluated using the validation data in the SWITCHBOARD dataset. WER (Word Error Rate) was used as the ASR accuracy metric. The WER of the reference ASR system was 7.2%, which means that the reference ASR system had strong modeling capabilities even before lattice rescoring (Comparative Example 1).

[0108] The evaluation results of Examples 1-3 and Comparative Examples 1 and 2 are summarized as follows:

[0109]

[0110] The symbol "*" indicates that the function specified by the column is applied.

[0111] By comparing with the results of Comparative Example 2, Examples 1 and 3 show improved performance. Even though Example 2 also shows comparable performance to the standard rescoring (Comparative Example 2), this also means that the reset operation hardly hurts the performance. Among Examples 1-3, Example 3, which combines the novel training modification in the training phase and the novel lattice rescoring in the inference phase, shows the best performance. Examples 1 and 3 outperform the strong baseline of LSTM standard rescoring (Comparative Example 2).

[0112] Computer hardware components

[0113] refer to Fig. 9 , which shows a schematic diagram of an example of a computer system 10 that can be used to implement the speech recognition system 100. Fig. 9 The computer system 10 shown is implemented as a computer system. The computer system 10 is only one example of a suitable processing device and is not intended to impose any limitations on the scope of use or functionality of the embodiments of the present invention described herein. Regardless, the computer system 10 can be implemented and / or perform any of the functions set forth above.

[0114] The computer system 10 may operate with numerous other general or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with the computer system 10 include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, vehicle-mounted devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems or devices, etc.

[0115] Computer system 10 may be described in the general context of computer system executable instructions, such as program modules, executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types.

[0116] like Fig. 9 As shown, computer system 10 is shown in the form of a general-purpose computing device. Components of computer system 10 may include, but are not limited to, a processor (or processing unit) 12 and a memory 16 coupled to processor 12 via a bus, including a memory bus or memory controller, and a processor or local bus using any of a variety of bus architectures.

[0117] Computer system 10 includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer system 10, and it includes volatile and non-volatile media, removable and non-removable media.

[0118] The memory 16 may include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM). The computer system 10 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 18 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium. As will be further depicted and described below, the storage system 18 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of an embodiment of the present invention. In some embodiments, the RNN language model 28 may be stored on the storage system 18, and the computer 10 may train the RNN language model 28 stored on the storage system 18.

[0119] A program / utility having a set (at least one) of program modules may be stored in storage system 18 and, for example but not limited to, an operating system, one or more application programs, other program modules, and program data. Each or a combination of the operating system, one or more application programs, other program modules, and program data may include an implementation of a networking environment. The program modules generally perform the functions and / or methods of embodiments of the present invention as described herein.

[0120] The computer system 10 may also communicate with one or more peripheral devices 24, such as a keyboard, a pointing device, a car navigation system, an audio system, etc.; a display 26; one or more devices that enable a user to interact with the computer system 10; and / or any device that enables the computer system 10 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur via an input / output (I / O) interface 22. However, the computer system 10 may communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 20. As described, the network adapter 20 communicates with other components of the computer system 10 via a bus. It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with the computer system 10, examples of which include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems. In some embodiments, the computer 10 may communicate with the RNN language model 30 using the network adapter 20, and the computer 10 may train the RNN language model 30 using the network adapter 20. The RNN language model 30 may be implemented as one or more nodes.

[0121] Computer program implementation

[0122] The present invention may be a computer system, method and / or computer program product. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the present invention.

[0123] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as a temporary signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0124] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0125] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as "C" programming language or similar programming languages). The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or all on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In certain embodiments, in order to perform various aspects of the present invention, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can execute the computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions.

[0126] Various aspects of the present invention are described herein with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to embodiments of the present invention. It will be understood that each box in the flow chart and / or block diagram and the combination of boxes in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0127] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0128] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0129] Flowchart and block diagram in the accompanying drawings show the possible implementation framework, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each frame in flow chart or block diagram can represent the module, segmentation or part of instruction, and it comprises one or more executable instructions for implementing the specified logical function.In some alternative embodiments, the function mentioned in the frame may not occur in the order mentioned in the figure.For example, two frames shown continuously can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order, depending on the function involved.It will also be noted that the combination of the frame in each frame of block diagram and / or flow chart illustration and block diagram and / or flow chart illustration can be implemented by the dedicated hardware-based system that performs the specified function or action or performs the combination of special hardware and computer instruction.

[0130] The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the present invention. As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0131] If present, the corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the appended claims are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of one or more aspects of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the invention to the form disclosed.

[0132] Without departing from the scope and spirit of the described embodiments, many modifications and variations will be apparent to those of ordinary skill in the art. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements existing in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.< / bos> < / eos> < / eos> < / bos> < / eos> < / eos> < / eos> < / bos> < / bos> < / eos> < / bos> < / eos> < / bos> < / eos> < / bos>

Claims

1. A computer-implemented method for training a model, the method comprising: obtaining a training sequence of elements for the model, the model being capable of retaining a history of one or more previous elements and having a selectable prediction direction; Splitting the training sequence into a plurality of parts; selecting one of the plurality of portions according to the selectable prediction direction to generate modified training data; as well as training the model using the modified training data, wherein selecting one of the multiple parts according to the selectable prediction direction comprises: when the prediction direction is selected as a forward direction, a latter part of the multiple parts is selected as the one part, and when the prediction direction is selected as a backward direction, a former part of the multiple parts is selected as the one part, and Wherein, the training sequence of the elements includes training sentences.

2. The method of claim 1, wherein: The model is used to rescore one or more input sequences that are obtained as hypotheses for the recognition process.

3. The method of claim 2, wherein: The model includes a recursive neural network language model, the training sentence includes a plurality of words, the recognition process includes speech recognition, and a result output from the speech recognition is rescored as a hypothesis using the model.

4. The method according to claim 2, wherein: The recognition process is selected from the group consisting of speech recognition, machine translation, part-of-speech tagging, parsing, optical character recognition, handwriting recognition, image captioning, and video captioning.

5. The method of claim 2, wherein: In rescoring each input sequence with the model, the history is discarded at least in part in response to satisfying a condition related to a confidence level of the recognition process with respect to an input element in the input sequence.

6. The method of claim 5, wherein: The model has a hidden state representing the history, the history is discarded by resetting the hidden state of the model to a state where a predetermined symbol has been accepted, and the rescoring is restarted in response to a word satisfying the condition.

7. The method according to claim 1, wherein: The method further comprises: A predetermined symbol is added at a boundary of the one portion, and the modified training data includes the one portion and the predetermined symbol.

8. The method according to claim 7, wherein: The predetermined symbol indicates the start of a sequence or the end of a sequence according to the prediction direction.

9. The method of claim 7, wherein: The predetermined symbol indicates occurrence of a recognition error, and the model has an additional mark corresponding to the predetermined symbol.

10. The method of claim 7, wherein: The prediction direction is a forward direction, a latter part of the plurality of parts is selected as the one part, and the boundary is the beginning of the latter part.

11. The method of claim 7, wherein: The prediction direction is a backward direction, a previous part of the plurality of parts is selected as the one part, and the boundary is an end of the previous part.

12. The method according to claim 1, wherein: The training sequence is divided at random or statistically determined positions.

13. A computer-implemented method for rescoring an input sequence obtained as a hypothesis for a recognition process, the method comprising: preparing a model for rescoring the hypothesis, the model being capable of retaining a history of one or more prior elements and having a selectable rescoring direction; feeding the input sequence into the model according to the selectable rescoring direction; In response to satisfying a condition related to a confidence level of the recognition process regarding an input element in the input sequence, at least partially discarding the history; as well as outputting the result of rescoring the input sequence, wherein feeding the input sequence into the model according to the selectable rescoring direction comprises: when the selectable rescoring direction is selected as a forward direction, the feeding direction of the input sequence is opposite to the feeding direction of the input sequence when the selectable rescoring direction is selected as a backward direction, and Wherein, the input sequence includes a word sequence.

14. The method of claim 13, wherein: The model has a hidden state representing the history, the history is discarded by resetting the hidden state of the model to a state where the predetermined symbol has been accepted, and the rescoring is restarted in response to a word satisfying the condition.

15. The method according to claim 13, wherein: The model is constructed by the following steps: Obtaining a training sequence of the element; Splitting the training sequence into a plurality of parts; selecting one of the plurality of parts according to a rescoring direction to generate modified training data; as well as The model is trained using the modified training data.

16. The method of claim 13, wherein: The model has additional markings that represent errors in the recognition process.

17. A computer system for training a model by executing program instructions, the computer system comprising: Memory for storing program instructions; a processing unit in communication with the memory for executing the program instructions, wherein the processing unit is configured to: obtaining a training sequence of elements of the model, the model being capable of retaining a history of one or more previous elements and having a selectable prediction direction; Splitting the training sequence into a plurality of parts; selecting one of the plurality of portions according to the selectable prediction direction to generate modified training data; as well as training the model using the modified training data, wherein selecting one of the multiple parts according to the selectable prediction direction comprises: when the prediction direction is selected as a forward direction, a latter part of the multiple parts is selected as the one part, and when the prediction direction is selected as a backward direction, a former part of the multiple parts is selected as the one part, and Wherein, the training sequence includes training sentences.

18. The computer system of claim 17, wherein: The model is used to rescore one or more input sequences that are obtained as hypotheses for the recognition process.

19. The computer system of claim 17, wherein: The processing unit is configured to: A predetermined symbol is added at a boundary of the one portion, wherein the modified training data includes the one portion and the predetermined symbol.

20. A computer system for rescoring an input sequence obtained as a hypothesis of a recognition process by executing program instructions, the computer system comprising: A memory storing the program instructions; a processing unit in communication with the memory for executing the program instructions, wherein the processing unit is configured to: preparing a model for rescoring the hypothesis, wherein the model is capable of retaining a history of one or more prior elements and having a selectable rescoring direction; feeding the input sequence into the model according to the selectable re-scoring direction; In response to a condition being satisfied relating to a confidence level of the recognition process with respect to an input element in the input sequence, at least partially discarding the history; and outputting the result of rescoring the input sequence, wherein feeding the input sequence into the model according to the selectable rescoring direction comprises: when the selectable rescoring direction is selected as a forward direction, the feeding direction of the input sequence is opposite to the feeding direction of the input sequence when the selectable rescoring direction is selected as a backward direction, and Wherein, the input sequence includes a word sequence.

21. The computer system of claim 20, wherein: The model has a hidden state representing the history, the history is discarded by resetting the hidden state of the model to a state where the predetermined symbol has been accepted, and the rescoring is restarted in response to a word satisfying the condition.

22. A computer program product for training a model, the computer program product comprising a computer-readable storage medium having program instructions included therein, the program instructions being executable by a computer to cause the computer to perform a computer-implemented method comprising: obtaining a training sequence of elements of the model, the model being capable of retaining a history of one or more previous elements and having a selectable prediction direction; Splitting the training sequence into a plurality of parts; selecting one of the plurality of portions according to the selectable prediction direction to generate modified training data; as well as training the model using the modified training data, wherein selecting one of the multiple parts according to the selectable prediction direction comprises: when the prediction direction is selected as a forward direction, a latter part of the multiple parts is selected as the one part, and when the prediction direction is selected as a backward direction, a former part of the multiple parts is selected as the one part, and Wherein, the training sequence includes training sentences.

23. The computer program product of claim 22, wherein: The model is used to rescore one or more input sequences that are obtained as hypotheses for the recognition process.

24. The computer program product of claim 22, wherein the method further comprises: A predetermined symbol is added at a boundary of the one portion, and the modified training data includes the one portion and the predetermined symbol.

25. A computer program product for rescoring an input sequence obtained from a recognition process, the computer program product comprising a computer-readable storage medium having program instructions included therein, the program instructions being executable by a computer to cause the computer to perform the method of claim 13.

Images