Language model learning device, language model learning method, program

Abstract

This language model training device includes: a word order reversal unit that acquires a complete token string used for FLM training and reverses the word order of the complete token string; a degenerate token string generation unit that generates a degenerate token string obtained by deleting, I by I (I is a natural number), tokens disposed next to a special symbol <eos> representing the end of a sentence out of the complete token string the word order of which is reversed; and an ISF-only BLM language model training unit that trains an ISF-only BLM language model on the basis of the degenerate token string.

Description

[Technical Field]

[0001] This disclosure relates to a language model learning device, a language model learning method, and a program. [Background technology]

[0002] Speech recognition is a technology that converts human-spoken speech into token sequences (word sequences / subword sequences / strings, etc.) using a computer. The introduction of neural network (NN) technology has dramatically improved the accuracy of speech recognition. Conventional speech recognition systems consist of modules such as an acoustic analysis unit, an acoustic model unit, a language model unit, and a search unit, and each of these modules (mainly the acoustic model unit and the language model unit) had NN-based modeling introduced individually. On the other hand, in recent years, end-to-end (E2E) speech recognition technology, which models the speech recognition system with a single NN, has been actively researched, and there have been reports of achieving speech recognition accuracy that exceeds human capabilities in telephone conversation speech recognition tasks. E2E speech recognition systems use a single NN, but internally, modules such as an encoder unit and a decoder unit are implemented (see, for example, Non-Patent Document 1).

[0003] End-to-end (E2E) speech recognition systems (models) are trained to directly capture the conversion from speech to token sequences using paired data of speech and its corresponding token sequences (text). While it is possible to train highly accurate E2E speech recognition models using a large amount of paired data, there are limits to the amount of paired data that can actually be prepared. This is because transcribing the token sequences corresponding to speech is usually done manually, which is costly in terms of both time and money.

[0004] On the other hand, text data alone can be obtained more easily and in larger quantities compared to paired data. Therefore, a method has been proposed to integrate a language model (a model that calculates the probability of occurrence of token sequences) trained using a large amount of text data prepared separately from paired data with an E2E speech recognition model as external knowledge (an external language model). The most commonly used integration method is shallow fusion (SF). SF is a simple method that weights and adds the score (log probability) output by the main E2E speech recognition model and the score (also log probability) output by the external language model during the search process, yet it has been confirmed to be highly effective (see, for example, Non-Patent Documents 1 and 2).

[0005] Since SF is executed sequentially according to the extension of the speech recognition hypothesis from the beginning of the sentence in beam search, a standard forward language model (FLM) is used as the external language model. In the following, "hypothesis" and "token sequence" are used interchangeably. Beam search is a process that simultaneously extends and expands multiple hypotheses while keeping the hypothesis most likely to yield the highest score for the input speech and pruning the hypotheses with low probability.

[0006] On the other hand, there is a method called speech recognition hypothesis rescoring. Rescoring is a method of obtaining the final speech recognition result (1 best hypothesis) by re-scoring (re-evaluating the hypotheses) multiple hypotheses obtained in the form of an N best list or lattice using an external rescoring model (mainly a language model). In this method, backward language models (BLMs) are often used in addition to FLMs, and their effectiveness has been confirmed (see, for example, Non-Patent Document 3). A backward language model (BLM) is a language model trained using reversed text data (reverse text data) of the text data with the normal word order used to train FLMs, and it is a model that calculates the probability of a hypothesis occurring in the reverse word order from the end of the sentence to the beginning of the sentence.

[0007] The reason BLM is used in rescoring is that it can evaluate the probability of a hypothesis occurring from a different perspective than FLM, and the two models are complementary to each other. By using both models together, it is possible to assign more accurate language scores to hypotheses. Another reason is that rescoring is performed on hypotheses for which the search process is complete, that is, when the token sequence from the beginning to the end of the sentence (or from the end to the beginning of the sentence) is completely obtained (referred to as a "complete hypothesis" in this specification), making it easy to apply BLM, which uses the end of the sentence as the starting point for score calculation. Conversely, it is difficult to apply BLM to hypotheses for which the end of the sentence is not determined during the search process (referred to as a "partial hypothesis" in contrast to a complete hypothesis; also, when simply written as "hypothesis," it refers to a "partial hypothesis"). Note that FLM uses the beginning of the sentence as the starting point for score calculation, so it can be easily applied to partial hypotheses as well.

[0008] Figure 1 shows an overview of the speech recognition process in the E2E speech recognition model. In this specification, the attention-based encoder-decoder (AED) model is given as an example of the E2E speech recognition model, and the label (token) synchronized beam search is given as an example of the search algorithm (see, for example, Non-Patent Document 4 for these). However, this disclosure is also applicable to other types of E2E speech recognition models and search algorithms. As shown in Figure 1, the input speech is first input to the encoder in the E2E speech recognition model and converted into a hidden state vector sequence X. Here, it is rare for the speech waveform to be input directly as the input speech; it is more common for a filter bank feature sequence extracted from the speech waveform to be input as the input speech.

[0009] Next, X is input to the decoder in the E2E speech recognition model, and a complete hypothesis of length T that satisfies the following equation (1) { <sos> ,w^ 1:T , <eos>}={ <sos>,w1,w2,...,w T , <eos>The result is obtained through the search process and is output as the final speech recognition result (1 best perfect hypothesis). Note that in the text of the specification, "w^" is originally intended to be written directly above "w", but due to limitations in the functionality of the word processing software, it is not possible to write it directly above, so for convenience it is written as "w^".

number

[0010] Figure 2 shows an overview of the speech recognition process in an E2E speech recognition model, specifically the case where a conventional SF uses FLM as the external language model. The process from the encoder to obtaining X from the input speech is the same as in Figure 1, but the search process in the decoder differs from Figure 1 in that, in addition to the decoder score (see equation (1)), the language score calculated by the FLM (FLM score) is also considered, as shown in equation (2) below.

number

[0011] Figure 3 shows the details of the search process with SF introduced in the decoder of Figure 2, where at time t, a hypothesis of length t-1 { <sos>,w 1:t-1 } contains token w t Concatenate the length t { <sos>,w 1:t The score calculation when extending to} is shown. In this specification, one token extension of the hypothesis is considered to be one time step. The premise is that given X, the hypothesis { <sos>,w 1:t-1 Assume that the score s1 of} has already been calculated at the previous time t-1. Note that at t=0, the score s1 is initialized to 0. <sos>,w 1:t-1 } to w t When concatenating X and { <sos>, w 1:t-1 when w is given t the decoder score s2 of, { <sos>,w 1:t-1 When} is given, w t The FLM score s3 is calculated. The above three scores are weighted and added together as shown in equation (3) below, and given X, { <sos>,w 1:t Get a score of s4 for}.

number

number

[0012] The above details the score calculation when extending a single hypothesis by linking one type of token. However, in reality, the number of token types that can be linked to a single hypothesis is equal to the total number of types, i.e., the vocabulary size. Below, we will denote the vocabulary size as V. Note that the vocabulary includes: <eos>It is included, <sos>It shall not be included.

[0013] Figure 4 shows the search process with SF introduced, where at time t, one hypothesis { <sos>,w 1:t-1 } contains V types of tokens w v By concatenating (v=1,...,V), we obtain V hypotheses { <sos>,w 1:t-1 ,w v The calculation of scores (s2 and s3) when expanding to}(v=1,...,V) is shown. During this expansion, it is necessary to calculate the decoder score s2 (see Figure 3) for the vocabulary size, but this can be done all at once using the softmax function (see, for example, Non-Patent Document 4). Similarly, for the FLM score s3 (see Figure 3), if an NN-based LM is used as the FLM, the V FLM scores s3 can be calculated all at once using the softmax function, just like the decoder score s2. Also, if a classical count-based n-gram LM is used as the FLM, the FLM score can be obtained by table lookup (however, when using n-gram LM, long hypotheses cannot be considered, so for example, in the case of 3-gram LM (a language model that calculates the chain probability of 3 tokens), the hypothesis that is the condition for score calculation { <sos>,w 1:t-1 } to {w t-2,t-1 It degenerates into a chain of 2 tokens. For FLM score calculations based on NN and n-grams, see, for example, Non-Patent Document 5).

[0014] The above describes the score calculation when a single hypothesis is expanded into V hypotheses by concatenating V types of tokens. However, in actual beam search, multiple hypotheses are each expanded into V hypotheses, and these are then pruned. Figure 5 shows the overall processing procedure for beam search with SF introduced. The beam width is B (V>>B). Note that in Figure 5, the output is one best perfect hypothesis (the single perfect hypothesis with the highest score) as the speech recognition result. However, there are cases where we want to output N best perfect hypotheses (N perfect hypotheses arranged in descending order from the highest score based on score s4). In that case, in S94, instead of selecting only one best perfect hypothesis, similar to step S92, (B-b+N) perfect hypotheses are sorted in descending order based on score s4, the top N perfect hypotheses are selected, and the others are pruned. [Prior art documents] [Non-patent literature]

[0015] [Non-Patent Document 1] Z. T¨uske, G. Saon, K. Audhkhasi, and B. Kingsbury, "Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard," in Proc. Interspeech, 2020, pp. 551-555. [Non-Patent Document 2] C. Gulcehre et al., "On using monolingual corpora in neural machine translation," arXiv:1503.03535v2 [cs.CL]. [Non-Patent Document 3] A. Ogawa, N. Tawara, M. Delcroix, and S. Araki, "Lattice rescoring based on large ensemble of complementary neural language models," in Proc. ICASSP, 2022, pp. 6517-6521. [Non-Patent Document 4] S. Watanabe, T. Hori, S. Kim, JR Hershey, and T. Hayashi, "Hybrid CTC / Attention architecture for end-to-end speech recognition," IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 11, no. 8, pp. 1240-1253, Dec. 2017. [Non-Patent Document 5] M. Sundermeyer, R. Schl¨uter, and H. Ney, "LSTM neural network for language modeling," in Proc. Interspeech, 2012, pp. 194-197. [Non-Patent Document 6] Y. Bengio, "Practical recommendations for gradient-based training of deep architectures," arXiv:1206.5533v2 [cs.LG]. [Overview of the project] [Problems that the invention aims to solve]

[0016] Since SF can be considered a type of rescoring using an external language model, it is thought that if BLM could be applied to SF in addition to FLM, more accurate language scores could be assigned to hypotheses during the search process, thereby enabling more accurate beam search. However, because SF extends and develops speech recognition hypotheses from the beginning of a sentence, FLM is used as the language model, and it is difficult to apply BLM to hypotheses where the end of the sentence is not determined during the search process.

[0017] Therefore, the purpose of this disclosure is to provide a language model learning device that can learn BLM and can be incorporated into SF. [Means for solving the problem]

[0018] The language model learning device of the present invention includes a word order reversal unit, a degenerate token sequence generation unit, and an ISF-specific BLM language model learning unit.

[0019] The word order reversal unit obtains a complete token sequence to be used for FLM learning and reverses the word order of the complete token sequence. The degenerate token sequence generation unit uses the special symbols representing the end of a sentence from the complete token sequence whose word order has been reversed. <eos>A degenerate token sequence is generated by removing I tokens (where I is a natural number) from the tokens placed after the specified character. The ISF-specific BLM language model learning unit learns the ISF-specific BLM language model based on the degenerate token sequence. [Effects of the Invention]

[0020] According to the language model learning device of the present invention, it is possible to learn BLM that can be incorporated into SF. [Brief explanation of the drawing]

[0021] [Figure 1] A diagram illustrating the overview of speech recognition processing in an E2E speech recognition model. [Figure 2] This diagram shows an overview of the speech recognition process in an E2E speech recognition model using FLM. [Figure 3] Figure 2 shows an example of score calculation when concatenating a token wt to a hypothesis of length t-1, illustrating the details of the search process in the decoder. [Figure 4] This figure shows an example of calculating the scores (s2 and s3) when expanding one hypothesis into V hypotheses. [Figure 5] A flowchart illustrating the processing procedure for conventional beam search using SF (Science Focus). [Figure 6] A diagram illustrating the overview of speech recognition processing in an E2E speech recognition model using BLM. [Figure 7] Figure 6 shows an example of score calculation when concatenating a token wt to a hypothesis of length t-1, illustrating the details of the search process in the decoder. [Figure 8] This figure shows an example of calculating the scores (s2 and s6) when expanding one hypothesis into V hypotheses. [Figure 9] A block diagram showing an example of the functional configuration of the speech recognition device in Example 1. [Figure 10] A block diagram showing an example of the functional configuration of the N-Best complete hypothesis output unit in Example 1. [Figure 11] A flowchart illustrating the operation of the speech recognition device in Example 1. [Figure 12] A flowchart showing the beam search operation of the speech recognition device in Example 1. [Figure 13] This diagram shows an overview of the speech recognition process in an E2E speech recognition model that uses both FLM and BLM. [Figure 14] Figure 13 shows an example of score calculation when concatenating a token wt to a hypothesis of length t-1, illustrating the details of the search process in the decoder. [Figure 15] A block diagram showing an example of the functional configuration of the speech recognition device in Example 2. [Figure 16] A flowchart illustrating the operation of the speech recognition device in Example 2. [Figure 17] A flowchart illustrating the beam search operation of the speech recognition device in Example 2. [Figure 18] A block diagram showing an example of the functional configuration of the speech recognition device in Example 4. [Figure 19] A block diagram showing an example of the functional configuration of the N-Best complete hypothesis output unit in Example 4. [Figure 20] A flowchart illustrating the operation of the speech recognition device in Example 4. [Figure 21] A flowchart showing the beam search operation of the speech recognition device in Example 4. [Figure 22] A block diagram showing an example of the functional configuration of the speech recognition device in Example 5. [Figure 23] A block diagram showing an example of the functional configuration of the N-Best complete hypothesis output unit in Example 5. [Figure 24] A flowchart illustrating the operation of the speech recognition device in Example 5. [Figure 25] A flowchart showing the beam search operation of the speech recognition device in Example 5. [Figure 26] A block diagram showing an example of the functional configuration of the speech recognition device in Example 6. [Figure 27] A block diagram showing an example of the functional configuration of the N-Best complete hypothesis output unit in Example 6. [Figure 28] A flowchart showing the beam search operation of the speech recognition device in Example 6. [Figure 29] A figure showing an example of a token sequence generated by the language model learning device of Example 9. [Figure 30] A block diagram showing an example of the functional configuration of the language model learning device in Examples 9 and 10. [Figure 31] A flowchart illustrating the operation of the language model learning devices in Examples 9 and 10. [Figure 32] A figure showing an example of a token sequence generated by the language model learning device of Example 10. [Figure 33] A figure showing an example of a token sequence generated by the language model learning device of Example 11. [Figure 34] A block diagram showing an example of the functional configuration of the language model learning device in Examples 11 and 12. [Figure 35] Flowcharts illustrating the operation of the language model learning devices in Examples 11 and 12. [Figure 36] A figure showing an example of a token sequence generated by the language model learning device of Example 12. [Figure 37] A figure illustrating the effect of improving speech recognition accuracy using the speech recognition device described in the example. [Figure 38] A diagram showing an example of a computer's functional configuration. [Modes for carrying out the invention]

[0022] The embodiments of this disclosure will be described in detail below. Components having the same function will be numbered the same, and redundant explanations will be omitted. [Examples]

[0023] Figure 6 shows an overview of the speech recognition process in the E2E speech recognition model, where BLM is used as an external language model by ISF as described in this disclosure. In Figure 2, the language score calculated by FLM by SF was considered in the search process of the decoder in the E2E speech recognition model, but in Figure 6, instead, the language score calculated by BLM by ISF is considered.

[0024] Figure 7 shows the details of the search process in the decoder in Figure 6, where at time t, a hypothesis w of length t-1 is formed. 1:t-1 ={w1,w2,...,w t-1 } contains token w t Connect them and hypothesize the length t w 1:t The score calculation when extending is shown. As a premise, given X, w 1:t-1 The score s5 is assumed to have already been calculated at the previous time t-1 (at t=0, the score s5 is initialized to 0). 1:t-1 ni w t When concatenating X and { <sos>,w 1:t-1 When} is given, w t The decoder score s2 is calculated. Here, <eos>of{ <sos>,w 1:t By temporarily concatenating to} (w t but <eos>(In this case, they are not connected), complete hypothesis { <sos> ,w 1:t , <eos>This generates}. For this complete hypothesis, the word order is { <eos> ,w t:1 , <sos>Reverse} and calculate the BLM score s6. After calculating the BLM score s6, { <sos>,w 1:t Temporarily concatenated to} <eos>Delete (lol) t but <eos>(In this case, do not delete). By adding the weighted scores of the above three scores, given X, { <sos>,w 1:t We obtain the score s8 of}, but in this process, we use the inverted transient complete hypothesis { included in the score s5 calculated at the previous time t-1. <eos> ,w t-1:1 , <sos>Subtract the BLM score s7 for} (the BLM score s7 is the result of the search process at time t-1, { <sos>,w 1:t-1 (This information should be kept in mind as it is associated with}). In other words, the score s8 is obtained by the following formula (5).

number

number

number

[0025] Figure 8 shows one hypothesis { <sos>,w 1:t-1 } contains V types of tokens w v By concatenating (v=1,...,V), we obtain V hypotheses { <sos>,w 1:t-1 ,w v This shows how to calculate the scores (s2 and s6) when expanding to}(v=1,...,V). The decoder score s2 is calculated all at once for the vocabulary size (V) using the softmax function, as in Figure 4, but the BLM score s6 is calculated individually for the vocabulary size.

[0026] The functional configuration of the speech recognition device of Embodiment 1, which incorporates the above-described processing, will be explained below with reference to Figure 9. As shown in the figure, the speech recognition device 1 of this embodiment includes a decoder score calculation unit 11, a complete hypothesis generation unit 12, a BLM score calculation unit 13, a symbol deletion unit 14, an ISF score calculation unit 15, and an N-best complete hypothesis output unit 16.

[0027] As shown in Figure 10, the N-Best complete hypothesis output unit 16 includes a pruning unit 161, a hypothesis classification unit 162, a hypothesis selection unit 163, and a termination processing unit 164.

[0028] The detailed operation of each component will be explained below with reference to Figures 11 and 12.

[0029] Hypothesis of length t-1 lol 1:t-1 Token lol t Connect them and hypothesize the length t w 1:t In calculating the ISF score when extending the order, the decoder score calculation unit 11 uses a sequence of hidden state vectors X and a hypothesis w of length t-1. 1:t-1 Based on this, token w t The decoder score in the speech recognition model is calculated (S11).

[0030] The complete hypothesis generation unit 12 generates token w t If it is not at the end of the sentence <eos>of{ <sos>,w 1:t } is temporarily linked to the complete hypothesis { <sos> ,w 1:t , <eos>Generate} (S12).

[0031] The BLM score calculation unit 13 reverses the word order of the complete hypothesis and calculates the BLM score (S13).

[0032] The symbol deletion section 14 is token w t A special symbol that is temporarily concatenated when it is not at the end of a sentence. <eos>Delete (S14).

[0033] The ISF score calculation unit 15 uses a hypothesis w of length t-1. 1:t-1 ISF score P isf ({ <sos>,w 1:t-1 }|X) and decoder score P dec (w t |X,{ <sos>,w 1:t-1 }) and BLM score P blm ({ <eos> ,w t:1 , <sos>}) and the hypothesis w of length t-1 1:t-1 BLM score P blm ({ <eos> ,w t-1:1 , <sos>Based on}), for example, equation (5) gives the hypothesis w of length t. 1:t ISF score P isf ({ <sos>,w 1:t Calculate |X) (S15).

[0034] The decoder score calculation unit 11, the complete hypothesis generation unit 12, the BLM score calculation unit 13, the symbol deletion unit 14, and the ISF score calculation unit 15 each execute their respective processes while increasing t by 1 until there are no more partial hypotheses. Details will be described later, so the diagram omits the notation of repeated branches and other such details.

[0035] The N-Best Complete Hypothesis Output Unit 16 outputs the N-Best Complete Hypothesis based on the ISF score (S16).

[0036] The above describes the general operation of each component. Below, with reference to Figure 12, we will describe the detailed processing procedure for beam search (beam width = B) for a total of bV hypotheses, which are obtained by further developing V hypotheses for each of the multiple (b) hypotheses at time t.

[0037] The flowchart in this figure is similar to the flowchart in Figure 5, but differs in that, in hypothesis development (S11-S15), scores s2 and s6 are calculated instead of scores s2 and s3, and in pruning (S161), pruning is performed based on score s8 instead of score s4.

[0038] The decoder score calculation unit 11, the complete hypothesis generation unit 12, the BLM score calculation unit 13, the symbol deletion unit 14, and the ISF score calculation unit 15 execute the aforementioned steps S11 to S15 and calculate scores s2, S6, and S8.

[0039] The pruning unit 161 sorts the hypotheses in descending order based on the score s8, keeping the top B hypotheses and pruning the others (S161).

[0040] The hypothesis classification unit 162 classifies hypotheses based on whether they are complete hypotheses or not (S162).

[0041] The hypothesis selection unit 163 selects the 1(N) best complete hypothesis from the 1(N) best complete hypothesis (if any) with the highest score, and the (Bb) new complete hypotheses (total of (B-b+1(N)) complete hypotheses), and prunes the others (S163).

[0042] In the subsequent termination check, if b=0, i.e., if no partial hypotheses remain, the termination processing unit 164 outputs 1(N) best complete hypotheses (S164), and the process terminates (end).

[0043] In the subsequent termination determination, if b≠0, i.e., if a partial hypothesis remains, the speech recognition device 1 increments t, executes steps S11-S15, S161, S162, and S163 again, and performs the termination determination once more. [Examples]

[0044] The speech recognition device of Example 2 performs SF and ISF simultaneously. Figure 13 shows an overview of the speech recognition process in the E2E speech recognition model, where the FLM is used as the external language model by conventional SF, and in addition, the BLM is used as the external language model by ISF.

[0045] Figure 14 shows the details of the search process in the decoder in Figure 13, where at time t, a hypothesis w of length t-1 is formed. 1:t-1 ={w1,w2,...,w t-1 } contains token w t Connect them and hypothesize the length t w 1:t The score calculation when extending is shown. As a premise, given X, w 1:t-1 The score s9 is assumed to have already been calculated at the previous time t-1 (at t=0, the score s9 is initialized to 0). 1:t-1 ni w t When concatenating X and { <sos>,w 1:t-1 When} is given, w t The decoder score s2 and { <sos>,w 1:t-1 When} is given, w t Calculate the FLM score s3. Here, <eos>of{ <sos>,w 1:t By temporarily concatenating to} (w t but <eos>(In this case, they are not connected), complete hypothesis { <sos> ,w 1:t , <eos>} is temporarily generated. For this complete hypothesis, the word order is { <eos> ,w t:1 , <sos>Reverse} and calculate the BLM score s6. After calculating the BLM score s6, { <sos>,w 1:t Temporarily concatenated to} <eos>Delete (lol) t but <eos>(In this case, do not delete). By adding the weighted scores of the above four scores, given X, { <sos>,w 1:t We obtain the score s10 of}, but in this process, we use the inverted transient complete hypothesis { included in the score s9 calculated at the previous time t-1. <eos> ,w t-1:1 , <sos>Subtract the BLM score s7 for} (the BLM score s7 is the result of the search process at time t-1, { <sos>,w 1:t-1 (Store it as information associated with {). That is, the score s10 is obtained by the following formula (8). [Number] Referring to FIG. 15 below, the functional configuration of the speech recognition apparatus according to Embodiment 2 incorporating the above-described processing will be described. As shown in the figure, the speech recognition apparatus 2 of the present embodiment includes a decoder score calculation unit 11, an FLM score calculation unit 22, a complete hypothesis generation unit 12, a BLM score calculation unit 13, a symbol deletion unit 14, an SF + ISF score calculation unit 25, and an N-best complete hypothesis output unit 16. The components other than the FLM score calculation unit 22 and the SF + ISF score calculation unit 25 operate in the same manner as in Embodiment 1.

[0046] Hereinafter, referring to FIG. 16, the detailed operations of each component will be described. Step S11 is executed in the same manner as in Embodiment 1. The FLM score calculation unit 22 is a symbol <sos>And the hypothesis of length t-1 lol 1:t-1 Based on this, the token w at time t t FLM score P flm (w t |{ <sos>, w 1:t-1 )}) is calculated (S22). Steps S12 to S14 are executed in the same manner as in the first embodiment. The SF+ISF score calculation unit 25 calculates the SF+ISF score P 1:t-1 in the hypothesis w of length t-1 sf+isf ({ <sos>,w 1:t-1 }|X) and decoder score P dec (w t |X,{ <sos>,w 1:t-1 }) and FLM score P flm (w t |{ <sos>,w 1:t-1 }) and BLM score P blm ({ <eos> ,w t:1 , <sos>}) and BLM score P blm ({ <eos> ,w t-1:1 , <sos>Based on}), for example, by equation (8), the hypothesis w of length t 1:t SF+ISF score P sf+isf ({ <sos>,w 1:t Calculate |X) (S25).

[0047] Figure 17 shows the overall processing procedure for beam search (beam width = B) incorporating SF and ISF. The flowchart in this figure is similar to the flowchart in Figure 12, but differs in that scores s2, s3, s6, and s10 are calculated in the hypothesis development (S11, S22, S12, S13, S14, S25), and in pruning (S161), pruning is performed based on score s10 instead of score s8.

[0048] If α=0 and β>0 in equation (8), then only ISF will be performed and SF will not be performed (equivalent to Example 1). If α>0 and β=0 in equation (8), then only SF will be performed and ISF will not be performed. If α=0 and β=0 in equation (8), then neither SF nor ISF will be performed.

[0049] As described above, the speech recognition devices described in Examples 1 and 2 allow for the incorporation of BLM into the SF, enabling the assignment of more accurate language scores to hypotheses during the search process, and thus achieving more accurate beam search. However, it is clear that the computational cost of ISF using BLM is high with the method described above. Examples 3 to 6 below disclose speech recognition devices that reduce this computational cost. [Examples]

[0050] In the methods of Examples 1 and 2, the BLM score s6 is calculated individually for each vocabulary size, resulting in high computational cost (see Figure 8). Therefore, in Example 3, the BLM score s6 is calculated using batch calculation, bundling hypotheses for each vocabulary size (for details on batch calculation, see, for example, Non-Patent Literature 6). In the label (token) synchronized beam search assumed in this specification (see, for example, Non-Patent Literature 4), batch calculation is particularly effective (there is no wasted computation) because the lengths of each hypothesis during the search process are basically the same, but it is also possible to introduce batch calculation to other search algorithms.

[0051] In other words, in the speech recognition device of Embodiment 3, the BLM score calculation unit is characterized by calculating the BLM score s6 by batch calculation that bundles hypotheses equal to the vocabulary size, thereby reducing the cost of calculating the BLM score. [Examples]

[0052] In the above embodiment, the BLM score s6 based on ISF was calculated each time the hypothesis was extended by 1 token (1 time interval). In Embodiment 4, however, the BLM score s6 based on ISF is calculated each time the hypothesis is extended by I tokens (1 time interval). That is, the BLM score s6 based on ISF is calculated only when the length (time) t of the hypothesis satisfies t%I=0 (where % is the modulo operator). Hereafter, I will be referred to as the interval (I≧1). Equation (7) in Embodiment 1 is changed to the following equation (9) in this embodiment.

number

[0053] As shown in Figure 19, the N-Best complete hypothesis output unit 46 includes a pruning unit 161, a hypothesis classification unit 162, a complete hypothesis score calculation unit 463, a hypothesis selection unit 163, and a termination processing unit 164. Except for the complete hypothesis score calculation unit 463, the unit operates the same as in Example 1 and Example 2.

[0054] The detailed operation of each component will be explained below with reference to Figure 20. Steps S11, S22, and S12 are performed in the same manner as in Example 2. The BLM score calculation unit 43 calculates the BLM score only when I is an arbitrary interval and the hypothetical length t satisfies t%I=0 (where % is the modulo operator) (S43). Steps S14 and S25 are performed in the same manner as in Example 2.

[0055] Figure 21 shows the overall processing procedure for beam search with ISF based on SF and interval I. The beam width is B. The flowchart in this figure is similar to the flowchart in Figure 17, but differs in that step S43 is executed instead of step S13 in hypothesis development, reducing the frequency of BLM score calculation; in step S161, pruning is performed based on score s10 if t%I=0, and based on a score other than the BLM score s9+s2+α×s3 if t%I≠0; and step S463 is inserted between steps S162 and S163, and score s10 is calculated for the complete hypothesis if t%I≠0.

[0056] As mentioned above, the effectiveness of rescoring for complete hypotheses (N-best rescoring and lattice rescoring) has been confirmed for a long time (see, for example, Non-Patent Document 3). Therefore, in this embodiment as well, in step S463, if t%I≠0 for a complete hypothesis, the score s10 is calculated. If t%I=0, that is, if the score s10 has already been calculated in the hypothesis development (S11, S22, S12, S43, S14, S25), there is no need to calculate it again. By introducing interval I, the calculation cost of ISF is reduced to approximately 1 / I compared to before the introduction.

[0057] If I=1 in equation (9), this embodiment is equivalent to embodiment 2. If α=0 and β>0 in equation (8), then SF will not be performed, and only ISF will be performed at interval I. If α>0 and β=0 in equation (8), then ISF will not be performed, and only SF will be performed. If α=0 and β=0 in equation (8), then neither SF nor ISF will be performed. [Examples]

[0058] In beam search, it is necessary to retain the hypotheses (multiple) that are most likely to yield the highest score for the input speech, and to prune the other hypotheses that are less likely to yield the highest score early on. If a highly likely hypothesis is mistakenly pruned early in the search process, it becomes difficult to obtain highly accurate speech recognition results in the end. Therefore, it is considered important to assign highly accurate scores to hypotheses at an early stage of the search process rather than at a later stage. Accordingly, in Example 5, ISF is performed only when the length t of the hypothesis is shorter than L (t≦L).

[0059] The functional configuration of the speech recognition device of Embodiment 5, which incorporates the above-described processing, will be explained below with reference to Figure 22. As shown in the figure, the speech recognition device 5 of this embodiment includes a decoder score calculation unit 11, an FLM score calculation unit 22, a complete hypothesis generation unit 12, a BLM score calculation unit 53, a symbol deletion unit 14, an SF+ISF score calculation unit 25, and an N-best complete hypothesis output unit 56. Except for the BLM score calculation unit 53 and the N-best complete hypothesis output unit 56, the configuration requirements are the same as in Embodiment 2.

[0060] As shown in Figure 23, the N-Best complete hypothesis output unit 56 includes a pruning unit 161, a hypothesis classification unit 162, a complete hypothesis score calculation unit 563, a hypothesis selection unit 163, and a termination processing unit 164. Except for the complete hypothesis score calculation unit 563, the unit operates the same as in Example 1 and Example 2.

[0061] The detailed operation of each component will be explained below with reference to Figure 24. Steps S11, S22, and S12 are performed in the same manner as in Example 2. The BLM score calculation unit 53 calculates the BLM score only when the length t of the hypothesis is shorter than L (t≦L), with L being an arbitrary implementation hypothesis length (S53). Steps S14 and S25 are performed in the same manner as in Example 2.

[0062] Figure 25 shows the overall processing procedure for beam search with ISF based on SF and ISF implementation hypothesis length L. The beam width is B. The flowchart in this figure is similar to the flowchart in Figure 17, but differs in that step S53 is executed instead of step S13 in hypothesis development, reducing the frequency of BLM score calculation; in step S161, pruning is performed based on score s10 if t≦L, and based on a score other than the BLM score s9+s2+α×s3 if t>L; and step S563 is inserted between steps S162 and S163, and score s10 is calculated for the complete hypothesis if t>L.

[0063] If L=∞, this embodiment becomes equivalent to Embodiment 2. If α=0 and β>0 in equation (8), then SF will not be performed, and only ISF will be performed with an ISF implementation hypothesis length L. If α>0 and β=0 in equation (8), then ISF will not be performed, and only SF will be performed. If α=0 and β=0 in equation (8), then neither SF nor ISF will be performed. [Examples]

[0064] Example 6 is a modification of Example 2, and performs pruning in two stages. The functional configuration of the speech recognition device of Example 6, which incorporates a two-stage pruning process, will be described below with reference to Figure 26. As shown in the figure, the speech recognition device 6 of this example includes a decoder score calculation unit 11, an FLM score calculation unit 22, a complete hypothesis generation unit 12, a BLM score calculation unit 13, a symbol deletion unit 14, an SF+ISF score calculation unit 25, and an N-best complete hypothesis output unit 66. Except for the N-best complete hypothesis output unit 66, the configuration requirements are the same as in Example 2.

[0065] As shown in Figure 27, the N-Best complete hypothesis output unit 66 includes a first pruning unit 661, a second pruning unit 662, a hypothesis classification unit 162, a hypothesis selection unit 163, and a termination processing unit 164. Except for the first pruning unit 661 and the second pruning unit 662, the configuration elements operate in the same way as in Embodiment 2.

[0066] Figure 28 shows the overall beam search procedure when Example 2 is modified to perform pruning in two stages. Let M be the beam width for the first stage of pruning, and B be the beam width for the second stage of pruning (M > B). The flowchart in this figure is similar to the flowchart in Figure 17, but differs in that only steps S11 and S22, which have low costs in hypothesis development, are executed; in step S661, the first pruning unit 661 performs pruning to retain the top M hypotheses based on scores s9+s2+α×s3 other than the BLM score; the BLM score calculation unit 13 calculates the BLM score only for the top M hypotheses obtained from the first stage of pruning, and the other constituent elements perform pre-processing and post-processing (S12, S13, S14, S25) to reduce the frequency of BLM score calculation; and after the BLM score calculation, in step S662, the second pruning unit 662 performs pruning to retain the top B hypotheses based on score S10 through a second stage of pruning.

[0067] In this embodiment, during hypothesis development and the first stage of pruning (S11, S22, S661), the first stage of pruning is performed based on the computationally inexpensive decoder score s2 and FLM score s3, leaving M hypotheses. Then, during BLM score calculation and the second stage of pruning (S12, S13, S14, S25), the second stage of pruning is performed considering the computationally expensive BLM score s6, leaving B hypotheses. In other words, instead of calculating the BLM score for all of the very large number of bV hypotheses, as in Embodiment 2, the computational cost is reduced by calculating the BLM score only for a certain number of M hypotheses (bV >> M > B). Possible ways to set M include, for example, M = B × B.

[0068] If M=bV, this embodiment is equivalent to Embodiment 2. If α=0 and β>0 in equation (8), then SF will not be performed (the first stage of pruning will be performed based only on the decoder score s2), and only ISF will be performed. If α>0 and β=0 in equation (8), then ISF will not be performed, and only SF will be performed (based on the score s9+s2+α×s3, the first stage of pruning will not be performed (there is no point in performing it), and only the second stage of pruning will be performed). If α=0 and β=0 in equation (8), then neither SF nor ISF will be performed (based on the score s9+s2, the first stage of pruning will not be performed, and only the second stage of pruning will be performed). [Examples]

[0069] Based on Example 1, it is also possible to modify the method to implement two or more of the following: batch calculation (Example 3), ISF based on interval I (Example 4), ISF based on ISF implementation hypothesis length L (Example 5), and two-stage pruning (Example 6). [Examples]

[0070] Based on Example 2, it is also possible to modify the method to implement two or more of the following: batch calculation (Example 3), ISF based on interval I (Example 4), ISF based on ISF implementation hypothesis length L (Example 5), and two-stage pruning (Example 6). [Examples]

[0071] The following was transcribed by hand. <sos>It begins <eos>A grammatically and word-orderly correct token sequence ending with is called a "complete token sequence." BLM is learned using complete token sequences with reversed word order, but as mentioned above, in the ISF of this disclosure, BLM is applied to partial hypotheses (temporarily to partial hypotheses). <eos>(These are linked together to form a complete hypothesis, to which BLM is applied). Therefore, as shown in Figure 29, the apparatus of Example 9 generates (enhances) a token sequence assuming that BLM will be applied to ISF, and uses the generated token sequence (hereinafter also called a degenerate token sequence) to learn an ISF-specific BLM.

[0072] Referring to Figure 30, the functional configuration of the language model learning device that generates the above-mentioned token sequence and learns a language model will be explained. As shown in the figure, the language model learning device 900 of Example 9 (and Example 10, described later) includes a word order inversion unit 901, a degenerate token sequence generation unit 902, an FLM language model learning unit 903, a BLM language model learning unit 904, and an ISF-dedicated BLM language model learning unit 905.

[0073] The operation of each component will be explained below with reference to Figure 31. The word order reversal unit 901 is used for the complete token sequence used for FLM learning (in the example in Figure 29, { <sos>, a, b, c, d, e, f, g, h, i, j, <eos>The} is obtained and the word order of the complete token sequence is reversed (S901). The degenerate token sequence generation unit 902 selects the special symbol representing the end of the sentence from the complete token sequence whose word order has been reversed. <eos>A degenerate token sequence is generated by deleting I tokens (I is a natural number, in this embodiment I=1) from the tokens placed after the (S902). The FLM language model learning unit 903 generates an FLM language model based on the complete token sequence used for FLM learning (S903). The BLM language model learning unit 904 learns a BLM language model based on the token sequence obtained by reversing the word order of the complete token sequence (S904). The ISF-specific BLM language model learning unit 905 learns an ISF-specific BLM language model based on the degenerate token sequence (S905).

[0074] In this embodiment, the interval I is set to 1 (Example 4), and the ISF implementation hypothesis length L is set to ∞ (Example 5). By setting it this way, it is possible to generate a sequence of inverted tokens of any length starting from any token, and to learn an ISF-specific BLM that can easily handle any situation in ISF.

[0075] However, as is clear from Figure 29, setting I=1 and L=∞ results in an enormous amount of training data for the ISF-specific BLM. In the following 10th to 12th embodiments, the amount of training data for the ISF-specific BLM is limited by appropriately setting I and L. [Examples]

[0076] As shown in Figure 32, the language model learning device of Example 10 limits the amount of training data for the ISF-specific BLM by setting I to a value greater than 1 (I=2 in Figure 32) while keeping L as ∞. [Examples]

[0077] As shown in Figure 33, the language model learning device of Example 11 limits the amount of training data for the ISF-specific BLM by keeping I as 1 and setting L to a finite integer value greater than 1 (L=6 in Figure 33). Referring to Figure 34, the functional configuration of the language model learning device of Example 11 (and Example 12, described later) will be explained. As shown in the figure, the language model learning device 1100 of Example 11 includes a word order reversal unit 901, a degenerate token sequence generation unit 1102, an FLM language model learning unit 903, a BLM language model learning unit 904, and an ISF-dedicated BLM language model learning unit 905. Except for the degenerate token sequence generation unit 1102, the configuration is the same as that of Examples 9 and 10. The language model learning device 1100 of this example aims to use only token sequences shorter than the implementation hypothesis length L for learning the ISF-dedicated BLM language model.

[0078] Referring to Figure 35, the operation of each component of the language model learning device 1100 of Example 11 will be described in a way that differs from the language model learning device 900 of Example 9. The degenerate token sequence generation unit 1102 generates a complete token sequence with the word order reversed, <eos>Remove the token placed next to it, <eos>and special symbols to indicate the beginning of a sentence <sos>The first degenerate token sequence is generated such that the length of the token sequence excluding the specified token is equal to L, and thereafter, degenerate token sequences are generated by deleting I tokens (I=1 in this embodiment) at a time (S1102). [Examples]

[0079] As shown in Figure 36, the language model learning device of Example 12 limits the amount of training data for the ISF-specific BLM by setting I to a value greater than 1 (I=2 in Figure 36) and L to a value greater than 1 (L=6 in Figure 36). [Examples]

[0080] The speech recognition device of Example 13 implements Examples 1 to 8 using the ISF-specific BLM trained in Examples 9 to 12.

[0081] <Experimental Results> Figure 37 shows the effect of improving speech recognition accuracy (reduction of word error rate [%]) using the speech recognition device described in the example. A neural language model (LSTMLM) based on long short-term memory (LSTM) is used as the language model. The effect of SF using conventional FLM can be confirmed by comparing Method 0 and Method 1. The effect of ISF using BLM with the speech recognition device described in the example can be confirmed by comparing Method 0 with Method 2 or 3. The effect of ISF using BLM with the speech recognition device described in the example can be confirmed by comparing Method 2 and Method 3, showing that the ISF-specific BLM from Example 9 is even more suitable for ISF than the regular BLM. The effect of using SF using FLM and ISF using (ISF-specific) BLM in combination can be confirmed by comparing Methods 1, 2, and 3 with Method 4.

[0082] <Note> The device of this disclosure, for example, as a single hardware entity, has an input section to which a keyboard or the like can be connected, an output section to which a liquid crystal display or the like can be connected, a communication section to which a communication device (e.g., a communication cable) that can communicate with the outside of the hardware entity can be connected, a CPU (Central Processing Unit, which may include cache memory and registers), RAM or ROM as memory, an external storage device such as a hard disk, and a bus that connects these input section, output section, communication section, CPU, RAM, ROM, and external storage device to enable data exchange between them. Furthermore, if necessary, the hardware entity may be equipped with a device (drive) that can read and write recording media such as CD-ROMs. An example of a physical entity equipped with such hardware resources is a general-purpose computer.

[0083] The external storage device of the hardware entity stores the programs necessary to realize the above-mentioned functions, as well as the data required for processing these programs (this is not limited to external storage; for example, programs may be stored in ROM, a read-only storage device). Furthermore, data obtained through the processing of these programs is appropriately stored in RAM or other external storage devices.

[0084] In hardware entities, each program stored in an external storage device (or ROM, etc.) and the data necessary for processing each program are loaded into memory as needed, and interpreted, executed, and processed by the CPU as appropriate. As a result, the CPU realizes the predetermined functions (each of the configuration requirements expressed as ...part, ...means, etc. above).

[0085] This disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from the spirit of this disclosure. Furthermore, the processes described in the embodiments above may not only be executed in chronological order according to the order described, but may also be executed in parallel or individually as needed, depending on the processing capacity of the device performing the processes.

[0086] As described above, when the processing functions of the hardware entity (device of this disclosure) described in the above embodiment are implemented by a computer, the processing content of the functions that the hardware entity should have is described by a program. Then, by executing this program on the computer, the processing functions of the hardware entity are implemented on the computer.

[0087] The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 10020 of the computer 10000 shown in Figure 38, and then causing the control unit 10010, input unit 10030, output unit 10040, etc. to operate.

[0088] The program describing this processing can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory. Specifically, examples include hard disk drives, flexible disks, and magnetic tapes as magnetic recording devices; DVDs (Digital Versatile Discs), DVD-RAMs (Random Access Memory), CD-ROMs (Compact Disc Read Only Memory), and CD-Rs (Recordable) / RWs (ReWritable) as optical discs; MOs (Magneto-Optical Discs) as magneto-optical recording media; and EEP-ROMs (Electrically Erasable and Programmable-Read Only Memory) as semiconductor memory.

[0089] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0090] A computer executing such a program may, for example, first store the program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. Then, when processing is to be executed, the computer reads the program stored on its own storage medium and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the above processing may be executed by a so-called ASP (Application Service Provider) type service, where the server computer does not transfer programs to this computer, but the processing function is realized only by execution instructions and result acquisition. In this form, the program includes information used for processing by an electronic computer that is equivalent to a program (data that is not a direct instruction to the computer but has the property of defining the processing of the computer, etc.).

[0091] Furthermore, in this configuration, a hardware entity is constructed by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware.< / sos> < / eos> < / eos> < / eos> < / eos> < / sos> < / eos> < / eos> < / sos> < / sos> < / sos> < / eos> < / sos> < / eos> < / sos> < / sos> < / sos> < / sos> < / sos> < / sos> < / sos> < / eos> < / sos> < / eos> < / eos> < / sos> < / sos> < / eos> < / eos> < / sos> < / eos> < / sos> < / eos> < / sos> < / sos> < / sos> < / sos> < / eos> < / sos> < / eos> < / sos> < / sos> < / eos> < / eos> < / sos> < / sos> < / eos> < / sos> < / sos> < / eos> < / eos> < / sos> < / sos> < / eos> < / sos> < / eos> < / eos> < / sos> < / sos> < / eos> < / eos> < / sos> < / eos> < / sos> < / eos> < / sos> < / eos> < / sos> < / sos> < / sos> < / sos> < / eos> < / sos> < / sos> < / sos> < / sos> < / sos> < / sos> < / sos> < / sos> < / sos> < / sos> < / sos> < / eos> < / sos> < / eos> < / sos> < / eos> < / sos> < / eos> < / sos>

Claims

1. A token sequence is a text such as a word sequence / subword sequence / string corresponding to a sound, and a complete token sequence is a grammatically and word-orderly correct token sequence. A word order reversal unit obtains a complete token sequence to be used for FLM learning, and reverses the word order of the complete token sequence. Among the complete sequence of tokens with reversed word order, the special symbol representing the end of the sentence. <eos> A degenerate token sequence generator generates a degenerate token sequence by removing I tokens (where I is a natural number) from the tokens placed after the next one,< / eos> Includes an ISF-specific BLM language model learning unit that learns an ISF-specific BLM language model based on a degenerate token sequence. Language model learning device.

2. A language model learning device according to claim 1, The aforementioned degenerate token sequence generation unit, Of the complete token sequence with the word order reversed, <eos>Remove the token placed next to it, <eos>and special symbols to indicate the beginning of a sentence <sos> The first degenerate token sequence is generated such that the length of the token sequence excluding the specified token is equal to L (where L is a finite integer greater than 1), and subsequent degenerate token sequences are generated by removing I tokens at a time.< / sos> < / eos> < / eos> Language model learning device.

3. A language model learning method performed by a language model learning device, A token sequence is a text sequence such as a word sequence, subword sequence, or string of characters that corresponds to a sound, and a complete token sequence is a grammatically and word-orderly correct token sequence. The steps involve obtaining a complete token sequence to be used for FLM learning, and reversing the word order of the complete token sequence. Among the complete sequence of tokens with reversed word order, the special symbol representing the end of the sentence. <eos> The steps include generating a degenerate token sequence by removing I tokens (where I is a natural number) from the tokens placed after the previous one,< / eos> This includes a step of training an ISF-specific BLM language model based on a degenerate token sequence. Language model learning methods.

4. A program that causes a computer to function as a language model learning device according to claim 1 or 2.

Images