Speech recognition system, speech recognition method and program

Abstract

To provide a speech recognition system with higher inference accuracy by using a large-scale language model.SOLUTION: A speech recognition system includes: a first pre-trained inference model (SSL model 20) that can output tokens corresponding to words contained in speech signals on a token-by-token basis, a second pre-trained inference model (large-scale language model 30) that outputs one or more corresponding tokens on the basis of inference based on one or more token inputs, and a multiplier 26 that multiplies a first inference output 22 of the first pre-trained inference model, and a second inference output 32 corresponding to the first inference output, which is an output of the second pre-trained inference model, on a word-by-word basis. Then, the word selected based on the new probability distribution, which is an output of the multiplier, is input to the second pre-trained inference model.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a voice recognition system, a voice recognition method, and a program. [Background technology]

[0002] As an example of an application of ASR (Automatic Speech Recognition), a system that performs speech recognition including a large-scale language model that has been trained on a large scale is known. As the large-scale language model, typically, there are an encoder-based model and a decoder-based model.

[0003] Known examples of encoder-based models include BERT (see Non-Patent Document 1, etc.) and Transformer's bidirectional encoder (see Non-Patent Document 2, etc.). Known examples of decoder-based models include GPT-2 (see Non-Patent Document 3, etc.) and unidirectional Transformer decoder.

[0004] Large language models may also be used primarily for error correction, model compression, rescoring, multimodal applications, etc. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint arXiv:1810.04805, 2018. [Non-Patent Document 2] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin, Attention is all you need, Proc. NeurlPS 30 (2017). [Non-Patent Document 3] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019. Summary of the Invention [Problem to be solved by the invention]

[0006] The large-scale language model described above is designed to output a sequence of vectors corresponding to the input of a sequence of vectors. Based on such a sequence-to-sequence design, a speech recognition system that includes a large-scale language model and processes sentences has been proposed.

[0007] An object of the present invention is to provide a speech recognition system with higher inference accuracy by utilizing a large-scale language model. [Means for solving the problem]

[0008] A speech recognition system according to one embodiment includes a first trained inference model capable of outputting tokens corresponding to words included in a speech signal on a token-by-token basis, a second trained inference model that outputs one or more corresponding tokens by inference based on input of one or more tokens, and a multiplier that multiplies a first inference output of the first trained inference model by a second inference output of the second trained inference model, the second inference output corresponding to the first inference output, on a word-by-word basis. A word selected based on a new probability distribution, which is the output of the multiplier, is input to the second trained inference model.

[0009] The output of the multiplier may be used as the recognition output of a speech recognition system. The second trained inference model may be pre-trained at least on the task of predicting the next word for an input sequence using a corpus of a target language of speech recognition.

[0010] The second trained inference model may be fine-tuned using dialect data of the target language.

[0011] A computer-implemented speech recognition method according to another embodiment includes the steps of: using a first trained inference model to output tokens corresponding to words contained in a speech signal on a token-by-token basis; using a second trained inference model to output one or more corresponding tokens by inference based on input of one or more tokens; multiplying, on a word-by-word basis, a first inference output of the first trained inference model by a second inference output of the second trained inference model, the second inference output corresponding to the first inference output; and inputting words selected based on a new probability distribution obtained as a result of the multiplication into the second trained inference model.

[0012] According to yet another embodiment, there is provided a program for performing speech recognition, the program causing a computer to execute the steps of: outputting, on a token-by-token basis, tokens corresponding to words included in a speech signal using a first trained inference model; outputting, on a token-by-token basis, one or more corresponding tokens using a second trained inference model by inference based on input of one or more tokens; multiplying, on a word-by-word basis, a first inference output of the first trained inference model by an output of a second trained inference model, the second inference output corresponding to the first inference output; and inputting a word selected based on a new probability distribution obtained as a result of the multiplication to the second trained inference model. Effect of the Invention

[0013] According to the present invention, a speech recognition system with higher inference accuracy can be provided by utilizing a large-scale language model. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram showing an example of a model configuration of a voice recognition system according to a related art. [Diagram 2] 2 is a diagram for explaining an example of a multiplication process performed by a multiplier shown in FIG. 1; [Diagram 3] FIG. 2 is a schematic diagram showing an example of a model configuration of a voice recognition system according to the present embodiment. [Figure 4] 1 is a diagram for illustrating an example of processing in a voice recognition system according to an embodiment of the present invention. [Diagram 5] FIG. 13 is a schematic diagram showing another example of a model configuration of the speech recognition system according to the present embodiment. [Figure 6] 1 is a diagram for illustrating an example of processing in a voice recognition system according to an embodiment of the present invention. [Figure 7] FIG. 2 is a schematic diagram showing an example of a hardware configuration for implementing the voice recognition system according to the present embodiment; [Figure 8] 5 is a flowchart showing an example of a processing procedure in the voice recognition system according to the present embodiment. [Figure 9] 5 is a flowchart showing an example of a processing procedure in the voice recognition system according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings, in which the same or corresponding parts in the drawings are designated by the same reference characters and will not be described repeatedly.

[0016] [A. Model configuration] A model configuration of the speech recognition system according to the present embodiment will be described.

[0017] FIG. 1 is a schematic diagram showing an example of a model configuration of a voice recognition system 3 according to the related art. First, a basic concept for understanding this embodiment will be described with reference to Fig. 1. A speech recognition system 3 includes a self-supervised learning (SSL) model 20 that ultimately calculates tokens 6 corresponding to each word included in an input sequence 4, and a large-scale language model 30 that receives the tokens 6 as input and ultimately outputs a word sequence.

[0018] The SSL model 20 is a model for an automatic speech recognition (ASR) task. The SSL model 20 corresponds to a first trained inference model, and outputs tokens corresponding to words included in a speech signal on a token-by-token basis. The SSL model 20 includes a multi-layered Convolutional Neural Network (CNN) that extracts features from an input speech signal.

[0019] The SSL model 20 may be, for example, an encoder-based model, and Word2Vec (W2V) or the like may be used.

[0020] The speech recognition system 3 has a vocabulary dictionary (not shown) that is commonly referred to throughout the system. Words (tokenized units) that are units of speech recognition processing are registered in the vocabulary dictionary. Identification information (hereinafter also referred to as "word ID") for identifying each registered word is assigned. The sentence to be output may be a string of recognized words themselves, but for convenience of processing, it is preferable that the output sentence be a string of word IDs assigned to each word in the vocabulary dictionary. In the speech recognition system 3 shown in FIG. 1, word IDs defined in the vocabulary dictionary are output from the SSL model 20 to the large-scale language model 30. It should be noted that the corresponding word itself can be extracted at any time by referring to the vocabulary dictionary based on the word ID.

[0021] The SSL model 20 can be generated by pre-training based on SSL using unlabeled speech. The SSL model 20 is fine-tuned using labeled data according to the Connectionist Temporal Classification (CTC) algorithm. Therefore, the inference output 22 (one or more tokens 6) of the SSL model 20 is denoted as ProbCTC.

[0022] The inference output 22 of the SSL model 20 indicates the speech recognition result, and is a probability distribution for the words included in the vocabulary dictionary. The argmax layer 24 outputs the word (word ID) with the maximum probability included in the inference output 22 as the final speech recognition result (token 6). One or more tokens 6 output in sequence form a sentence 8 (corresponding to one sentence).

[0023] The large-scale language model 30 may be an encoder-based model (e.g., BERT, etc.) or a decoder-based model (e.g., GPT-2 or its successors, etc.). The large-scale language model 30 has been pre-trained on at least the task of predicting the next word for an input sequence using a corpus of a target language for speech recognition by the SSL model 20.

[0024] Furthermore, the large-scale language model 30 may be fine-tuned using dialect data of the target language of speech recognition by the SSL model 20. For example, as described below, the large-scale language model 30 may be pre-trained using a Chinese (Simplified characters) corpus and then fine-tuned using a Chinese (Simplified characters)-Chinese (Traditional characters) dataset.

[0025] For input of sentence 8 output from the SSL model 20, the large-scale language model 30 outputs the probability of the word that exists next to each word included in the sentence 8. The large-scale language model 30 corresponds to a second trained inference model, and outputs one or more corresponding tokens by inference based on the input of sentence 8 (one or more tokens 6). The inference output 32 of the large-scale language model 30 is a probability distribution for words included in the vocabulary dictionary, similar to the inference output 22 of the SSL model 20. For this reason, the inference output 32 of the large-scale language model 30 is denoted as ProbPLM.

[0026] The multiplier 34 multiplies the inference output 22 (probability distribution for words) of the SSL model 20 by the inference output 32 (probability distribution for words) of the large-scale language model 30, and outputs the multiplication result 10. The multiplication result 10 is a probability distribution for words included in the vocabulary dictionary, similar to the inference output 22 of the SSL model 20 and the inference output 32 of the large-scale language model 30. The multiplier 34 multiplies the inference output 22 of the SSL model 20 by the inference output 32, which corresponds to the inference output 22, from the outputs of the large-scale language model 30, on a word-by-word basis.

[0027] Fig. 2 is a diagram for explaining an example of the multiplication process by the multiplier 34 shown in Fig. 1. With reference to Fig. 2, the example of the multiplication process by the multiplier 34 will be described in further detail.

[0028] In Fig. 2, inference output 22 shows a probability distribution for a word output by SSL model 20 at a certain position in input sequence 4. In inference output 22, a probability is calculated for every word contained in the vocabulary dictionary. In Fig. 2, inference output 32 shows a probability distribution for a word output by large-scale language model 30 at the same position in input sequence 4.

[0029] The multiplication result 10 obtained by multiplying the inference output 22 and the inference output 32 on a word-by-word basis indicates a new probability distribution for all words included in the vocabulary dictionary. Since probability takes a value between 0 and 1, words having a relatively large probability value in the inference output 22 and the inference output 32 will indicate a larger value in the multiplication result 10 than words having a relatively small probability value. As a result, the inference output 22 of the SSL model 20 is corrected by the inference output 32 of the large-scale language model 30.

[0030] As shown in FIGS. 1 and 2, the inference output 22 of the SSL model 20 is corrected by the inference output 32 of the large-scale language model 30, thereby improving the inference accuracy.

[0031] 1 and 2 show an example in which GPT-2 is used as the large-scale language model 30, but inference accuracy can also be improved when BERT is used as the large-scale language model 30.

[0032] Fig. 3 is a schematic diagram showing an example of a model configuration of a speech recognition system 1 according to the present embodiment. With reference to Fig. 3, the speech recognition system 1 includes a multiplier 26 instead of the multiplier 34, as compared with the speech recognition system 3 shown in Fig. 1. Fig. 3 shows an example configuration using GPT-2 or its successor model as a large-scale language model 30.

[0033] The multiplier 26 multiplies the inference output 22 (probability distribution for words) of the SSL model 20 by the inference output 32 (probability distribution for words) of the large-scale language model 30, and outputs the multiplication result. The multiplication result is a probability distribution for words included in the vocabulary dictionary, similar to the inference output 22 of the SSL model 20 and the inference output 32 of the large-scale language model 30. The multiplier 26 multiplies the inference output 22 of the SSL model 20 by the inference output 32, which corresponds to the inference output 22, from the outputs of the large-scale language model 30, on a word-by-word basis.

[0034] In the speech recognition system 1 shown in FIG. 3, the inference output of the SSL model 20 at the current step t+1 and the inference output of the large-scale language model 30 at the previous step (step t) are combined and input to the large-scale language model 30 to obtain the inference output of the current step t+1.

[0035] Thus, the words selected based on the new probability distribution that is the output of multiplier 26 are input to large scale language model 30 .

[0036] In this way, by sequentially feeding back the inference output of each model obtained earlier, the accuracy of the tokens input to the large-scale language model 30 can be improved, and as a result, the inference accuracy of the large-scale language model 30 can be improved.

[0037] 4 is a diagram for explaining a processing example in voice recognition system 1 according to the present embodiment. With reference to FIG. 4, a processing example in voice recognition system 1 will be described in further detail.

[0038] In FIG. 4, for each processing step, the output state of the SSL model 20 (Prob CTC & Combine & argmax), the output state of the large-scale language model 30, and the state of the signal for masking the input to the large-scale language model 30 (hereinafter also referred to as the "mask signal") (Prob PLM-GPT) are shown.

[0039] In the example shown in FIG. 4, one sentence contains four words. In the following description, for a probability distribution P, the inference output of the SSL model 20 is represented using a subscript "S", and the inference output of the large-scale language model 30 is represented using a subscript "L". Furthermore, for the probability distribution P, the state of which step is the probability distribution is represented using a superscript. For example, "P S 1 " indicates the inference output 22 of the SSL model 20 at step (r=1).

[0040] Referring to FIG. 4, in the first step (r=1), since the inference output 32 of the large-scale language model 30 has not been obtained, a predetermined dummy value (P L 1 =1) is used. As a result, the multiplier 26 multiplies the probability distribution P S 1 The probability distribution P of the SSL model 20 is S 1 is directly fed to the argmax layer 24. In the argmax layer 24, the probability distribution P S 1 The word (word ID) with the highest probability of being in 1 The output word h 1 is used as a speech recognition result and also as an input to the large-scale language model 30. That is, the output of the multiplier 26 is used as the recognition output of the speech recognition system 1.

[0041] In the large-scale language model 30, the token “CLS” that indicates the beginning of a sentence is 1 before h 0 h 0 (CLS)h 1 is input. At this time,

[1100] is used as the mask signal. According to the mask signal

[1100] , the large-scale language model 30 0 (CLS)h 1That is, the large-scale language model 30 uses a probability distribution P L 2 Output.

[0042] In the next step (r=2), the SSL model 20 calculates the probability distribution P S 2 The multiplier 26 outputs the probability distribution P S 2 and the probability distribution P of the large-scale language model 30 calculated in the previous step. L 2 (the inference output of the second word). In the argmax layer 24, the word (word ID) with the maximum probability included in the multiplication result (new probability distribution) is selected, and the word h 2 The output word h 2 is used as the speech recognition result and also as an input to the large-scale language model 30.

[0043] In the large-scale language model 30, the token “CLS” that indicates the beginning of a sentence is 1 before h 0 h 0 (CLS)h 1 h 2 is input. At this time,

[1110] is used as the mask signal. According to the mask signal

[1100] , the large-scale language model 30 0 (CLS)h 1 h 2 That is, the large-scale language model 30 uses a probability distribution P L 3 Output.

[0044] In the next step (r=3), the SSL model 20 calculates the probability distribution P S 3 The multiplier 26 outputs the probability distribution P S 3and the probability distribution P of the large-scale language model 30 calculated in the previous step. L 3 (the inference output of the third word). In the argmax layer 24, the word (word ID) with the maximum probability included in the multiplication result (new probability distribution) is selected, and the word h 3 The output word h 3 is used as the speech recognition result and also as an input to the large-scale language model 30.

[0045] In the large-scale language model 30, the token “CLS” that indicates the beginning of a sentence is 1 before h 0 h 0 (CLS)h 1 h 2 h 3 is input. At this time,

[1110] is used as the mask signal. According to the mask signal

[1110] , the large-scale language model 30 0 (CLS)h 1 h 2 h 3 That is, the large-scale language model 30 uses a probability distribution P L 4 Output.

[0046] In the next step (r=4), the SSL model 20 calculates the probability distribution P S 4 The multiplier 26 outputs the probability distribution P S 4 and the probability distribution P of the large-scale language model 30 calculated in the previous step. L 4 (the inference output of the fourth word). In the argmax layer 24, the word (word ID) with the maximum probability included in the multiplication result (new probability distribution) is selected, and the word h 4 The output word h 4 is used as the speech recognition result.

[0047] Next, an example of using BERT as a large-scale language model will be described. Fig. 5 is a schematic diagram showing another example of a model configuration of the speech recognition system 2 according to the present embodiment. Referring to Fig. 5, the speech recognition system 2 includes a multiplier 28, similar to the speech recognition system 1 shown in Fig. 3. Fig. 5 shows a configuration example using BERT as a large-scale language model 36.

[0048] As with the speech recognition system 1 shown in FIG. 3, in the speech recognition system 2, the inference output 22 of the SSL model 20 is corrected by the inference output 38 of the large-scale language model 36 and input to the large-scale language model 36, thereby performing processing on a word-by-word basis.

[0049] That is, the multiplier 28 multiplies the inference output 22 (probability distribution for words) of the SSL model 20 by the inference output 38 (probability distribution for words) of the large-scale language model 36, and outputs the multiplication result. The multiplication result is a probability distribution for words included in the vocabulary dictionary, similar to the inference output 22 of the SSL model 20 and the inference output 38 of the large-scale language model 36. The multiplier 26 multiplies the inference output 22 of the SSL model 20 by the inference output 38 corresponding to the inference output 22, among the outputs of the large-scale language model 36, on a word-by-word basis. The words selected based on the new probability distribution, which is the output of the multiplier 26, are input to the large-scale language model 36.

[0050] In this way, by sequentially feeding back the inference output of each model obtained earlier, the accuracy of the tokens input to the large-scale language model 36 can be improved, and as a result, the inference accuracy of the large-scale language model 36 can be improved.

[0051] 6 is a diagram for explaining an example of processing in the voice recognition system 2 according to the present embodiment. With reference to FIG. 6, the example of processing in the voice recognition system 2 will be described in further detail.

[0052] In Figure 6, as in Figure 4, the output state of the SSL model 20 (Prob CTC & Combine & argmax), the output state of the large-scale language model 36, and the state of the signal (mask signal) for masking the input to the large-scale language model 36 are shown for each processing step.

[0053] In the example shown in Figure 6, one sentence contains four words. The probability distribution P is the same as in Figure 4.

[0054] Referring to FIG. 6, in the first step (r=1), the probability distribution P S 1 The probability distribution P S 1 is the speech recognition result of the first word. At this stage, the inference output 38 of the large-scale language model 36 has not been obtained, so a predetermined dummy value (P L 1 = 1), the probability distribution P S 1 and the probability distribution P L 1 This multiplication results in a probability distribution P S 1 is corrected (however, at this stage, no substantial correction is made, and the probability distribution P S 1 is output as is). In the argmax layer 24, the multiplication result (probability distribution P S 1 ) is selected, and the first word in the word sequence h 1 The output word h 1 is used as the speech recognition result. That is, the output of the multiplier 28 is used as the recognition output of the speech recognition system 2.

[0055] In the next step (r=2), the SSL model 20 calculates the probability distribution P S 2In the previous step (r=1), the inferred word h 1 Since we have obtained the above, the large-scale language model 36 stores the token “CLS” which indicates the beginning of a sentence as the word h 0 As the word h 1 The result of prepending

[1011] and the mask signal

[1011] are input. For this input, the probability distribution P L 2 Once obtained, the multiplier 28 multiplies the probability distribution P S 2 and the probability distribution P L 2 This multiplication results in a probability distribution P S 2 is corrected. The argmax layer 24 calculates the corrected probability distribution P S 2 The word (word ID) with the highest probability in is selected, and the second word in the word sequence, h 2 The output word h 2 is used as the speech recognition result.

[0056] In the next step (r=3), the SSL model 20 calculates the probability distribution P S 3 In the previous step, the inferred word h 1 and the word h 2 , the large-scale language model 36 has the word h as a token “CLS” indicating the beginning of a sentence. 0 and the word h 1 , the word h 2 The concatenated result of the above and the mask signal

[1101] are input. For this input, the probability distribution P L 3 Once obtained, the multiplier 28 multiplies the probability distribution P S 3 and the probability distribution P L 3 This multiplication results in a probability distribution P S 3is corrected. The argmax layer 24 calculates the corrected probability distribution P S 3 The word (word ID) with the highest probability in is selected, and the third word in the word sequence, h 3 The output word h 3 is used as the speech recognition result.

[0057] In the next step (r=4), the SSL model 20 calculates the probability distribution P S 4 In the previous step, the inferred word h 1 , the word h 2 and the word h 3 , the large-scale language model 36 has the word h as a token “CLS” indicating the beginning of a sentence. 0 and the word h 1 , the word h 2 , the word h 3 The concatenated result of the above and the mask signal

[1110] are input. For this input, the probability distribution P L 4 Once obtained, the multiplier 28 multiplies the probability distribution P S 4 and the probability distribution P L 4 This multiplication results in a probability distribution P S 4 is corrected. The argmax layer 24 calculates the corrected probability distribution P S 4 The word (word ID) with the highest probability in is selected, and the fourth word in the word sequence, h 4 The output word h 4 is used as the speech recognition result.

[0058] Next, the above-mentioned process will be generalized and explained using mathematical expressions. Assume that one sentence contains R words. When processing on a word-by-word basis, (R+1) processes are required, including the initial state.

[0059] When GPT-2 is used as the large-scale language model 30, the speech recognition result of the rth word is r (where 1≦r≦t), the probability distribution P S r and the inference output P of the large-scale language model 30 for the rth word L r Using these, it can be shown as equation (1).

[0060] The word sequence output up to the current step is input to the large-scale language model 30. The word sequence H L r can be expressed as in equation (2). The probability distribution P L t+1 can be expressed as in (3). L denotes a large-scale language model 30.

[0061]

number

[0062] When BERT is used as the large-scale language model36, the speech recognition result of the rth word is r is the probability distribution P S r and the inference output P of the large-scale language model 36 for the rth word L r Using these, it can be expressed as in (4). In addition, the word sequence H L r can be expressed as equation (5).

[0063]

number

[0064] Table 1 below shows examples of results and corresponding mask signals output sequentially when GPT-2 is used as the large-scale language model 30 (example GPT) and when BERT is used as the large-scale language model 36 (example BERT). Table 1 shows an example of speech recognition of Chinese speech mainly used in Taiwan. Therefore, the large-scale language model outputs an inference output consisting of words in traditional Chinese.

[0065] [Table 1]

[0066] In Table 1, the rows corresponding to each of steps 0 to 5 show one or more words included in the sequence input to the large-scale language model.

[0067] When GPT-2 is used as the large-scale language model 30, a token string including the word to be processed at each step and the processing result obtained preceding the word is input to the large-scale language model 30.

[0068] On the other hand, when BERT is used as the large-scale language model 36, the tokens to be inferred from the token sequence output from the SSL model 20 are masked and input to the large-scale language model 36.

[0069] [B. Evaluation Example] Next, an example of evaluation of the recognition accuracy of the speech recognition system according to the present embodiment will be described. In the following evaluation example, an automatic speech recognition (ASR) task for a Taiwanese accented speech is assumed. Although the Taiwanese accented speech is similar to Mandarin Chinese, there may be significant differences in vocabulary (see, for example, http: / / old-site.clsp.jhu.edu / ws04 / groups / ws04casr).

[0070] The model was fine-tuned and evaluated using the Chinese (Simplified)-Chinese (Traditional) dataset (Zh-Tw dataset) from Common Voice (https: / / commonvoice.mozilla.org / zh-TW). The fine-tuning dataset was 3.3 hours long (3,507 utterances), and the large-scale language model was 2.9 hours long (2,895 utterances) for testing. The vocabulary size was 21,128 words. Speech input was sampled at 16 kHz. Due to data limitations, the size of the dataset used was small, so we focused on a text-based large-scale language model.

[0071] We used "wav2vec2-large-xlsr-533" (https: / / huggingface.co / facebook / wav2vec2-large-xlsr-53) as a pre-trained SSL model. The SSL model was fine-tuned using the Chinese (Simplified) - Chinese (Traditional) dataset (Zh-Tw dataset).

[0072] As a pre-trained large-scale language model, we used "gpt2-base-chinese" as an example of GPT-2 and "bert-base-chinese" (https: / / github.com / ckiplab / ckip-transformers) as an example of BERT. Both models were fine-tuned using text from the Chinese (Simplified) - Chinese (Traditional) dataset (Zh-Tw dataset).

[0073] The parameters of each model are shown in Table 2 below.

[0074] [Table 2]

[0075] As an evaluation index for the model, we used CER (Character Error Rate), which is an example of recognition accuracy, and RTF (Real-Time Factor), which is an index of processing speed. An evaluation example is shown in Table 3 below. The arrows in Table 3 show the connections (data flow) between models.

[0076] [Table 3]

[0077] Both "baseline" (related technology) and "proposed" (present embodiment) show the effect of a large-scale language model. As shown in Table 3, whether GPT-2 or BERT is used, performance that exceeds the indicators in the related technology is demonstrated. In addition, it can be seen that the performance of GPT-2 can be further improved by fine tuning.

[0078] Regarding processing speed, it has been shown that the processing speed decreases when GPT-2 is used, but it can be evaluated that it is within an acceptable range. In addition, when BERT is used, there is no substantial effect on the processing speed. It is possible that the processing speed can be improved by adopting parallel processing.

[0079] Next, an example of an inference output by the speech recognition system according to the present embodiment is shown below. Table 4 below shows two examples of inference output.

[0080] [Table 4]

[0081] In the inference output shown in example 1 of Table 4, there are errors in three words (the second, sixth, and seventh) in the sentence output by SSL model 20 (W2V).

[0082] According to the "baseline" (related technology), two words (the second and sixth) are corrected in the large-scale language model 30, but one word (the seventh) remains incorrect. In contrast, according to the "proposed" (present embodiment), all words in the sentence are correctly recognized.

[0083] Similarly, in the inference output shown in example 2 of Table 4, there are errors in three words (the fourth, tenth, and eleventh) in the sentence output by SSL model 20 (W2V).

[0084] According to the "baseline" (related technology), two words (the fourth and eleventh) are corrected in the large-scale language model 30, but one word (the twelfth) remains incorrect. In contrast, according to the "proposed" (present embodiment), all words in the sentence are correctly recognized.

[0085] [C. Hardware configuration example] Next, an example of a hardware configuration for realizing the voice recognition system according to the present embodiment will be described.

[0086] 7 is a schematic diagram showing an example of a hardware configuration for implementing a voice recognition system according to the present embodiment. The voice recognition system is implemented, for example, by an information processing device 100, which is an example of a computer.

[0087] Referring to FIG. 7, the information processing device 100 includes, as its main hardware components, a CPU (Central Processing Unit) 102, a GPU (Graphics Processing Unit) 104, a main memory 106, a display 108, a network interface (I / F) 110, an input device 112, an optical drive 114, and a secondary storage device 120.

[0088] The CPU 102 and / or the GPU 104 are processors that execute processes required to implement the speech recognition system. A plurality of CPUs 102 and a plurality of GPUs 104 may be provided, and each may have a plurality of cores.

[0089] The main memory 106 is a storage area that temporarily stores (or caches) program code, work data, and the like when the processor (CPU 102 and / or GPU 104) executes processing, and is composed of volatile memory devices such as DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory).

[0090] The display 108 is a display unit that outputs a user interface relating to processing, processing results, and the like, and is configured, for example, with an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display.

[0091] The network interface 110 exchanges data with any information processing device on the Internet or an intranet. As the network interface 110, any communication method such as Ethernet (registered trademark), wireless LAN (Local Area Network), Bluetooth (registered trademark), etc. can be adopted.

[0092] The input device 112 is a device that accepts instructions and operations from a user, and is composed of, for example, a keyboard, a mouse, a touch panel, a pen, etc. The input device 112 includes a device that collects audio signals, such as a microphone.

[0093] The optical drive 114 reads out information stored in an optical disk 116 such as a CD-ROM (Compact Disc Read Only Memory) or a DVD (Digital Versatile Disc) and outputs the information to other components. The optical disk 116 is an example of a non-transitory recording medium, and is distributed in a state in which an arbitrary program is stored in a non-volatile manner. The optical drive 114 reads out the program from the optical disk 116 and installs it in the secondary storage device 120 or the like, so that the computer functions as a voice recognition system. Therefore, the subject of the present invention may be the program itself installed in the secondary storage device 120 or the like, or a recording medium such as the optical disk 116 that stores a program for realizing the functions and processing according to the present embodiment.

[0094] FIG. 7 shows an optical recording medium such as an optical disk 116 as an example of a non-transient recording medium, but the present invention is not limited to this. Semiconductor recording media such as flash memory, magnetic recording media such as hard disks or storage tapes, and magneto-optical recording media such as MOs (Magneto-Optical Disks) may also be used.

[0095] The secondary storage device 120 stores programs and data necessary for the computer to function as the information processing device 100. For example, the secondary storage device 120 is configured as a non-volatile storage device such as a hard disk or a solid state drive (SSD).

[0096] More specifically, the secondary storage device 120 stores an SSL model program 122, a large scale language model program 124, a control program 126, and an OS (Operating System) 128.

[0097] The SSL model program 122 contains computer readable instructions for generating the SSL model 20. The large language model program 124 contains computer readable instructions for generating the large language models 30, 36. The control program 126 contains computer readable instructions corresponding to operations for generating control signals and controlling the sequence of inputs to the large language models 30, 36.

[0098] Some of the libraries and functional modules required when the processor (CPU 102 and / or GPU 104) executes a program may be replaced with libraries or functional modules provided as standard by OS 128. In this case, the program alone does not include all of the program modules required to realize the corresponding function, but the desired processing can be realized by installing it in the execution environment of OS 128. Even a program that does not include some of such libraries or functional modules is included in the technical scope of the present invention.

[0099] Furthermore, these programs may not only be distributed by being stored in any of the above-mentioned recording media, but may also be distributed by being downloaded from a server device or the like via the Internet or an intranet.

[0100] FIG. 7 shows an example in which information processing device 100 is configured using a single computer, but the present invention is not limited to this. A voice recognition system may be realized by having multiple computers connected via a computer network work together explicitly or implicitly.

[0101] All or part of the functions realized by the processor (CPU 102 and / or GPU 104) executing the program may be realized using a hard-wired circuit such as an integrated circuit. For example, they may be realized using an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0102] A person skilled in the art would be able to realize the information processing device 100 according to the present embodiment by appropriately using technology appropriate for the era in which the present invention is implemented.

[0103] [D. Processing Procedure Example] Next, an example of a processing procedure of the voice recognition system 1 according to the present embodiment will be described.

[0104] Fig. 8 is a flowchart showing an example of a processing procedure in the speech recognition system 1 according to the present embodiment. Each step shown in Fig. 8 may be realized by the CPU 102 and / or GPU 104 of the information processing device 100 executing the SSL model program 122, the large-scale language model program 124, and the control program 126.

[0105] 8, the information processing device 100 inputs the input sequence 4 to the SSL model 20 (step S100). The inference output 22 from the SSL model 20 (probability distribution P S ) is obtained (YES in step S102), the information processing device 100 performs the inference output 32 (probability distribution P L ) is present (step S104).

[0106] If there is no inference output 32 from the large-scale language model 30 in the previous process (NO in step S104), the information processing device 100 calculates the probability distribution P L A predetermined dummy value (=1) is set to (step S106).

[0107] If there is an inference output 32 from the large-scale language model 30 in the previous process (YES in step S104), or after execution of step S106, the information processing device 100 performs the probability distribution P S and the probability distribution P obtained in the previous process L Multiplying by and gives a new probability distribution P S (Step S108). The information processing device 100 obtains a new probability distribution P S The word h having the maximum probability contained in is output as the speech recognition result (step S110).

[0108] The information processing device 100 inputs the string of words h (token string) that have been output so far into the large-scale language model 30 (step S112). At this time, a token indicating the beginning of a sentence (CLS) is added to the beginning of the string of words h to be input. The information processing device 100 outputs an inference output 32 (probability distribution P L ) is temporarily stored (step S114).

[0109] The information processing device 100 receives the inference output 22 (probability distribution P S It is determined whether the inference output 22 from the SSL model 20 is continuing (step S116). If the inference output 22 from the SSL model 20 is continuing (YES in step S116), the processes from step S104 onwards are repeated.

[0110] If the inference output 22 from the SSL model 20 is not continuing (NO in step S116), the information processing device 100 outputs the string of words h (token string) that has been output so far as the speech recognition result (step S118), and the process then ends.

[0111] Fig. 9 is a flowchart showing an example of a processing procedure in the speech recognition system 2 according to the present embodiment. Each step shown in Fig. 9 may be realized by the CPU 102 and / or GPU 104 of the information processing device 100 executing the SSL model program 122, the large-scale language model program 124, and the control program 126.

[0112] 9, the information processing device 100 inputs the input sequence 4 to the SSL model 20 (step S200). The inference output 22 from the SSL model 20 (probability distribution P S When the result of step S202 is YES, the information processing device 100 determines whether or not the word h output so far exists (step S204).

[0113] If there is no word h that has been output so far (NO in step S204), the information processing device 100 uses the probability distribution P L A predetermined dummy value (=1) is set as the dummy value (step S206). Then, the process of step S212 is executed.

[0114] If there is a word h that has been output so far (YES in step S204), the information processing device 100 inputs the string of words h that have been output so far and the corresponding mask to the large-scale language model 30 (step S208). At this time, a token indicating the beginning of a sentence (CLS) is added to the beginning of the string of words h to be input. The information processing device 100 uses the probability distribution P L Then, an inference output 32 is obtained from the large-scale language model 30 (step S210).

[0115] The information processing device 100 uses the probability distribution P S and the probability distribution P obtained in this process L Multiplying by and gives a new probability distribution P S (Step S212). The information processing device 100 obtains a new probability distribution P SThe word h having the maximum probability contained in is output as the speech recognition result (step S214).

[0116] The information processing device 100 receives the inference output 22 (probability distribution P S It is determined whether the inference output 22 from the SSL model 20 is continuing (step S216). If the inference output 22 from the SSL model 20 is continuing (YES in step S216), the processes from step S202 onwards are repeated.

[0117] If the inference output 22 from the SSL model 20 is not continuing (NO in step S216), the information processing device 100 outputs the string of words h (token string) that has been output so far as the speech recognition result (step S218), and the process then ends.

[0118] [E. Modifications] In the above explanation, an example was given in which the process is repeated according to the number of tokens included in the sentence, but the number of times the process is performed and the tokens to be masked out may be changed as appropriate.

[0119] For example, when one or more specific words appear in the inference output output from the large-scale language model, processing of the sentence may be terminated at the point when the specific words appear.

[0120] The specific word may be, for example, one or more words that are unlikely to cause an error in the inference output, or one or more words that have some meaning. Table 5 shows an example in which the process is terminated after two times.

[0121] [Table 5]

[0122] Alternatively, when a specific word or words appear in the inference output from the large-scale language model, the token to be processed may be skipped or jumped at the point where the specific word or words appear. By processing the token to be processed earlier than the original order, the time required for inference processing can be shortened. Table 6 shows an example in which a sentence is processed only twice in total by processing the first token and then the last token.

[0123] [Table 6]

[0124] It should be noted that a control unit may be added for determining whether to abort, skip, or jump the above-mentioned processing.

[0125] In this way, when the inference output satisfies a predetermined condition, a token that is different from the token to be processed in the previous process may be changed to the token to be processed.

[0126] [F. Advantages] According to this embodiment, progressive error correction is possible by correcting the speech recognition result by the SSL model 20 using the inference output of a large-scale language model. By employing such progressive error correction, it is possible to solve problems specific to recognition performance due to not only the standard speech used to train the language model, but also non-standard speech such as different accents and dialects, which may occur in real applications.

[0127] According to this embodiment, since processing can be performed on a word-by-word basis in a large-scale language model, error correction can be performed on a word-by-word basis, and speech recognition accuracy (inference accuracy) can be improved.

[0128] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, not by the description of the embodiments described above, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0129] 1,2,3 Speech recognition system, 4 Input sequence, 6 Token, 8 Sentence, 10 Multiplication result, 20 SSL model, 22,32,38 Inference output, 24 Argmax layer, 26,28,34 Multiplier, 30,36 Large-scale language model, 40 Control signal generation unit, 100 Information processing device, 102 CPU, 104 GPU, 106 Main memory, 108 Display, 110 Network interface, 112 Input device, 114 Optical drive, 116 Optical disk, 120 Secondary storage device, 122 Model program, 124 Large-scale language model program, 126 Control program, 128 OS.

Claims

1. a first trained inference model capable of outputting tokens corresponding to words included in a speech signal on a token-by-token basis; a second trained inference model that outputs one or more corresponding tokens by inference based on one or more input tokens; a multiplier that multiplies a first inference output of the first trained inference model by a second inference output, the second inference output being an output of the second trained inference model and corresponding to the first inference output, on a word-by-word basis; A speech recognition system, wherein words selected based on a new probability distribution that is the output of the multiplier are input to the second trained inference model.

2. 2. The speech recognition system of claim 1, wherein an output of the multiplier is utilized as a recognition output of the speech recognition system.

3. The speech recognition system according to claim 1 or 2, wherein the second trained inference model has been pre-trained on at least the task of predicting the next word for an input sequence using a corpus of a target language of speech recognition.

4. The speech recognition system of claim 3 , wherein the second trained inference model is fine-tuned using dialect data of the target language.

5. 1. A computer implemented method for speech recognition, comprising: outputting, on a token-by-token basis, tokens corresponding to words included in the speech signal using a first trained inference model; using the second trained inference model to infer from the input of the one or more tokens, and outputting the corresponding one or more tokens; multiplying, on a word-by-word basis, a first inference output of the first trained inference model by a second inference output of the second trained inference model, the second inference output corresponding to the first inference output; and inputting the selected words based on the new probability distribution resulting from the multiplication into the second trained inference model.

6. A program for performing speech recognition, comprising: outputting, on a token-by-token basis, tokens corresponding to words included in the speech signal using a first trained inference model; using the second trained inference model to infer from the input of the one or more tokens, and outputting the corresponding one or more tokens; multiplying, on a word-by-word basis, a first inference output of the first trained inference model by a second inference output of the second trained inference model, the second inference output corresponding to the first inference output; and inputting the selected words based on the new probability distribution resulting from the multiplication into the second trained inference model.

Images