Voice recognition device, voice recognition method and program

Abstract

To provide a voice recognition device, a voice recognition method and a program, which can further improve accuracy of voice recognition.SOLUTION: A voice recognition device includes: an acquisition unit for acquiring a frame of a voice stream; a streaming feature generation unit for generating a first feature from the frame by using a streaming encoder; a streaming character generation unit for generating a first character from the first feature by using a streaming decoder; a non-streaming feature generation unit for generating a second feature string from a first feature string obtained by combining the first features of the plurality of frames by using a non-streaming encoder; a streaming character generation unit for generating a second character string from the second feature string by using a plurality of non-streaming decoders; and a leaning unit for performing knowledge distillation between the streaming encoder and the non-streaming encoder based on the first feature string and the second feature string.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a speech recognition device, a speech recognition method, and a program. [Background technology]

[0002] End-to-end (E2E) automatic speech recognition (ASR) operates in two modes: streaming and non-streaming (see, for example, Non-Patent Document 1). Streaming ASR and non-streaming ASR are known to each have advantages and disadvantages. Streaming ASR can process speech frames in real time as they are received, but speech recognition accuracy tends to be lower than that of non-streaming ASR. On the other hand, non-streaming ASR cannot process speech frames in real time and must wait for the entire speech utterance, but speech recognition accuracy tends to be higher than that of streaming ASR. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Arun Narayanan, Tara N. Sainath, Ruoming Pang, Jiahui Yu, Chung-Cheng Chiu, Rohit Prabhavalkar, Ehsan Variani, Trevor Strohman, “Cascaded encoders for unifying streaming and non-streaming ASR,” in Proc. ICASSP, 2021, DOI: org / 10.48550 / arXiv.2010.14606. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques have room for improvement in improving speech recognition accuracy by combining streaming ASR and non-streaming ASR.

[0005] The present invention has been made in consideration of the above circumstances, and one of its objects is to provide a voice recognition device, a voice recognition method, and a program that can further improve the accuracy of voice recognition. [Means for solving the problem]

[0006] The speech recognition device, speech recognition method, and program according to the present invention employ the following configurations. (1) A first example of the present invention is a speech recognition device including: an acquisition unit that acquires frames per unit time of an audio stream; a streaming feature generation unit that generates first features from the frames using a streaming encoder; a streaming character generation unit that generates a first character from the first features using a streaming decoder; a non-streaming feature generation unit that generates a second feature sequence from a first feature sequence that combines the first features of multiple frames using a non-streaming encoder; a streaming character generation unit that generates a second character string from the second feature sequence using multiple non-streaming decoders; and a learning unit that performs knowledge distillation between the streaming encoder and the non-streaming encoder based on the first feature sequence and the second feature sequence.

[0007] (2) A second example of the present invention is that in the first example, the learning unit performs the knowledge distillation between the streaming encoder and the non-streaming encoder so that the first feature sequence is similar to the second feature sequence.

[0008] (3) A third example of the present invention is that in the first or second example, the learning unit further performs the knowledge distillation between the streaming decoder and the plurality of non-streaming decoders based on the likelihood of a first string in which the first characters generated from each of the plurality of first features are arranged in chronological order, and the likelihood of the second string generated from the second feature sequence.

[0009] (4) A fourth example of the present invention is that in the third example, the streaming decoder includes at least a predictor that predicts the first character, and the plurality of non-streaming decoders include at least an attention decoder that is a decoder that includes an attention mechanism, and the learning unit performs the knowledge distillation between the predictor and the attention decoder so that the likelihood of the first string, which is a chronological arrangement of the first characters predicted by the predictor, is similar to the likelihood of the second string output by the attention decoder.

[0010] (5) A fifth example of the present invention is a speech recognition method including: acquiring frames per unit time of an audio stream; generating first features from the frames using a streaming encoder; generating a first character from the first features using a streaming decoder; generating a second feature sequence from a first feature sequence combining the first features of a plurality of the frames using a non-streaming encoder; generating a second character string from the second feature sequence using a plurality of non-streaming decoders; and performing knowledge distillation between the streaming encoder and the non-streaming encoder based on the first feature sequence and the second feature sequence.

[0011] (6) A sixth example of the present invention is a program to be executed by a computer, the program including: acquiring frames per unit time of an audio stream; generating a first feature from the frames using a streaming encoder; generating a first character from the first feature using a streaming decoder; generating a second feature sequence from a first feature sequence combining the first features of a plurality of the frames using a non-streaming encoder; generating a second character string from the second feature sequence using a plurality of non-streaming decoders; and performing knowledge distillation between the streaming encoder and the non-streaming encoder based on the first feature sequence and the second feature sequence. [Effects of the Invention]

[0012] According to the above example, the accuracy of speech recognition can be further improved. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a configuration diagram of a speech recognition device 100 according to an embodiment. [Figure 2] 10 is a flowchart showing the flow of a series of processes by a processing unit 120 according to the embodiment. [Figure 3] FIG. 1 is a diagram for explaining the configuration of a streaming ASR module and a non-streaming ASR module, and a method of knowledge distillation thereof. [Figure 4] FIG. 10 is a diagram showing the absolute and relative values ​​of the character error rate for each method when using a Japanese corpus. [Figure 5] FIG. 10 is a diagram showing changes in word error rate with respect to block size for the method of this embodiment and the comparative method. [Figure 6] FIG. 10 is a diagram showing changes in word error rate with respect to block size for the method of this embodiment and the comparative method. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of a speech recognition device, a speech recognition method, and a program according to the present invention will be described with reference to the drawings.

[0015] [Configuration of speech recognition device] 1 is a configuration diagram of a speech recognition device 100 according to an embodiment. The speech recognition device 100 may be a single device, or may be a system in which multiple devices connected via a network NW such as a LAN (Local Area Network) or a WAN (Wide Area Network) operate in cooperation with each other. In other words, the speech recognition device 100 may be implemented by multiple computers (processors) included in a distributed computing system or a cloud computing system.

[0016] The speech recognition device 100 includes, for example, a processing unit 120 and a storage unit 130. The processing unit 120 includes, for example, an acquisition unit 121, a streaming feature generation unit 122, a streaming character generation unit 123, a non-streaming feature generation unit 124, a non-streaming character generation unit 125, and a learning unit 126.

[0017] The components of the processing unit 120 are realized by a processor such as a central processing unit (CPU) or a graphics processing unit (GPU) executing a program stored in the storage unit 130. The components of the processing unit 120 may be realized by hardware such as a large scale integration (LSI), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a system on chip (SOC), or may be realized by a combination of software and hardware.

[0018] The processing unit 120 uses a streaming ASR module (also referred to as a streaming ASR model) and a non-streaming ASR module (also referred to as a non-streaming ASR model) to recognize a character string representing the content of an utterance from the audio data of the utterance (hereinafter also referred to as an audio stream). The utterance here typically refers to an utterance made by a human (user), but is not limited to this and may also be an artificial utterance made by, for example, a robot, a machine, or a computer. In other words, the utterance may be an utterance artificially created using speech synthesis technology.

[0019] The streaming ASR module includes a streaming encoder ENC1 and a streaming decoder DEC1. The non-streaming ASR module includes a non-streaming encoder ENC2 and multiple non-streaming decoders DEC2 of different types. These various encoders and decoders will be described in detail later.

[0020] The storage unit 130 is realized by, for example, a hard disk drive (HDD), flash memory, electrically erasable programmable read only memory (EEPROM), read only memory (ROM), random access memory (RAM), etc. The storage unit 130 stores firmware, application programs, etc. Furthermore, the storage unit 130 stores programs, algorithms, or architectures that define streaming ASR modules and non-streaming ASR modules.

[0021] [Processing flow] The processing contents of each component of the processing unit 120 will be described below using a flowchart. Fig. 2 is a flowchart showing the flow of a series of processes by the processing unit 120 according to the embodiment. The processing of this flowchart may be repeatedly executed at a predetermined cycle.

[0022] First, the acquisition unit 121 acquires frames (also called blocks) of an audio stream from the microphone 10 (step S102). A frame is a part or a fragment of an audio stream divided into units of time.

[0023] The streaming feature generation unit 122 generates a first feature h from frames of the audio stream X using the streaming encoder ENC1 included in the streaming ASR module (step S104).

[0024] Next, the streaming character generation unit 123 generates characters from the first feature h using the streaming decoder DEC1 included in the streaming ASR module (step S106).

[0025] Next, the acquiring unit 121 determines whether or not the speech has ended (step S108). For example, the acquiring unit 121 may perform voice activity detection (VOD) on the audio stream and determine whether or not the speech has ended based on the result of the voice activity detection.

[0026] If the speech is continuing, the acquisition unit 121 acquires a new frame of the audio stream from the microphone 10. In response to this, the streaming feature generation unit 122 generates a new first feature h from the new frame, and the streaming character generation unit 123 generates a new character from the new first feature h. In this way, characters are generated for each frame using the streaming ASR module frames until the speech is interrupted. In other words, a series of character strings (i.e., words, sentences, etc.) are generated until the speech is interrupted. A character string in which characters generated from each of the multiple first features h are arranged in chronological order is an example of a "first character string."

[0027] On the other hand, if the speech continues, the non-streaming feature generation unit 124 generates a first feature sequence H in which the first features h of the frames acquired until the speech ends are arranged in chronological order. stis generated (step S110).

[0028] Next, the non-streaming feature generator 124 generates a first feature sequence H using a non-streaming encoder ENC2 included in the non-streaming ASR module. st From the second feature column H te (Step S112). For example, the first feature sequence H st and the second feature sequence H te may be represented by a matrix or a vector, which have the same number of dimensions.

[0029] Next, the non-streaming character generation unit 125 generates a second feature sequence H using at least one of the plurality of non-streaming decoders DEC2 included in the non-streaming ASR module. te A character string is generated from the second feature string H (step S114). te is an example of a "second string."

[0030] Next, the learning unit 126 learns the non-streaming ASR module and the streaming ASR module using knowledge distillation (step S116).

[0031] Knowledge distillation generally refers to the process of transferring knowledge from a large-scale model to a small-scale model. In this embodiment, knowledge distillation involves transferring knowledge from the non-streaming ASR module (student) to the streaming ASR module (teacher), where the non-streaming ASR module is the student. Specifically, the non-streaming ASR module and the streaming ASR module are optimized so that the output of the student non-streaming ASR module is similar to the output of the teacher streaming ASR module. Knowledge distillation that focuses on such similarities is also referred to as similarity-preserving knowledge distillation (sp-KD).

[0032] [Knowledge distillation process] The knowledge distillation process will be specifically described below. Figure 3 is a diagram for explaining the configuration of a streaming ASR module and a non-streaming ASR module, and the method of knowledge distillation for them.

[0033] As described above, the streaming ASR module includes a streaming encoder ENC1 and a streaming decoder DEC1.

[0034] The streaming encoder ENC1 may be, for example, an encoder with M stacked encoder layers, as described in References 1 to 4. By employing such an encoder, frames of an audio stream can be processed efficiently.

[0035] Reference 1: Emiru Tsunoo, Yosuke Kashiwagi, Toshiyuki Kumakura, Shinji Watanabe, “Transformer ASR with Contextual Block Processing,” in Proc. ASRU, 2019, pp. 427-433.

[0036] Reference 2: Tianzi Wang, Yuya Fujita, Xuankai Chang, Shinji Watanabe, “Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models,” in Proc. Interspeech, 2021.

[0037] Reference 3: Niko Moritz, Takaaki Hori, Jonathan Le Roux, “Streaming automatic speech recognition with the transformer model,” in Proc. ICASSP, 2020.

[0038] Reference 4: Yangyang Shi, Yongqiang Wang, Chunyang Wu, Ching-Feng Yeh, Julian Chan, Frank Zhang, Duc Le, Mike Seltzer, “Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition,” in Proc. ICASSP, 2021.

[0039] The streaming decoder DEC1 is connected to the bus of the streaming encoder ENC1. The streaming decoder DEC1 may be, for example, a recurrent neural network transducer (RNN-T)-based decoder as described in Reference 5. Hereinafter, the RNN-T-based decoder will be specifically referred to as an "RNN-T decoder."

[0040] Reference 5: Alex Graves, “Sequence Transduction with Recurrent Neural Networks,” in Proc. ICML, 2012.

[0041] The RNN-T decoder includes, for example, a joiner M1-1, a predictor M1-2, and a unit M1-3 whose activation function is a softmax function.

[0042] The combiner M1-1 may be implemented by, for example, a feedforward network. The combiner M1-1 receives a first feature h converted from a frame by the streaming encoder ENC1 and a first feature predicted by the predictor M1-2. The combiner M1-1 combines the two input first features and outputs the combined vector to the softmax function unit M1-3.

[0043] The predictor M1-2 may be implemented by an RNN including a Long Short-Term Memory (LSTM). The predictor M1-2 receives the previous character y output by the unit M1-3. u-1 is fed back. The predictor M1-2 uses the first feature h so far and the previous character y u-1 The unit M1-3 converts each component of the vector output by the combiner M1-1 into a likelihood and outputs the likelihood. Each component corresponds to a character, and the character corresponding to the component with the highest likelihood is determined to be the character corresponding to the frame input to the streaming decoder DEC1.

[0044] As described above, the non-streaming ASR module includes a non-streaming encoder ENC2 and a plurality of non-streaming decoders DEC2 of different types.

[0045] The non-streaming encoder ENC2 is connected in series with the streaming encoder ENC1. The non-streaming encoder ENC2 may be, for example, an encoder with N encoder layers stacked as described in Reference 6.

[0046] Reference 6: Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, Ruoming Pang, “Conformer: Convolution-augmented Transformer for Speech Recognition,” in Proc. Interspeech, 2020, pp. 5036￢5040.

[0047] Each of the non-streaming decoders DEC2 is connected to a bus of the non-streaming encoder ENC2. The non-streaming decoders DEC2 may include, for example, an RNN-T decoder DEC2-1, an attention decoder DEC2-2, a CTC (Connectionist Temporal Classification) decoder DEC2-3, and an MLM (Masked Language Model) decoder DEC2-4.

[0048] The RNN-T decoder DEC2-1 is an RNN-T-based decoder as described above, and includes, for example, a combiner M2-1, a predictor M2-2, and a unit M2-3 whose activation function is a softmax function.

[0049] The attention decoder DEC2-2 is an attention mechanism-based decoder as described in Reference 7.

[0050] Reference 7: Yui Sudo, Muhammad Shakeel, Brian Yan, Jiatong Shi, Shinji Watanabe, “4D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoders,” ArXiv, 2022.

[0051] The CTC decoder DEC2-3 is a CTC-based decoder as described in Reference 7.

[0052] The MLM decoder DEC2-4 is an MLM-based decoder as described in Reference 7.

[0053] The streaming feature generation unit 122 inputs a frame of the audio stream X to the streaming encoder ENC1. In response to the input frame, the streaming encoder ENC1 outputs a first feature h. In other words, the streaming encoder ENC1 converts the frame into a first feature h by embedding the frame in a certain feature space (also called a latent space).

[0054] The streaming character generation unit 123 inputs the first feature h converted from each frame to the streaming decoder DEC1. In response to the input of the first feature h, the streaming decoder DEC1 outputs a character y with a likelihood. In other words, the streaming decoder DEC1 converts the first feature h into a character y by embedding the first feature h in a certain feature space.

[0055] The non-streaming feature generation unit 124 generates a first feature sequence H in which the first features h of the frames acquired until the end of the utterance are arranged in chronological order. st and generate this first feature sequence H st is input to the non-streaming encoder ENC2. The non-streaming encoder ENC2 inputs the first feature sequence H st In response to the input, the second feature column H teIn other words, the non-streaming encoder ENC2 outputs a first feature sequence H st By embedding the first feature column H st the second feature column H te Convert to.

[0056] The non-streaming character generation unit 125 selects at least one decoder from among the RNN-T decoder DEC2-1, the attention decoder DEC2-2, the CTC decoder DEC2-3, and the MLM decoder DEC2-4, and outputs the second feature sequence H te Enter.

[0057] For example, the non-streaming character generator 125 may select a decoder depending on the application in which ASR is used and the computer resources.

[0058] Each of the RNN-T decoder DEC2-1, the attention decoder DEC2-2, the CTC decoder DEC2-3, and the MLM decoder DEC2-4 receives the second feature sequence H te In other words, these four decoders DEC2-1 to DEC2-4 output a character string Y with likelihood in response to the input of the second feature sequence H te By embedding the second feature sequence H te Each second feature in the sequence H is converted to a character y. te A string Y is generated that contains as many characters y as there are for each second feature contained in

[0059] Furthermore, the non-streaming character generation unit 125 selects a plurality of decoders from among the RNN-T decoder DEC2-1, the attention decoder DEC2-2, the CTC decoder DEC2-3, and the MLM decoder DEC2-4, and outputs the second feature sequence H te For example, the non-streaming character generation unit 125 may input the second feature sequence H to the RNN-T decoder DEC2-1, the attention decoder DEC2-2, and the CTC decoder DEC2-3, which have relatively high ASR accuracy among the four decoders.te Each of these three decoders generates the second feature sequence H te into a character string Y, the non-streaming character generation unit 125 may combine the three character strings Y to generate a character string Y that combines the most likely character y.

[0060] The learning unit 126 calculates three types of losses and optimizes the streaming ASR module and the non-streaming ASR module based on the three types of losses.

[0061] [Online Loss] First, the learning unit 126 calculates the online loss L onl and offline loss L off Calculate the online loss L onl is a loss related to the likelihood of the output from the streaming ASR module, and is calculated, for example, by Equation (1).

[0062]

number

[0063] As shown in equation (1), the online loss L onl is the negative log-likelihood P onl The likelihood P onl is the likelihood of the character y per frame output by the streaming decoder DEC1. That is, the online loss L onl is the likelihood P of character y corresponding to each of all frames acquired until the end of the utterance. onl It is calculated by adding up the

[0064] [Offline Loss] Offline Loss L off is a loss related to the likelihood of the output from the non-streaming ASR module, and is calculated, for example, by Equation (2).

[0065]

number

[0066] As shown in equation (2), the offline loss L off is expressed as a weighted sum of the losses of the RNN-T decoder DEC2-1, the attention decoder DEC2-2, the CTC decoder DEC2-3, and the MLM decoder DEC2-4. ctc , λ rnnt , λ att , and λ mlm is a tunable hyperparameter that may be determined through experimentation or simulation.

[0067] Loss L of CTC decoder DEC2-3 ctc is calculated, for example, by the formula (3).

[0068]

number

[0069] As shown in equation (3), the loss L of the CTC decoder DEC2-3 ctc is the negative log-likelihood P ctc The likelihood P ctc is the likelihood of each character y included in the string Y output by the CTC decoder DEC2-3. In other words, the loss L ctc is the likelihood P of all characters y included in the string Y output by the CTC decoder DEC2-3. ctc It is calculated by adding up the

[0070] Loss L of RNN-T decoder DEC2-1 rnnt is calculated, for example, by the formula (4).

[0071]

number

[0072] As shown in equation (4), the loss L of the RNN-T decoder DEC2-1 rnnt is the negative log-likelihood P rnnt The likelihood P rnnt is the likelihood of each character y included in the string Y output by the RNN-T decoder DEC2-1. In other words, the loss L rnnt is the likelihood P of all characters y included in the string Y output by the RNN-T decoder DEC2-1. rnnt It is calculated by adding up the

[0073] Loss L of attention decoder DEC2-2 att is calculated, for example, by the formula (5).

[0074]

number

[0075] As shown in Equation (5), the loss L of the attention decoder DEC2-2 att is the negative log-likelihood P att The likelihood P att is the likelihood of each character y included in the string Y output by the attention decoder DEC2-2. In other words, the loss L att is the likelihood P of all characters y included in the string Y output by the attention decoder DEC2-2. att It is calculated by adding up the

[0076] Loss L of MLM decoder DEC2-4 mlm is calculated, for example, by the formula (6).

[0077]

number

[0078] As shown in equation (6), the loss L mlm is the negative log-likelihood Pmlm It is expressed by y mask is a masked character among the multiple characters (tokens) contained in the string Y output by the MLM decoder DEC2-4. mask is the remaining unmasked characters (tokens) y obs The likelihood P mlm is the masked character y in the string Y output by the MLM decoder DEC2-4. mask In other words, the loss L of the MLM decoder DEC2-4 mlm is the masked letter y mask The likelihood P mlm It is calculated by adding up the

[0079] [Knowledge distillation loss] Next, the learning unit 126 calculates the knowledge distillation loss L dist Calculate the knowledge distillation loss L dist is calculated, for example, by the formula (7).

[0080]

number

[0081] G treated as a teacher te is the second feature sequence H output by the non-streaming encoder ENC2. te It represents the G who is treated as a student. st is the first feature sequence H, which is a time series of the first features h output by the streaming encoder ENC1. st represents.

[0082] ζ adds a similar pair of (l, l′) layers at the end of each of the streaming encoder ENC1 and the non-streaming encoder ENC2. In equation (7), F is the Frobenius norm, and G te (l) and G st (l′) Calculate the mean element-wise squared difference between the matrices B×B and Gte (l) and G st (l′) represents the size of the matrix.

[0083] The learning unit 126 generates a first feature sequence H corresponding to the output of the streaming encoder ENC1. st and the second feature sequence H corresponding to the output of the non-streaming encoder ENC2. te The difference between (G te (l) -G st (l′) ) and calculate it as (G te (l) -G st (l′) ) is the knowledge distillation loss between encoders L dist The encoder of each ASR module calculates the knowledge distillation loss L dist In other words, the first feature sequence H st is the teacher (also called target) second feature sequence H te The parameters of each encoder are optimized similarly to:

[0084] Furthermore, the learning unit 126 may calculate the knowledge distillation loss between decoders in addition to the knowledge distillation loss between encoders. In this case, in Equation (7), G te is the likelihood P of the string Y output by the attention decoder DEC2-2. att G, who is treated as a student st is the likelihood P of a string Y in which the characters y are arranged in time series, output by the predictor M1-2 of the streaming decoder DEC1. prd Represents.

[0085] The learning unit 126 calculates the likelihood P corresponding to the output of the predictor M1-2 of the streaming decoder DEC1. prd and the likelihood P corresponding to the output of the attention decoder DEC2-2 att The difference between (G te (l) -G st (l′)) and calculate it as (G te (l) -G st (l′) ) is the knowledge distillation loss between decoders L dist The decoder and encoder of each ASR module calculate the knowledge distillation loss L dist That is, the likelihood of being a student P prd is the teacher (also called target) likelihood P att The parameters of the ASR module are optimized to be similar to the above.

[0086] Finally, the learning unit 126 calculates the online loss L as expressed in Equation (9). onl and offline loss L off and knowledge distillation loss L dist Total loss L for multitasking combined with mtl Calculate the knowledge distillation loss L dist is the knowledge distillation loss L between the encoders. dist and the knowledge distillation loss between decoders L dist and may be included.

[0087]

number

[0088] λ onl , λ off , and λ dist is a weighting coefficient. The learning unit 126 uses the online loss L onl and offline loss L off and knowledge distillation loss L dist The weighted sum of the total loss L mtl It may be calculated as:

[0089] Then, the learning unit 126 calculates the total loss L mtl The parameters of the streaming ASR module and the non-streaming ASR module are optimized using a gradient method or the like so that σ becomes small.

[0090] [Comparison of the method of this embodiment with other methods] Below, we will explain the results of comparing the method of this embodiment with other methods. Figure 4 is a diagram showing the absolute (abs.) and relative (rel.) values ​​of the character error rate (CER) of each method when using a Japanese corpus. ST1 to ST5 represent the CER of the streaming ASR module of each method, and NST1 to NST5 represent the CER of the non-streaming ASR module of each method.

[0091] ST1 and NST1 are methods in which the streaming ASR module and the non-streaming ASR module are not connected in series, in other words, ST1 and NST1 are methods in which the output of the streaming encoder ENC1 is not processed by the non-streaming encoder ENC2.

[0092] ST2 and NST2 are methods that connect a streaming ASR module and a non-streaming ASR module in series. However, both ASR modules have only one decoder.

[0093] ST3 and NST3 are methods that connect a streaming ASR module and a non-streaming ASR module in series. The streaming ASR module has one decoder and the non-streaming ASR module has four decoders.

[0094] ST4 and NST4 are methods in which one streaming ASR module with a decoder and four non-streaming ASR modules with a decoder are connected in series, and furthermore, the output of the attention decoder of the four decoders included in the non-streaming ASR module is subjected to knowledge distillation to the predictor of the decoder of the streaming ASR module.

[0095] ST5 and NST5 are methods of this embodiment, which, in addition to the methods of ST4 and NST4, further perform knowledge distillation from the output of the encoder of the non-streaming ASR module to the encoder of the streaming ASR module.

[0096] As shown in Figure 4, compared to the baseline ST1, all ST2-5 had improved CER, and in particular, ST5 of the present method had a significantly improved CER in all evaluations from eval.1 to eval.3. The same is true for NST. In other words, compared to the baseline NST1, all NST2-5 had improved CER, and in particular, NST5 of the present method had a significantly improved CER in all evaluations from eval.1 to eval.3.

[0097] 5 and 6 are diagrams showing the change in word error rate (WER) versus block size (frame size) for the method of this embodiment and the comparative method. FIG. 5 shows the evaluation results for the performance of the non-streaming ASR module, and FIG. 6 shows the evaluation results for the performance of the streaming ASR module. As mentioned above, sp-KD represents similarity-preserving knowledge distillation, i.e., the method of this embodiment. MSE represents the mean square error-based knowledge distillation method. For both the non-streaming ASR module and the streaming ASR module, the method of this embodiment shows a significant improvement in WER compared to the comparative method.

[0098] According to the embodiment described above, the processing unit 120 of the speech recognition device 100 acquires frames of an audio stream and generates a first feature h from the frames using the streaming encoder ENC1. The processing unit 120 generates a character y from the first feature h using the streaming decoder DEC1. The processing unit 120 generates a first feature sequence H by combining the first features h of each of a series of multiple frames acquired until the utterance is segmented. st The processing unit 120 generates the first feature sequence H using the non-streaming encoder ENC2. stFrom the second feature column H te The processing unit 120 generates the second feature sequence H using the plurality of non-streaming decoders DEC2. te Then, the processing unit 120 generates a character string Y from the first feature string H st and the second feature sequence H te In other words, the processing unit 120 performs knowledge distillation between the streaming encoder ENC1 and the non-streaming encoder ENC2 based on the knowledge distillation loss L dist Each encoder is trained to minimize . As a result, the streaming ASR module and the non-streaming ASR module connected in series can be optimized, further improving the accuracy of speech recognition.

[0099] The above-described embodiment can be expressed as follows. a storage medium for storing computer-readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: Get the frames per unit time of the audio stream, generating first features from the frames using a streaming encoder; generating a first character from the first features using a streaming decoder; generating a second sequence of features from a first sequence of features that combines the first features of each of the plurality of frames using a non-streaming encoder; generating a second character string from the second feature sequence using a plurality of non-streaming decoders; performing knowledge distillation between the streaming encoder and the non-streaming encoder based on the first feature sequence and the second feature sequence; Voice recognition device.

[0100] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0101] 10...microphone, 100...speech recognition device, 120...processing unit, 121...acquisition unit, 122...streaming feature generation unit, 123...streaming character generation unit, 124...non-streaming feature generation unit, 125...non-streaming character generation unit, 126...learning unit, 130...storage unit, ENC1...streaming encoder, DEC1...streaming decoder, ENC2...non-streaming encoder, DEC2...non-streaming decoder, DEC2-1...RNN-T decoder, DEC2-2...attention decoder, DEC2-3...CTC decoder, DEC2-4...MLM decoder

Claims

1. an acquisition unit that acquires frames per unit time of an audio stream; a streaming feature generator for generating first features from the frames using a streaming encoder; a streaming character generator for generating a first character from the first features using a streaming decoder; a non-streaming feature generator that generates a second feature sequence from a first feature sequence obtained by combining the first features of each of the plurality of frames using a non-streaming encoder; a streaming character generator that generates a second character string from the second feature sequence using a plurality of non-streaming decoders; a learning unit that performs knowledge distillation between the streaming encoder and the non-streaming encoder based on the first feature sequence and the second feature sequence; A speech recognition device comprising:

2. the learning unit performs the knowledge distillation between the streaming encoder and the non-streaming encoder such that the first feature sequence is similar to the second feature sequence. The speech recognition device according to claim 1 .

3. the learning unit further performs the knowledge distillation between the streaming decoder and the plurality of non-streaming decoders based on a likelihood of a first character string in which the first characters generated from each of the plurality of first features are arranged in time series, and a likelihood of the second character string generated from the second feature sequence.

3. The speech recognition device according to claim 1.

4. the streaming decoder includes at least a predictor for predicting the first character; the plurality of non-streaming decoders include at least an attention decoder, which is a decoder including an attention mechanism; the learning unit performs the knowledge distillation between the predictor and the attention decoder so that the likelihood of the first character string, which is a time series of the first characters predicted by the predictor, is similar to the likelihood of the second character string output by the attention decoder.

4. The speech recognition device according to claim 3.

5. obtaining frames per unit time of the audio stream; generating a first feature from the frame using a streaming encoder; generating a first character from the first features using a streaming decoder; generating a second sequence of features from a first sequence of features that combines the first features of each of the plurality of frames using a non-streaming encoder; generating a second character string from the second sequence of features using a plurality of non-streaming decoders; performing knowledge distillation between the streaming encoder and the non-streaming encoder based on the first feature sequence and the second feature sequence; A speech recognition method comprising:

6. A program to be executed by a computer, obtaining frames per unit time of the audio stream; generating a first feature from the frame using a streaming encoder; generating a first character from the first features using a streaming decoder; generating a second sequence of features from a first sequence of features that combines the first features of each of the plurality of frames using a non-streaming encoder; generating a second character string from the second sequence of features using a plurality of non-streaming decoders; performing knowledge distillation between the streaming encoder and the non-streaming encoder based on the first feature sequence and the second feature sequence; Programs including.

Images