Voice recognition device, voice recognition method, and voice recognition program

Abstract

This speech recognizer (1b) comprises a label estimation unit (103), a trigger-firing-type label estimation unit (302), and an RNN-T trigger estimation unit (301). The label estimation unit (103) predicts a symbol sequence of speech data on the basis of an intermediate acoustic feature quantity sequence and an intermediate symbol feature quantity sequence of the speech data by using a model learned by RNN-T. The trigger-firing-type label estimation unit (302) predicts the next symbol of the speech data using an attention mechanism on the basis of the intermediate acoustic feature value sequence of the speech data. The RNN-T trigger estimation unit (301) calculates the timing at which the probability of occurrence of a symbol other than blank in the speech data is maximized on the basis of the symbol sequence of the speech data predicted by the label estimation unit (103). The RNN-T trigger estimation unit (301) then outputs the calculated timing as a trigger for operating the trigger-firing-type label estimation unit (302).

Description

[Technical field]

[0001] The present invention relates to a voice recognition device, a voice recognition method, and a voice recognition program. [Background technology]

[0002] Conventionally, there is an end-to-end speech recognition system that outputs any character sequence (e.g., phonemes, characters, subwords, words, etc.) directly from acoustic features. As a training method for this end-to-end speech recognition system, there is a training method using a Recurrent Neural Network Transducer (RNN-T) (see Non-Patent Document 1). This end-to-end speech recognition system trained with RNN-T can operate frame-by-frame, and therefore can perform streaming operation.

[0003] Another end-to-end speech recognition system uses an attention-based encoder-decoder (see Non-Patent Document 2). This technology can perform speech recognition with higher accuracy than the end-to-end speech recognition system trained using the RNN-T.

[0004] However, when performing speech recognition processing in this technology, the entire series of intermediate outputs is used, which makes streaming operation difficult.

[0005] To address this issue, there is a technology that performs pseudo-streaming operation of an Attention-based Encoder-Decoder (see Non-Patent Document 3). According to this technology, the intermediate output of the Encoder is passed through an output layer trained with a loss function called Connectionist Temporal Classification (CTC, see Non-Patent Document 4), and a frame-by-frame output is obtained. This output is similar to the output by the RNN-T described above, and the parts that do not output characters have a high probability of being blank, and the probability of being blank decreases at the moment when the corresponding phoneme, character, subword, word sequence, etc. are output.

[0006] In the above technology, by utilizing the characteristics of CTC, when the blank probability falls below a certain threshold, the decoder operates using the intermediate output of the encoder up to that point in time. This allows the attention-based encoder-decoder to operate in a pseudo frame-by-frame manner, enabling streaming operation. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Alex Graves, “Sequence Transduction with Recurrent Neural Networks,” in Proc. of ICML, 2012. [Non-Patent Document 2] J. Chorowski et.al., “Attention-Based Models for Speech Recognition,” in Advances in NIPS, 2015, pp. 577-585. [Non-Patent Document 3] N. Moritz et.al., “TRIGGERED ATTENTION FOR END-TO-END SPEECH RECOGNITION,” in Proc. of ICASSP, 2019, pp. 5666-5670. [Non-Patent Document 4] A. Graves, et.al., “Connectionist Temporal Classification: Labeling Unsegmented Sequence Data with Recurrent Neural Networks,” in Proc. of ICML, 2006, pp. 369-376. Summary of the Invention [Problem to be solved by the invention]

[0008] Among the above technologies, an end-to-end speech recognition system trained with RNN-T is capable of streaming operation, but has a problem of lower speech recognition accuracy compared to a technology using an Attention-based Encoder-Decoder. Also, a technology using an Attention-based Encoder-Decoder has a problem of high recognition accuracy but difficulty in streaming operation. Furthermore, a technology using CTC to perform pseudo-streaming operation of an Attention-based Encoder-Decoder has a problem of the timing of decoder operation depending on the performance of the CTC.

[0009] Therefore, an object of the present invention is to solve the above-mentioned problems, to make the operation timing of the decoder accurate when performing streaming operation of an end-to-end speech recognition system, and to improve the speech recognition accuracy. [Means for solving the problem]

[0010] In order to solve the above-mentioned problems, the present invention is characterized by comprising: a first decoder that predicts a symbol sequence of a speech signal based on an intermediate acoustic feature sequence and an intermediate symbol feature sequence of the speech signal to be recognized using a model trained by an RNN-T (Recurrent Neural Network Transducer); a second decoder that predicts a next symbol of the speech signal using an attention mechanism based on the intermediate acoustic feature sequence of the speech signal; and a trigger output unit that calculates a timing at which the probability that a symbol other than blank will occur in the speech signal is maximized based on the symbol sequence of the speech signal predicted by the first decoder, and outputs the calculated timing as a trigger for operating the second decoder. Effect of the Invention

[0011] According to the present invention, it is possible to make the operation timing of the decoder accurate when performing streaming operation of an end-to-end speech recognition system, and to improve the speech recognition accuracy. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a voice recognition device that is the basic technology of the voice recognition device of this embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of a voice recognition device that is the basic technology of the voice recognition device of this embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of a CTC path. [Figure 4] FIG. 4 is a diagram showing an example of the configuration of a voice recognition device according to the present embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a maximum likelihood path in the RNN-T. [Figure 6] FIG. 6 is a diagram showing the maximum likelihood path shown in FIG. 5 with the blank paths moving laterally removed. [Figure 7] FIG. 7 is a diagram showing the point corresponding to the maximum value among the points corresponding to each symbol shown in FIG. [Figure 8] FIG. 8 is a diagram showing an example of a range of frames used by the trigger-and-fire label estimation unit shown in FIG. 4 to calculate the attention α. [Figure 9] FIG. 9 is a diagram showing an example of a range of frames used by the trigger-and-fire label estimation unit shown in FIG. 4 to calculate the attention α. [Figure 10] FIG. 10 is a flowchart showing an example of a processing procedure of the voice recognition device of this embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of the configuration of a computer that executes a voice recognition program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, a mode (embodiment) for carrying out the present invention will be described with reference to the drawings. First, the basic technology of the voice recognition device of this embodiment will be described. The first basic technology is a voice recognition device 1 that performs voice recognition processing of voice data by RNN-T. The second basic technology is a voice recognition device 1a that uses CTC to perform pseudo streaming operation of an Attention-based Encoder-Decoder. Note that both the voice recognition devices 1 and 1a are voice recognition devices that perform end-to-end voice recognition.

[0014] [Voice recognition device 1] The speech recognition device 1 will be described with reference to Fig. 1. When an acoustic feature sequence and a symbol sequence of speech data to be recognized are input, the speech recognition device 1 outputs an estimate of the label of the symbol sequence of the speech data (output probability distribution of the label).

[0015] The speech recognition device 1 includes a first conversion unit 101, a second conversion unit 102, a label estimation unit 103, and a learning unit 105. The learning unit 105 includes an RNN-T loss calculation unit 104.

[0016] [First conversion unit 101] Input: Acoustic feature sequence X Output: Intermediate acoustic feature sequence H Processing: The first conversion unit 101 is an encoder that converts an input acoustic feature X into an intermediate acoustic feature sequence H by using a multi-stage neural network.

[0017] [Second conversion unit 102] Input: Symbol sequence c (length U) Output: Intermediate character feature sequence C (length U) Processing: The second conversion unit 102 is an encoder that converts the input symbol sequence c into corresponding continuous value features. For example, the second conversion unit 102 converts the input symbol sequence c into a one-hot vector, and then converts it into an intermediate character feature sequence C using a multi-stage neural network.

[0018] [Label Estimation Unit 103] Input: Intermediate acoustic feature sequence H, intermediate character feature sequence C (length U) Output: Output probability distribution Y Processing: The label estimation unit 103 calculates and outputs an output probability distribution Y of the labels of symbols of the speech data from the intermediate acoustic feature sequence H and the intermediate character feature sequence C described above, using a neural network.

[0019] For example, the label estimation unit 103 estimates the output probability y t,u Calculate.

[0020] y t,u =Softmax(W3(tanh(W1h t +W2c u +b))…Formula (1)

[0021] In addition, if the dimensions of t and u above are different, the number of elements in the neural network is also included in addition to t and u, making the number three-dimensional.

[0022] Specifically, when the label estimation unit 103 performs addition based on the above equation (1), W1H copies and expands the same value in the U dimension, and similarly W2C copies and expands the same value in the T dimension to adjust the dimensions, and then adds two three-dimensional tensors together, so that the output is also a three-dimensional tensor.

[0023] In general, when learning RNN-T, the model is trained by RNN-T loss on the assumption that it will be a three-dimensional tensor. When estimating labels by the label estimation unit 103, the output is a two-dimensional matrix because there is no expansion operation.

[0024] [RNN-T loss calculation unit 104] Input: Output probability distribution Y (3-dimensional tensor), correct symbol sequence c (length U) Output: Loss L RNN-T Processing: As shown in FIG. 1, the RNN-T loss calculation unit 104 calculates the loss L based on the output probability distribution Y output by the label estimation unit 103 and the correct symbol sequence c. RNN-T Calculate.

[0025] For example, the RNN-T loss calculation unit 104 calculates a path of optimal transition probability on the U×T plane in a tensor with vertical axis U (symbol sequence length), horizontal axis T (input sequence length), and depth K (number of classes: number of symbol entries) based on a forward-backward algorithm. Then, the RNN-T loss calculation unit 104 calculates the loss L using the path of optimal probability transition obtained by the calculation. RNN-T The detailed process of the above calculation is described in "2. Recurrent Neural Network Transducer" in Non-Patent Document 1.

[0026] [Learning Section 105] The learning unit 105 calculates the loss L RNN-T The parameters of the first conversion unit 101, the second conversion unit and the label estimation unit 103 are updated using

[0027] [Speech recognition device 1a] Next, a speech recognition device 1a will be described with reference to Fig. 2. The speech recognition device 1a also receives an acoustic feature sequence and a symbol sequence of speech data to be recognized, and outputs an estimate of the label of the symbol sequence of the speech data (output probability distribution of the label). This speech recognition device 1a uses CTC to perform a pseudo streaming operation of an Attention-based Encoder-Decoder. The same components as those in the speech recognition device 1 described above are denoted by the same reference numerals, and description thereof will be omitted.

[0028] The speech recognition device 1a includes a first conversion unit 101, a label estimation unit 103, a CTC loss calculation unit 202, a CTC trigger estimation unit 203, a trigger-fire type label estimation unit 204, a CE loss calculation unit 205, and a learning unit 207. The learning unit 207 includes a loss integration unit 206.

[0029] [Label Estimation Unit 201] Input: Intermediate acoustic feature sequence H Output: Output probability distribution Y' Processing: The label estimation unit 201 uses the intermediate acoustic feature sequence H from time 1 to T to obtain an output probability distribution Y′ of the symbol label based on the following equation (2).

[0030] y t =Softmax(Wh t +b)…Formula (2)

[0031] As shown in the above formula, unlike RNN-T, the output of CTC is a two-dimensional matrix both when learning the model parameters and when estimating using the model. The parameters to be learned are W and b.

[0032] [CTC loss calculation section 202] Input: Output probability distribution Y', correct symbol sequence c (length U) Output: Loss L CTC Processing: The CTC loss calculation unit 202 calculates the loss L using the output probability distribution Y′ output from the label estimation unit 201 and the correct symbol sequence c. CTCFor example, the CTC loss calculation unit 202 calculates the maximum likelihood path from the output matrix, which is the output probability sequence obtained by the label estimation unit 201, using a forward-backward algorithm. Then, the CTC loss calculation unit 202 calculates the loss L CTC For example, the CTC loss calculation unit 202 calculates the loss L CTC Calculate.

[0033] [CTC trigger estimation unit 203] Input: Output probability distribution Y', correct symbol sequence c (length U) Output: Trigger Z Processing: The CTC trigger estimation unit 203 is similar to the CTC loss calculation unit 202, and calculates the maximum likelihood path from the output matrix, which is the output probability sequence output from the label estimation unit 201, using a forward-backward algorithm.

[0034] Fig. 3 is an image diagram of a CTC path. The CTC trigger estimation unit 203 extracts the position of the smallest index (the symbol arranged on the vertical axis in Fig. 3) among the positions in the time direction (corresponding to the horizontal axis in Fig. 3) where each correct symbol occurs in the maximum likelihood path calculated using the forward-backward algorithm. Then, the CTC trigger estimation unit 203 outputs the extracted position as the position where the symbol occurs (=trigger Z).

[0035] [Trigger-fire type label estimation unit 204] Input: Intermediate acoustic feature sequence H, correct symbol sequence c (length U), trigger Z Output: Output probability distribution Y'' Processing: The trigger-firing label estimation unit 204 is a trigger-firing attention-equipped label estimation unit. Based on the trigger Z, the trigger-firing label estimation unit 204 calculates the output probability distribution Y'' of the label of the next symbol (e.g., "Hello.") using a symbol (e.g., "Hello") and an intermediate acoustic feature sequence H, which is a high-order acoustic feature.

[0036] The attention-based label estimator (Attention-based Encoder-Decoder) without using a trigger described in Non-Patent Document 2 operates, for example, based on formulas (1)-(9) in Non-Patent Document 2. This attention-based label estimator needs to use the entire intermediate acoustic feature sequence H from the beginning to the end of the audio (time index is written as L in Non-Patent Document 2) for attention calculation (formula (1) in Non-Patent Document 2). For this reason, it is difficult for the attention-based label estimator to perform streaming operation.

[0037] To address this problem, the trigger-firing label estimation unit 204 performs pseudo-streaming operation by using the framework of Non-Patent Document 3. For this reason, Equations (1) and (2) of Non-Patent Document 2 are defined as Equations (8) and (9) of Non-Patent Document 3.

[0038] Specifically, the trigger-firing label estimation unit 204 calculates the trigger Z as τ u (τ=z u +ε) to calculate Equation (8) and Equation (9) in Non-Patent Document 3 (Non-Patent Document 3 writes u as l). This is because the trigger-and-fire label estimation unit 204 uses the 1st to uth trigger points z u This means that the attention α is calculated using the intermediate acoustic feature sequence H up to . In this way, the trigger-firing label estimation unit 204 operates every time the trigger Z occurs, making it possible to perform pseudo-streaming operations.

[0039] [CE loss calculation section 205] Input: Output probability distribution Y'', correct symbol sequence c (length U) Output: Loss L CE Processing: The CE loss calculation unit 205 calculates the loss L using the prediction result of the next symbol (output probability distribution Y'') and the correct symbol sequence c. CE Calculate the loss L CE is computed by a simple cross entropy loss.

[0040] [Loss integration section 206] Input: Loss L CE , loss L CTC , hyperparameter ρ(0<ρ<1) Output: Loss L Processing: The loss integration unit 206 weights the losses calculated by each loss calculation unit (the CTC loss calculation unit 202 and the CE loss calculation unit 2205) by the hyper-parameter ρ, and calculates the integrated loss L (equation (3)).

[0041] L = (1-ρ)L CE +ρL CTC ...Formula (3)

[0042] [Learning Section 207] The learning unit 207 performs learning (parameter update) of the first conversion unit 101, the label estimation unit 201, and the trigger-firing label estimation unit 204 based on the loss L calculated by the loss integration unit 206.

[0043] [Speech recognition device 1b] Next, the voice recognition device 1b will be described with reference to Fig. 4. The same components as those in the voice recognition devices 1 and 1a are denoted by the same reference numerals and will not be described.

[0044] The difference between the speech recognition device 1 and the speech recognition device 1 is that the speech recognition device 1b uses a trigger-firing label estimation unit for predicting symbols and learning models. Also, the difference between the speech recognition device 1b and the speech recognition device 1a is that the speech recognition device 1b uses the output of an RNN-T for estimating triggers and operates the trigger-firing label estimation unit at high speed (details will be described later).

[0045] According to the above-described speech recognition device 1b, the operation timing of the decoder (trigger-firing label estimation unit) during streaming operation can be made more accurate and the speech recognition accuracy can be improved compared to when CTC is used as in the case of the speech recognition device 1a.

[0046] 4, the speech recognition device 1b includes a first conversion unit 101, a second conversion unit 102, a label estimation unit (first decoder) 103, an RNN-T loss calculation unit 104, a CE loss calculation unit 205, an RNN-T trigger estimation unit 301, a trigger-firing label estimation unit (second decoder) 302, and a learning unit 304. The learning unit 304 includes a loss integration unit 303.

[0047] [RNN-T trigger estimation unit 301] Input: Output probability distribution Y, Correct symbol sequence c (length U) Output: Trigger Z' Processing: The RNN-T trigger estimation unit 301 uses a forward-backward algorithm to calculate the maximum likelihood path from the three-dimensional output tensor, which is the output probability sequence obtained by the label estimation unit 103. For example, the RNN-T trigger estimation unit 301 calculates the maximum likelihood path shown in FIG. 5 from the output probability distribution Y. Note that the vertical axis in FIG. 5 is U and the horizontal axis is T. In this maximum likelihood path, moving in the horizontal axis direction indicates that a blank is output, and moving in the vertical axis direction indicates that the correct symbol is output.

[0048] FIG6 shows the maximum likelihood path shown in FIG5 with the blank paths moving in the horizontal direction removed. Each point (white portion) in FIG6 represents a predicted value of the timing at which the next symbol will occur. The RNN-T trigger estimation unit 301 outputs, as a trigger, the time index of the point at which the probability value of the occurrence of the symbol is maximum among the points corresponding to each symbol shown in FIG6. The RNN-T trigger estimation unit 301 sets the time index of the point at which the probability value is maximum among the points corresponding to the correct symbol sequence c during learning as the trigger Z' of each correct symbol.

[0049] [Trigger firing type label estimation unit 302] Input: Intermediate acoustic feature sequence H, correct symbol sequence c (length U), trigger Z' Output: Output probability distribution Y'' Processing: The trigger-firing label estimation unit 302 is similar to the trigger-firing label estimation unit 204 (see FIG. 2), but differs in that it calculates an output probability distribution Y″ of the label of the next symbol (e.g., “Hello.”) using the intermediate acoustic feature sequence H based on the trigger Z′ output by the RNN-T trigger estimation unit 301.

[0050] In addition, the trigger-and-fire label estimation unit 302 uses the previous trigger z u-1 (=t u-1 ) and subsequent frames (see Figure 8).

[0051] In addition, the trigger-and-fire label estimation unit 302 uses the trigger z u A certain number of frames before and after (Lookahead length=z u +ε, Lookback length=z u - A frame of ε, ε: hyperparameters that can be set arbitrarily may be used.

[0052] The trigger-firing label estimation unit 302 calculates the attention α using the above-mentioned calculation methods, thereby saving memory and enabling high-speed operation. As a result, the speech recognition device 1b is capable of high-speed streaming operation.

[0053] In addition, since the output probability distribution Y″ output by the trigger firing type label estimation unit 302 has a similar format to the output probability distribution Y″ output by the trigger firing type label estimation unit 204 (see FIG. 2), the loss can be calculated by the CE loss calculation unit 205.

[0054] [Loss integration section 303] Input: Loss L CE , loss L RNN-T , hyperparameter λ(0≦λ≦1) Output: Loss L Processing: The loss integrating unit 303 integrates the losses calculated by each loss calculation unit (the RNN-T loss calculation unit 104 and the CE loss calculation unit 205) using a weighted sum based on the hyper-parameter λ to calculate the loss L (equation (4)).

[0055] L = (1-λ)L CE +λL RNN-T ...Formula (4)

[0056] [Learning Section 304] The learning unit 304 performs learning (parameter update) of the first conversion unit 101, the second conversion unit 102, and the label estimation unit 103, and the trigger-firing label estimation unit 302, based on the loss L calculated by the loss integration unit 303.

[0057] Here, the learning unit 304 may, for example, first train each unit (the first conversion unit 101, the second conversion unit 102, and the label estimation unit 103) that uses RNN-T, fix the parameters of each unit, and then train the trigger firing label estimation unit 302. For example, the learning unit 304 may assign λ=1 to the above formula (4) to train the first conversion unit 101, the second conversion unit 102, and the label estimation unit 103, and then assign λ=0 to train the trigger firing label estimation unit 302.

[0058] By the learning unit 304 learning as described above, the RNN-T trigger estimation unit 301 can output an accurate trigger. This allows the learning unit 304 to learn the trigger-firing type label estimation unit 302 using the accurate trigger. As a result, the learning unit 304 can improve the estimation accuracy by the trigger-firing type label estimation unit 302. In other words, the speech recognition accuracy of the speech recognition device 1b can be improved.

[0059] [Example of processing procedure] Next, an example of a processing procedure of the speech recognition device 1b after learning will be described with reference to Fig. 10. When the speech recognition device 1b receives input of speech data to be recognized, it performs the following processing. First, the first conversion unit 101 converts the acoustic feature sequence of the input speech data into an intermediate acoustic feature sequence (S1). Also, the second conversion unit 102 converts the symbol feature sequence of the input speech data into an intermediate character feature sequence (S2).

[0060] Thereafter, the label estimation unit 103 calculates an output probability sequence of labels of symbols of the speech data from the intermediate acoustic feature sequence and the intermediate character feature sequence using the model trained by the RNN-T (S3).

[0061] Then, the RNN-T trigger estimation unit 301 calculates the timing at which the probability that a symbol other than blank will occur in the voice data is maximized from the output probability sequence of the label of the symbol of the voice data calculated in S3, and outputs the calculated timing as a trigger to operate the trigger-fire label estimation unit 302 (S4: calculate a trigger from the output probability sequence of the symbol and output the calculated trigger).

[0062] The trigger-firing label estimation unit 302 predicts symbols of the speech data using the intermediate acoustic feature sequence based on the trigger output from the RNN-T trigger estimation unit 301 (S5). After that, if the trigger-firing label estimation unit 302 receives the next trigger input from the RNN-T trigger estimation unit 301 (Yes in S6), it executes the process of S5. On the other hand, if the trigger-firing label estimation unit 302 does not receive the next trigger input from the RNN-T trigger estimation unit 301 (No in S6), it returns to S6 and waits for the next trigger input.

[0063] In this way, the voice recognition device 1b can perform voice recognition of voice data by streaming operation.

[0064] [System configuration, etc.] In addition, each component of each unit shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in part by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

[0065] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0066] [program] The above-mentioned voice recognition device 1b can be implemented by installing a program (voice recognition program) as package software or online software on a desired computer. For example, the above-mentioned program can be executed by an information processing device, so that the information processing device can function as the voice recognition device 1b. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), as well as terminals such as PDAs (Personal Digital Assistants).

[0067] 11 is a diagram showing an example of a computer that executes a voice recognition program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0068] The memory 1010 includes a read only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0069] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process executed by the above-mentioned voice recognition device 1b is implemented as a program module 1093 in which a computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing a process similar to the functional configuration of the voice recognition device 1b is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0070] Data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.

[0071] Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0072] 1,1a,1b Voice recognition device 101 First conversion unit 102 Second conversion section 103,201 Label Estimation Unit 104 RNN-T loss calculation section 202 CTC loss calculation section 203 CTC Trigger Estimation Unit 204,302 Trigger-fire label estimation unit 205 CE loss calculation section 206,303 Loss integration department 207,304 Learning Department 301 RNN-T Trigger Estimation Unit

Claims

1. a first decoder for predicting a symbol sequence of a speech signal based on an intermediate acoustic feature sequence and an intermediate symbol feature sequence of the speech signal to be recognized, using a model trained by a recurrent neural network transducer (RNN-T); a second decoder for predicting a next symbol of the speech signal based on a sequence of intermediate acoustic features of the speech signal using an attention mechanism; a trigger output unit that calculates a timing at which the probability that a symbol other than blank will occur in the voice signal is maximized based on the symbol sequence of the voice signal predicted by the first decoder, and outputs the calculated timing as a trigger for operating the second decoder; A speech recognition device comprising:

2. The second decoder comprises: The next symbol is estimated using an intermediate acoustic feature sequence of the speech signal from a point onward corresponding to the timing when the second decoder was last operated.

2. The speech recognition device according to claim 1.

3. The second decoder comprises: The next symbol is estimated using an intermediate acoustic feature sequence of a predetermined section before and after a point corresponding to the timing at which the second decoder operates, among the intermediate acoustic feature sequence of the speech signal.

2. The speech recognition device according to claim 1.

4. a learning unit that uses a correct symbol sequence for the speech signal as learning data and determines parameters of a model used by the first decoder and the second decoder; The speech recognition device according to claim 1, further comprising:

5. The learning unit is After determining the parameters of the model used by the first decoder, the parameters of the model used by the second decoder are determined.

5. The speech recognition device according to claim 4.

6. A speech recognition method executed by a speech recognition device, comprising: a first step of predicting a symbol sequence of a speech signal based on an intermediate acoustic feature sequence and an intermediate symbol feature sequence of the speech signal to be recognized, using a model trained by a recurrent neural network transducer (RNN-T); a second step of predicting a next symbol of the speech signal based on a sequence of intermediate acoustic features of the speech signal using an attention mechanism; a third step of calculating a timing at which the probability that a symbol other than blank will occur in the voice signal is maximized based on the symbol sequence of the voice signal predicted in the first step, and outputting the calculated timing as a trigger for executing the second step; 13. A speech recognition method comprising:

7. a first step of predicting a symbol sequence of a speech signal based on an intermediate acoustic feature sequence and an intermediate symbol feature sequence of the speech signal to be recognized, using a model trained by a recurrent neural network transducer (RNN-T); a second step of predicting a next symbol of the speech signal based on a sequence of intermediate acoustic features of the speech signal using an attention mechanism; a third step of calculating a timing at which the probability that a symbol other than blank will occur in the voice signal is maximized based on the symbol sequence of the voice signal predicted in the first step, and outputting the calculated timing as a trigger for executing the second step; A speech recognition program that allows a computer to execute the following:

Images