Learning devices, speech recognition devices, and their programs

Abstract

This invention provides a learning device that can apply an AED-type model to streaming speech recognition. [Solution] The learning device 1 comprises an acoustic feature extraction unit 10, a duplication unit 11 for duplicating acoustic feature vectors 100, a mask processing unit 12 for masking each acoustic feature vector 100 in a stepwise manner, a control signal insertion unit 13 for inserting streaming control signals, an encoder 14, a language feature extraction unit 15, a decoder 16, an α calculation unit 17 for calculating α, a Dirichlet distribution calculation unit 18 for calculating the Dirichlet distribution, a streaming loss calculation unit 19 for calculating the streaming loss, an output probability calculation unit 20, an offline loss calculation unit 21 for calculating the offline loss, and a learning unit 22.

Description

[Technical Field]

[0001] This invention relates to a learning device, a speech recognition device, and programs for the same. [Background technology]

[0002] Conventional speech recognition has predominantly relied on DNN-HMM models, which are composed of the HMM (Hidden Markov Model), a probability and statistical model, and the DNN (Deep Neural Network). In recent years, end-to-end models that can consistently process speech input to text output using neural networks have become the focus of research due to their high speech recognition accuracy. The AED (Attention-based Encoder-Decoder) model, a type of end-to-end model, has achieved high performance in offline speech recognition (Non-Patent Literature 1) and is superior in speech recognition accuracy compared to other end-to-end models (Non-Patent Literature 2).

[0003] In offline speech recognition, as shown in Figure 10(a), silent intervals are separated from the speech data 90 (step S90) to obtain the speech data 91 that says "I held a press conference at the Prime Minister's official residence." Then, in offline speech recognition, acoustic features are extracted from the speech data 91 (step S91), and speech recognition is performed using the extracted acoustic features (step S92). In this case, the recognition result 92 that says "I held a press conference at the Prime Minister's official residence" is obtained.

[0004] Conventionally, streaming speech recognition, in which fixed-length audio is input piece by piece, has also been used. As shown in Figure 10(b), in streaming speech recognition, fixed-length audio data 91 such as "Kantei de..." or "Kantei de kishaka..." is input piece by piece. Then, in streaming speech recognition, acoustic features are extracted from the audio data 91 (step S91), and speech recognition is performed using the extracted acoustic features (step S92). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] A. Radford, JW Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision,” in arXiv preprint arXiv:2212.04356, 2022. [Non-Patent Document 2] TN Sainath, R. Pang, D. Rybach, Y. He, R. Prabhavalkar, W. Li, M. Visontai, Q. Liang et al., “Two-pass end-to-end speech recognition,” in Proc. Interspeech, 2019. [Overview of the project] [Problems that the invention aims to solve]

[0006] However, when AED-type models are applied to streaming speech recognition, they have the following challenges. Typically, offline speech recognition models such as AED-type models use pairs of audio and transcripts, each consisting of a single utterance separated by silence, as training data. In contrast, the input data for streaming speech recognition is mechanically fixed-length audio (e.g., in units of one second), so at the end of the audio, words may be left unsaid.

[0007] This incomplete speech is not included in the training data for offline speech recognition, leading to speech recognition errors. In the example in Figure 10(b), the word "kishakaiken" is incomplete, so instead of recognizing it as "a press conference at the Prime Minister's official residence," it is misrecognized as "a reporter at the Prime Minister's official residence." Note that in Figure 10(b), the misrecognized "ka" is underlined. Thus, there is a challenge in applying AED-type models to streaming speech recognition.

[0008] Therefore, the object of the present invention is to provide a learning device, a speech recognition device, and a program thereof that can apply an AED-type model to streaming speech recognition. [Means for solving the problem]

[0009] To solve the aforementioned problems, the learning device according to the present invention is a learning device for learning an AED-type model capable of streaming speech recognition, and comprises an acoustic feature extraction unit, a replication unit, a mask processing unit, a control signal insertion unit, an encoder, a language feature extraction unit, a decoder, an input value calculation unit, a Dirichlet distribution calculation unit, a streaming loss calculation unit, an output probability calculation unit, an offline loss calculation unit, and a learning unit.

[0010] The acoustic feature extraction unit extracts acoustic features from training audio data and generates an acoustic feature vector in which multiple elements are consecutive in the time direction. The replication unit replicates the acoustic feature vector a predetermined number of times. The masking unit sets the number of elements to be masked, which are a predetermined number of consecutive elements from the end of an acoustic feature vector, to decrease the number of elements to be masked included in each acoustic feature vector, and then masks the elements to be masked. The control signal insertion unit inserts a streaming control signal into each masked acoustic feature vector, indicating that streaming speech recognition should be performed. The encoder outputs a first vector matrix by encoding each acoustic feature vector into which a streaming control signal is inserted.

[0011] The language feature extraction unit extracts language features from character / word sequences corresponding to speech data and generates a language feature vector in which multiple elements are consecutive in the time direction. The decoder outputs a second vector matrix by decoding the language feature vector and the first vector matrix. The input value calculation unit calculates input values ​​representing evidence in EDL (Evidential Deep Learning) from the second vector matrix. The Dirichlet distribution calculation unit calculates the Dirichlet distribution from the input values. The streaming loss calculation unit calculates the streaming loss, which is the loss function during streaming speech recognition, from the Dirichlet distribution. The output probability calculation unit calculates the output probability of a word from the input value using the softmax function. The offline loss calculation unit calculates the offline loss, which is the loss function for offline speech recognition, from the word output probability. The learning unit trains an AED-type model based on the sum of streaming loss and offline loss.

[0012] With this configuration, the learning device learns the speech in progress necessary for streaming speech recognition by duplicating and masking acoustic feature vectors. Since the learning device uses EDL, it is possible to calculate the confidence level, i.e., uncertainty, of the output words of the AED-type model. Then, during streaming speech recognition, the reliability of the speech recognition result can be determined by calculating the uncertainty of the output words of the AED-type model.

[0013] Furthermore, in order to solve the above-mentioned problems, the speech recognition device according to the present invention is a speech recognition device that performs streaming speech recognition using an AED-type model learned by the learning device described above, and comprises an acoustic feature extraction unit, a control signal insertion unit, an encoder, a language feature extraction unit, a decoder, an input value calculation unit, an output probability calculation unit, an uncertainty calculation unit, and a determination unit.

[0014] The acoustic feature extraction unit extracts acoustic features from the speech data for recognition and generates an acoustic feature vector. The control signal insertion unit inserts a streaming control signal into the acoustic feature vector, indicating that streaming speech recognition is to be performed. The encoder encodes the acoustic feature vector into which the streaming control signal has been inserted.

[0015] The language feature extraction unit extracts language features from the character / word sequences resulting from speech recognition by the AED-type model and generates language feature vectors. The decoder decodes the language feature vector and the encoder output. The input value calculation unit calculates an input value representing the evidence in EDL from the decoder's output. The output probability calculation unit calculates the output probability of a word from the input value using the softmax function. The uncertainty calculation unit calculates the uncertainty of the AED-type model using the output probability of the words.

[0016] The determination unit determines whether the uncertainty is below a preset threshold. If the uncertainty is below the threshold, it outputs the result of streaming speech recognition using the AED-type model. If the uncertainty is not below the threshold, it causes the acoustic feature extraction unit to process the next audio data.

[0017] With this configuration, the speech recognition device uses an AED-type model that has learned the necessary speech in progress, enabling streaming speech recognition. At this time, the speech recognition device can calculate the uncertainty of the output words of the AED-type model and determine the reliability of the speech recognition result according to that uncertainty. If the speech recognition result is unreliable, the speech recognition device reads the next audio data to increase the amount of information and performs speech recognition again.

[0018] Furthermore, the present invention can also be implemented by a program that causes a computer to function as the aforementioned learning device or speech recognition device. [Effects of the Invention]

[0019] According to the present invention, an AED-type model can be applied to streaming speech recognition. [Brief explanation of the drawing]

[0020] [Figure 1] This is an explanatory diagram illustrating the overview of the learning device in the first embodiment. [Figure 2] This is a block diagram showing the configuration of the learning device according to the first embodiment. [Figure 3] (a) to (d) are explanatory diagrams illustrating a stepped mask in the first embodiment. [Figure 4] This is a flowchart showing the operation of the learning device according to the first embodiment. [Figure 5] This is a block diagram showing the configuration of the speech recognition device according to the second embodiment. [Figure 6] This is a block diagram showing the configuration of a streaming speech recognition device according to the second embodiment. [Figure 7] This is a block diagram showing the configuration of an offline speech recognition device according to the second embodiment. [Figure 8] This is a flowchart showing the operation of the streaming speech recognition device according to the second embodiment. [Figure 9] This is a flowchart showing the operation of the offline speech recognition device according to the second embodiment. [Figure 10] (a) is a diagram illustrating conventional offline speech recognition, and (b) is a diagram illustrating conventional streaming speech recognition. [Modes for carrying out the invention]

[0021] Embodiments of the present invention will be described below with reference to the drawings. However, the embodiments described below are intended to embody the technical concept of the present invention, and unless otherwise specified, the present invention is not limited to these embodiments. In addition, the same reference numerals are used for the same means, and their descriptions may be omitted.

[0022] (First Embodiment) In streaming speech recognition, the end of the input audio may be interrupted. Since these interrupted speech patterns are not included in the training data for streaming speech recognition, they can lead to errors. Therefore, it is considered more accurate to hold off on processing until a definitive speech recognition result can be determined, and then proceed to the next audio if a result cannot be determined.

[0023] This invention consists of two stages: learning and recognition. During learning, the system learns the speech in progress necessary for streaming speech recognition by duplicating and masking acoustic feature vectors. During speech recognition, uncertainty is calculated for the output of the AED-type model. If the uncertainty is below a threshold, the speech recognition result is considered reliable and a word is output. On the other hand, if the uncertainty is above the threshold, the speech recognition result is considered unreliable, and the system reads the next audio data to increase the amount of information.

[0024] In conventional neural networks, the concept of uncertainty does not exist, so for example, a biased output probability distribution may be output even when there is insufficient evidence to support the classification. In this invention, uncertainty is calculated using a learning method called EDL (Reference 1). That is, EDL calculates uncertainty as the degree of confidence with which the neural network model outputted the recognition result. This makes it possible to make judgments about the recognition results of the speech recognition model while taking this uncertainty into account.

[0025] Reference 1: Gao, M. Chen, L. Xiang, and C. Xu, “A Comprehensive Survey on Evidential Deep Learning and Its Applications,” in arXiv preprint arXiv:2409.04720, 2024.

[0026] Two types of uncertainty are defined: data-related uncertainty, such as data ambiguity or low discriminability, like data contaminated with noise, and uncertainty related to the output of the neural network model, such as insufficient training of the neural network model. Streaming speech recognition needs to be effective not only for clean speech but also for speech containing noise. Therefore, this invention does not utilize the former type of data-related uncertainty, but rather utilizes uncertainty related to the output of the neural network model.

[0027] Uncertainty U arising from the output of the neural network model dis It is defined by the following equation (1) (Reference 2). Reference 2: M. Xie, S. Li, R. Zhang, and CH Liu, “Dirichlet-based uncertainty calibration for active domain adaptation,” in Proc. ICLR, 2023.

[0028]

number

[0029] Here, C represents the number of words in the neural network model, and p c The output probability of word c is (Σ C c=1 p c =1) represents the digamma function.

[0030] Below, α in equation (1) c This will be explained. α is used to calculate the softmax function, loss function, and uncertainty for calculating the probability of word output. α is a concept called "evidence" in EDL, and its value increases as the amount of information in the audio data increases. In equation (1), α is a C-dimensional vector. As the amount of information in the audio data increases, the values ​​of the C elements contained in α increase, and the uncertainty value decreases. In particular, if word c is correctly classified, the value of the c-th element contained in α increases.

[0031] The output probability p of a neural network model in class classification c is represented by the following formula (2) using the softmax function.

[0032] [Number]

[0033] Here, o c represents the value of the c-th dimension in the output vector of the final layer of the neural network model. α c is e in formula (2) Oc That is, α c = e Oc is represented by.

[0034] The Dirichlet distribution is often used for uncertainty learning. The Dirichlet distribution Dir is represented by the following formula (3).

[0035] [Number]

[0036] Here, p = (p1,..., p c ,..., p C ) and α = (α1,..., α c ,..., α C ). Also, B(α) represents the normalization term.

[0037] The Dirichlet distribution is a probability distribution in which a certain set of probabilities (p1,..., p c ,..., p C ) is generated. For example, in a normal neural network for class classification, probabilities for each class are output. In contrast, in the Dirichlet distribution, this set of output probabilities itself is generated with a probability on the Dirichlet distribution. For example, in 3-class classification, possible sets of output probabilities are (p1, p2, p3) = (0.5, 0.3, 0.2) and (p1, p2, p3) = (0.2, 0.1, 0.7).

[0038] The Dirichlet distribution defines the probability of generating the set of probabilities themselves. A uniform Dirichlet distribution means that the model can generate an infinite number of equally generated sets of probabilities, i.e., a state of high uncertainty. On the other hand, a steep Dirichlet distribution means that the set of probabilities output by the model is somewhat limited, i.e., a state of low uncertainty.

[0039] As shown in Figure 1, during training, when inputting training audio data 90, the following states are reproduced: (1) silence, (2) streaming, and (3) offline. (1) silence is a state where no audio before speaking is input, (2) streaming is a state where audio in the middle of speaking is input, similar to streaming speech recognition, and (3) offline is a state where all spoken audio is input, similar to offline speech recognition.

[0040] The Dirichlet distribution 94 for the silent state is trained to be uniform as an uncertain state. The Dirichlet distribution for the offline state is trained to have low uncertainty. At the same time, the Kullback-Leibler divergence 93 between the Dirichlet distribution 94 for the silent state and the Dirichlet distribution 94 for the streaming state is calculated for each word. In addition, the Kullback-Leibler divergence 93 between the Dirichlet distribution 94 for the streaming state and the Dirichlet distribution 94 for the offline state is calculated for each word. Then, the sum of the two Kullback-Leibler divergences 93 calculated for each word is minimized, and each Dirichlet distribution 94 is trained to be as close as possible.

[0041] Thus, for each word in the streaming state, if the amount of information in the audio data 90 is sufficient, the AED-type model 95 is trained to approach the Dirichlet distribution 94 in the offline state. On the other hand, if the amount of information in the audio data 90 is insufficient, the AED-type model 95 is trained to approach the Dirichlet distribution 94 in the silent state.

[0042] [Configuration of the learning device] Referring to Figure 2, the configuration of the learning device 1 according to the first embodiment will be described. The learning device 1 learns an AED-type model capable of streaming speech recognition. As shown in Figure 2, the learning device 1 comprises an acoustic feature extraction unit 10, a replication unit 11, a mask processing unit 12, a control signal insertion unit 13, an encoder 14, a language feature extraction unit 15, a decoder 16, an α calculation unit (input value calculation unit) 17, a Dirichlet distribution calculation unit 18, a streaming loss calculation unit 19, an output probability calculation unit 20, an offline loss calculation unit 21, and a learning unit 22.

[0043] The user inputs learning audio data and corresponding character / word sequences into the learning device 1. The learning audio data is audio data of an arbitrary sentence being read aloud. The character / word sequences are data that is a text version of the learning audio data.

[0044] The acoustic feature extraction unit 10 extracts acoustic features from training audio data and generates an acoustic feature vector 100 in which multiple elements are consecutive in the time direction. Examples of acoustic features include Mel spectra and Mel-frequency cepstrum coefficients. As shown in Figure 3(a), the acoustic feature vector 100 is a vector represented by T elements 101 (for example, T=3). T represents the maximum number of elements 101 included in the acoustic feature vector 100 in a given time and can be manually set by the user. Also, t represents the index of the element 101 (t=1,2,...,T). The acoustic feature extraction unit 10 outputs the generated acoustic feature vector 100 to the replication unit 11.

[0045] The replication unit 11 replicates the acoustic feature vector a predetermined number of times. As shown in Figure 3(b), the replication unit 11 replicates the acoustic feature vector 100 generated by the acoustic feature extraction unit 10 M times (for example, M=4). In other words, there are a total of M+1 acoustic feature vectors 100, including the original. The number M of acoustic feature vectors 100 to be replicated can be manually set by the user, for example, depending on the amount of GPU memory. Also, m represents which number acoustic feature vector 100 it is (m=1,2,...,M). Note that in Figure 3(b), only some elements 101 have their symbols shown to make the diagram easier to read (the same applies to Figures 3(c) and 3(d)). The replication unit 11 outputs each acoustic feature vector 100 to the mask processing unit 12.

[0046] The masking unit 12 sets the number of elements 101 that are consecutive from the end of the acoustic feature vector 100 to be masked, so that the number of masked elements in each acoustic feature vector 100 decreases, and then masks the masked elements. In other words, the masking unit 12 performs the masking in a stepwise manner in order to reproduce the streaming state in which sound is input sequentially. Note that masking means setting the value of element 101 to 0.

[0047] As shown in Figure 3(c), for the first acoustic feature vector 100 (m=1), the masking unit 12 sets the three consecutive elements 102 from the end as the mask target and masks them. In Figure 3(c), the element 101 that is not the mask target is shown in white, and the element 102 that is the mask target is hatched. For the second acoustic feature vector 100 (m=2), the masking unit 12 sets the two consecutive elements 102 from the end as the mask target and masks them. For the third acoustic feature vector 100 (m=3), the masking unit 12 sets the single element 102 located at the end as the mask target and masks it. For the fourth acoustic feature vector 100 (M=m=4), the masking unit 12 does not set a mask target and does not perform masking in order to reproduce the offline state in which all sound is input. In this way, the masking unit 12 reproduces the streaming state by gradually reducing the number of elements 102 to be masked and masking them.

[0048] In the example in Figure 3(c), since M=T+1, the number of elements 102 to be masked decreases by one each time. In reality, M may be significantly less than T, which depends on the time length of the audio data. In this case, all elements 101 included in the acoustic feature vector 100 should be masked so that they are equally divided by M. If elements 101 remain because they cannot be equally divided by M, the remaining elements 101 should be added to the last group of elements. For example, if T=100 and M=3, the acoustic feature vector 100 can be divided into three groups of elements: elements 101 from t=1 to 33, elements 101 from t=1 to 66, and elements 101 from t=1 to 100, and each group should be masked.

[0049] The mask processing unit 12 does not set a mask target for one of the acoustic feature vectors 100 duplicated by the duplication unit 11. In the example in Figure 3(c), the mask processing unit 12 does not set a mask target for the 5th (M+1=m=5) acoustic feature vector 100 in order to insert the offline control signal 104 described later. Subsequently, the mask processing unit 12 outputs each acoustic feature vector 100 to the control signal insertion unit 13.

[0050] The control signal insertion unit 13 inserts a streaming control signal 103 for each masked acoustic feature vector 100, indicating that streaming speech recognition should be performed. As shown in Figure 3(d), the mask processing unit 12 inserts the streaming control signal 103 at the beginning of the mask target, which consists of three elements 102, for the first acoustic feature vector 100. In this case, the streaming control signal 103 is inserted at the beginning of the first acoustic feature vector 100. The mask processing unit 12 inserts the streaming control signal 103 at the end of the unmasked elements 101 for the second and third acoustic feature vectors 100. In this case, the streaming control signal 103 is inserted between the unmasked elements 101 and the elements 102 that are to be masked. The mask processing unit 12 inserts the streaming control signal 103 at the end of the unmasked elements 101 for the fourth acoustic feature vector 100. In this case, the streaming control signal 103 is inserted at the end of the fourth acoustic feature vector 100.

[0051] The control signal insertion unit 13 inserts an offline control signal 104 to acoustic feature vectors 100 for which no mask target has been set, indicating that offline speech recognition should be performed. In the example shown in Figure 3(d), the control signal insertion unit 13 inserts the offline control signal 104 at the end of the fifth acoustic feature vector 100.

[0052] In this way, the control signal insertion unit 13 inserts the streaming control signal 103 or the offline control signal 104 into the acoustic feature vector 100 as a learnable vector so that it can switch between streaming speech recognition and offline speech recognition. The control signal insertion unit 13 outputs each acoustic feature vector 100 to the encoder 14.

[0053] The encoder 14 outputs a first vector matrix 110 by encoding each acoustic feature vector 100 into which a streaming control signal is inserted. As shown in Figure 2, the encoder 14 outputs from the first layer encoder 141 to N e Layer (for example, N e Encoder 14 (=12) Ne It includes up to the number of layers N of encoder 14. e Increasing the number of layers improves the accuracy of the judgment, but it consumes more computer resources and slows down the processing speed of learning and inference. Number of layers N for encoder 14 e This can be arbitrarily set, taking into account both the required judgment accuracy and the computer resources (the number of layers N of the decoder 16 described later). d (The same applies to the other.) The encoder 14 encodes the acoustic feature vector 100 and outputs a first vector matrix 110 of size (M+1)×(T+1) to the decoder 16.

[0054] The language feature extraction unit 15 extracts language features from character / word sequences corresponding to speech data and generates a language feature vector 120 in which multiple elements 121 are consecutive in the time direction. For example, a linear transformation layer with learnable parameters can be used as a language feature. L represents the maximum number of elements 121 included in the language feature vector 120 in the time direction (l=1,2,…,L), and can be manually set by the user of the learning device 1. The language feature extraction unit 15 outputs the generated language feature vector 120 to the decoder 16.

[0055] Decoder 16 outputs a second vector matrix by decoding the language feature vector 120 and the first vector matrix 110. As shown in Figure 2, decoder 16 outputs from the first layer decoder 161 to N d Layer (for example, N d Decoder 16 (=6) Nd Each layer decoder 16 includes a self-attention unit 16S that performs self-attention processing and a cross-attention unit 16C that performs cross-attention processing.

[0056] Decoder 16 is a second vector matrix, Decoder 16 Nd From vector o (l,m) ,o (l,offline) Vector o is obtained. (l,m) This is the decoder 16 of state m in the l-th word. Nd This is the output of . Also, vector o (l,m) It is defined as a C-dimensional vector, as shown in equation (4) below.

[0057]

number

[0058] Decoder 16 is the second vector matrix o (l,m) ,o (l,offline) The result is output to the α calculation unit 17.

[0059] The α calculation unit 17 calculates the second vector matrix o (l,m) ,o (l,offline) From this, the α (input value) representing the evidence in EDL is calculated. Specifically, the α calculation unit 17 calculates α as shown in the following equations (5) and (6). (l,m) ,α (l,offline) Calculate α. (l,m) ,α (l,offline) Both are C-dimensional.

[0060]

number

[0061] The α calculation unit 17 calculates the α (l,m) ,α (l,offline) The results are output to the Dirichlet distribution calculation unit 18 and the output probability calculation unit 20.

[0062] The Dirichlet distribution calculation unit 18 calculates the Dirichlet distribution from α. Specifically, the Dirichlet distribution calculation unit 18 uses equation (3) to calculate the Dirichlet distribution Dir(p|α (l,m) ),Dir(p|α (l,offline) Calculate ). The Dirichlet distribution calculation unit 18 calculates the Dirichlet distribution Dir(p|α (l,m) ),Dir(p|α (l,offline) The results are output to the streaming loss calculation unit 19 and the offline loss calculation unit 21.

[0063] The streaming loss calculation unit 19 calculates the streaming loss, which is the loss function during streaming speech recognition, from the Dirichlet distribution. Specifically, the streaming loss calculation unit 19 calculates the streaming loss L for training streaming speech recognition. hierarchical ,L skip Calculate the streaming loss L. hierarchical ,L skip The calculation will be described later.

[0064] The output probability calculation unit 20 is α (l,m) ,α (l,offline) The output probability of a word is calculated using the softmax function. Specifically, the output probability calculation unit 20 calculates the output probability of a word using equations (2), (5), and (6). The output probability calculation unit 20 outputs the calculated output probability of the word to the offline loss calculation unit 21.

[0065] The offline loss calculation unit 21 calculates the offline loss, which is the loss function for offline speech recognition, from the output probability of the words. Specifically, the offline loss calculation unit 21 calculates the offline loss L for learning offline speech recognition. offline Calculate.

[0066] <Calculation of Offline Loss and Streaming Loss> Below is the offline loss L offline and streaming loss L hierarchical ,L skip I will explain the calculation in detail.

[0067] <<Offline Loss L offline Calculation >> The offline loss calculation unit 21, similar to a typical AED model, uses the following equation (7) to calculate the offline loss Loffline Calculate.

[0068]

number

[0069] Here, KL represents the Kullback-Leibler divergence, and P (l,target) This represents the correct answer distribution for the l-th word. Also, P (l,offline) This represents the output probability distribution of the softmax function in an offline state.

[0070] Offline loss L offline By minimizing this, learning can be performed offline. Along with this, α (l,offline) You can learn that at the same time.

[0071] <<Streaming Loss L hierarchical Calculation >> Streaming Loss L hierarchical This is the loss for learning the process by which the Dirichlet distribution changes from an ambiguous state to a steep state. The ambiguous state of the Dirichlet distribution is when α=1. The steep state of the Dirichlet distribution is when α=α (l,offline) This is the state. The Dirichlet distribution for each word, which is the most ambiguous state, can be calculated by substituting 1 for α in equation (3) into equation (8) below.

[0072]

number

[0073] Here, we define the most ambiguous state, i.e., the silent state where no sound is input before utterance, as m=0. By also using equation (8), the streaming loss L hierarchical To approximate the Dirichlet distribution between the time relationships related to speech input, it is expressed by the following equation (9).

[0074]

number

[0075] Streaming Loss L hierarchical By minimizing this, we can learn the following for each word in a streaming state. In other words, if the amount of information in the audio data is sufficient, the system will approach the Dirichlet distribution of an offline state where all audio is present, while if the amount of information in the audio data is insufficient, it will approach the Dirichlet distribution of an ambiguous silent state.

[0076] <<Streaming Loss L skip Calculation >> Streaming Loss L skip This is a loss that facilitates the Dirichlet distribution during streaming speech recognition to approach the Dirichlet distribution during offline speech recognition. Streaming loss L skip The Dirichlet distribution in an offline state can be calculated using the following equation (10).

[0077]

number

[0078] The learning unit 22 processes the streaming loss L hierarchical ,L skip and offline loss L offline Based on the sum of these, the AED type model is learned. Specifically, the learning unit 22 calculates the loss L using the following equation (11). Then, the learning unit 22 causes each means 1A of the learning device 1 to execute processing so that the loss L is minimized.

[0079]

number

[0080] As described above, the learning device 1 learns the speech in progress necessary for streaming speech recognition by duplicating and masking the acoustic feature vector 100. At this time, since the learning device 1 uses EDL, it is possible to calculate the confidence level, i.e., uncertainty, for the output words of the AED model. Then, during streaming speech recognition, by calculating the uncertainty for the output words of the AED model, the reliability of the speech recognition result can be determined based on that uncertainty. In this way, the learning device 1 can apply the AED model to streaming speech recognition.

[0081] [Operation of the learning device] Refer to Figure 4 to explain the operation of the learning device 1. As shown in Figure 4, in step S1, the acoustic feature extraction unit 10 extracts acoustic features from the training audio data and generates an acoustic feature vector 100.

[0082] In step S2, the replication unit 11 replicates the acoustic feature vector 100 M times. In step S3, the mask processing unit 12 performs masking in a stepped manner. In step S4, the control signal insertion unit 13 inserts a streaming control signal or an offline control signal into each masked acoustic feature vector 100.

[0083] In step S5, the encoder 14 outputs a first vector matrix 110 by encoding each acoustic feature vector 100 into which a streaming control signal or an offline control signal has been inserted. In step S6, the language feature extraction unit 15 extracts language features from the character / word sequence corresponding to the speech data and generates a language feature vector 120.

[0084] In step S7, the decoder 16 decodes the language feature vector 120 and the first vector matrix 110 to obtain the second vector matrix o (l,m) ,o (l,offline) Outputs. In step S8, the α calculation unit 17 calculates α from the second vector matrix o (l,m) , o (l,offline) . (l,m) , α (l,offline) . In step S9, the Dirichlet distribution calculation unit 18 calculates the Dirichlet distributions Dir(p|α (l,m) ), Dir(p|α (l,offline) ). (l,m) ), Dir(p|α (l,offline) ).

[0085] In step S10, the streaming loss calculation unit 19 calculates the streaming losses L (l,m) ), Dir(p|α (l,offline) ). hierarchical , L skip . In step S11, the output probability calculation unit 20 calculates the output probability of a word by the softmax function from α (l,m) , α (l,offline) .

[0086] In step S12, the offline loss calculation unit 21 calculates the offline loss L offline from the output probability of a word. In step S13, the learning unit 22 determines whether to end the learning of the AED type model. For example, when the parameters of the AED type model are no longer updated, the learning unit 22 determines to end the learning.

[0087] If the learning is ended (Yes in step S13), the learning device 1 ends its operation. If the learning is not ended (No in step S13), the learning device 1 reads the next audio data and returns to the process of step S1.

[0088] (Second Embodiment) Referring to FIG. 5, the speech recognition device 2 according to the second embodiment will be described. The speech recognition device 2 performs speech recognition using an AED-type model learned by the learning device 1. As shown in Figure 5, the speech recognition device 2 comprises a streaming speech recognition device 2A that performs streaming speech recognition and an offline speech recognition device 2B that performs offline speech recognition. The speech recognition device 2 also receives speech data for recognition. The speech data for recognition is audio data of an arbitrary sentence being read aloud.

[0089] In other words, the speech recognition device 2 can switch between streaming speech recognition and offline speech recognition by a control signal. In the case of streaming speech recognition, the AED-type model calculates uncertainty appropriate for the streaming state, enabling streaming speech recognition based on uncertainty. In the case of offline speech recognition, the AED-type model can no longer calculate effective uncertainty, but there is no delay in the output of the speech recognition result.

[0090] The speech recognition device 2 outputs speech recognition results sequentially using streaming speech recognition by combining it with voice activity detection (VAD). Furthermore, after detecting a speech segment, the speech recognition device 2 can switch to offline speech recognition and output the remaining speech recognition results. Additionally, the user can switch between streaming speech recognition and offline speech recognition at any time.

[0091] [Configuration of a streaming speech recognition device] Referring to Figure 6, the configuration of the streaming speech recognition device 2A will be explained. The streaming speech recognition device 2A performs streaming speech recognition using an AED-type model trained on the learning device 1. As shown in Figure 6, the streaming speech recognition device 2A comprises an acoustic feature extraction unit 30, a control signal insertion unit 31, an encoder 32, a language feature extraction unit 33, a decoder 34, an α calculation unit (input value calculation unit) 35, an output probability calculation unit 36, an uncertainty calculation unit 37, and a determination unit 38.

[0092] The streaming speech recognition device 2A receives a sequence of characters or words corresponding to the speech data to be recognized. The recognition result of the streaming speech recognition device 2A can be used as the sequence of characters or words.

[0093] The acoustic feature extraction unit 30 extracts acoustic features from the speech data for recognition and generates an acoustic feature vector 200. The acoustic feature extraction unit 30 can extract acoustic features using the same method as the acoustic feature extraction unit 10 in Figure 2. In the example in Figure 6, the acoustic feature vector 200 consists of two elements 201. The acoustic feature extraction unit 30 outputs the generated acoustic feature vector 200 to the control signal insertion unit 31.

[0094] The control signal insertion unit 31 inserts a streaming control signal 103 into the acoustic feature vector 200, indicating that streaming speech recognition is to be performed. Specifically, the control signal insertion unit 31 inserts the streaming control signal 103 at the end of each element 201 that makes up the acoustic feature vector 200. The control signal insertion unit 31 outputs the acoustic feature vector 200 to the encoder 32.

[0095] Encoder 32 encodes the acoustic feature vector 200 into which the streaming control signal 103 is inserted. Encoder 32, like encoder 14 in Figure 2, is encoded from the first layer encoder 321 to N e Encoder 32 of the first layer Ne It includes everything up to that point. The encoder 32 outputs its output result, the acoustic feature vector 210, to the decoder 34. In the example in Figure 6, the acoustic feature vector 210 consists of two elements 221 and a streaming control signal 103.

[0096] The language feature quantity extraction unit 33 extracts language feature quantities from the character / word sequence which is the speech recognition result by the AED type model, and generates a language feature vector 220. The language feature quantity extraction unit 33 can extract language feature quantities by the same method as the language feature quantity extraction unit 15 in FIG. 2. In the example of FIG. 6, the language feature vector 220 is composed of one element 221. The language feature quantity extraction unit 33 outputs the generated language feature vector 220 to the decoder 34.

[0097] The decoder 34 decodes the language feature vector 220 and the output of the encoder 32. The decoder 34, similar to the decoder 16 in FIG. 2, includes a decoder 341 in the first layer to the decoder 34 in the N d layer. Nd Each layer of the decoder 34 includes a self-attention unit 34S that performs self-attention processing and a cross-attention unit 34C that performs cross-attention processing. In the example of FIG. 6, the vector 230 output by the decoder 34 is composed of one element 231.

[0098] The α calculation unit 35 calculates α representing the evidence in the EDL from the output of the decoder 34. The α calculation unit 35 calculates α by the same method as the α calculation unit 17 in FIG. 2. (l) k (l) k The α calculation unit 35 outputs the calculated α (l) k to the output probability calculation unit 36.

[0099] The output probability calculation unit 36 calculates the output probability p of the word from α (l) k by the softmax function. The output probability calculation unit 36 calculates the output probability p of the word by the same method as the output probability calculation unit 20 in FIG. 2. (l) k (l) k The output probability calculation unit 36 outputs the calculated output probability p of the word (l) k to the uncertainty calculation unit 37.

[0100] The uncertainty calculation unit 37 calculates the word output probability p (l) k The uncertainty of the AED type model is calculated using the following. Specifically, the uncertainty calculation unit 37 calculates the uncertainty U by the uncertainty calculation process shown in equation (12) below. (l) dis The uncertainty calculation unit 37 then calculates the calculated uncertainty U. (l) dis The result is output to the determination unit 38.

[0101]

number

[0102] The determination unit 38 determines uncertainty U (l) dis A pre-set threshold U TH This determines whether or not it is less than the threshold U. TH This can be set arbitrarily by the user. The determination unit 38 then determines uncertainty U (l) di Threshold U TH If the uncertainty U is less than the value, the result of streaming speech recognition by the AED-type model is output. On the other hand, the determination unit 38 determines the uncertainty U (l) di Threshold U TH If it is not less than [a certain value], the next audio data is read and processed by the acoustic feature extraction unit 30.

[0103] As described above, the streaming speech recognition device 2A can perform streaming speech recognition because it uses an AED-type model that has learned the necessary speech in progress. At this time, the streaming speech recognition device 2A can calculate the uncertainty of the output words of the AED-type model and determine the reliability of the speech recognition result according to that uncertainty. If the speech recognition result is unreliable, the streaming speech recognition device 2A reads the next audio data to increase the amount of information and performs speech recognition again.

[0104] [Configuration of Offline Speech Recognition Device] Referring to Figure 7, the configuration of the offline speech recognition device 2B will be explained in terms of how it differs from the streaming speech recognition device 2A. The offline speech recognition device 2B performs offline speech recognition using the AED-type model learned by the learning device 1. Furthermore, the offline speech recognition device 2B is similar to speech recognition of a general AED-type model, except for the insertion of the offline control signal 104.

[0105] As shown in Figure 7, the offline speech recognition device 2B comprises an acoustic feature extraction unit 30, a control signal insertion unit 31B, an encoder 32, a language feature extraction unit 33, a decoder 34, an output probability calculation unit 36, and a recognition result output unit 39.

[0106] The control signal insertion unit 31B inserts an offline control signal 104 into the acoustic feature vector 200. Specifically, the control signal insertion unit 31B inserts the offline control signal 104 at the end of each element 201 that makes up the acoustic feature vector 200. The control signal insertion unit 31B outputs the acoustic feature vector 200 to the encoder 32.

[0107] The recognition result output unit 39 outputs the output probability p of the word calculated by the output probability calculation unit 36. (l) k Based on this, the recognition result of the offline speech recognition device 2B is output. In other words, the recognition result output unit 39 outputs the word output probability p (l) k The system outputs the character / word sequence that maximizes the value as the recognition result.

[0108] Note that the means other than the control signal insertion unit 31B and the recognition result output unit 39 are the same as those in the streaming speech recognition device 2A, so their explanation will be omitted. As described above, the offline speech recognition device 2B can perform offline speech recognition.

[0109] [Operation of the Streaming Speech Recognition Device] Referring to Figure 8, the operation of the streaming speech recognition device 2A will be explained. As shown in Figure 8, in step S21, the acoustic feature extraction unit 30 extracts acoustic features from the speech data for recognition and generates an acoustic feature vector 200. In step S22, the control signal insertion unit 31 inserts the streaming control signal 103 into the acoustic feature vector 200.

[0110] In step S23, the encoder 32 encodes the acoustic feature vector 200 into which the streaming control signal 103 is inserted. In step S24, the language feature extraction unit 33 extracts language features from the character / word sequence, which is the speech recognition result, and generates a language feature vector 220. In step S25, the decoder 34 decodes the language feature vector 220 and the output of the encoder 32.

[0111] In step S26, the α calculation unit 35 calculates α from the output of the decoder 34. (l) k Calculate. In step S27, the output probability calculation unit 36 ​​calculates α (l) k From this, the output probability of a word p is calculated using the softmax function. (l) k Calculate. In step S28, the uncertainty calculation unit 37 calculates the word output probability p (l) k Using the uncertainty U of the AED type model, (l) dis Calculate.

[0112] In step S29, the determination unit 38 determines the uncertainty U (l) dis A pre-set threshold U TH Determine whether it is less than or equal to. Uncertainty U (l) di Threshold U TH If the result is less than (Yes in step S29), the determination unit 38 proceeds to the process in step S30. Uncertainty U (l) diThreshold U TH If it is not less than (No in step S29), the determination unit 38 proceeds to the process in step S31.

[0113] In step S30, the determination unit 38 outputs the result of streaming speech recognition by the AED type model and returns to the processing in step S24. In step S31, the determination unit 38 reads the next audio data and returns to the process in step S21.

[0114] [Offline speech recognition device operation] Referring to Figure 9, the operation of the offline speech recognition device 2B will be explained. The processes in steps S21, S23-S25, and S27 are the same as in Figure 8, so their explanation is omitted.

[0115] As shown in Figure 9, in step S32, the control signal insertion unit 31B inserts an offline control signal 104 into the acoustic feature vector 200. In step S33, the recognition result output unit 39 outputs the word output probability p (l) k Based on this, the recognition result of the offline speech recognition device 2B is output.

[0116] Although each embodiment has been described in detail above, the present invention is not limited to the embodiments described above, and includes design changes and the like that that do not depart from the spirit of the present invention.

[0117] Although the explanation described the learning device as training an AED-type model that supports both streaming and offline speech recognition, it is not limited to this. In other words, the learning device may train an AED-type model that supports only streaming speech recognition. In this case, the learning device 1 does not need to train the offline loss.

[0118] In the embodiments described above, the learning device and the streaming speech recognition device were described as independent hardware, but the present invention is not limited thereto. For example, the present invention can also be realized by a program that causes hardware resources such as the CPU, memory, and hard disk of a computer to function as the learning device or streaming speech recognition device described above. This program may be distributed via a communication line, or it may be written to a recording medium such as a CD-ROM or flash memory and distributed. [Explanation of symbols]

[0119] 1. Learning device 10 Acoustic Feature Extraction Unit 11 Reproduction Department 12 Mask Processing 13 Control signal insertion section 14 encoders 15. Language Feature Extraction Unit 16 Decoders 17. Alpha Calculation Unit (Input Value Calculation Unit) 18. Dirichlet distribution calculation unit 19 Streaming Loss Calculation Unit 20 Output probability calculation unit 21 Offline Loss Calculation Unit 22 Learning Department 2A Streaming Speech Recognition Device 30 Acoustic Feature Extraction Unit 31 Control signal insertion section 32 encoders 33. Language Feature Extraction Unit 34 Decoders 35α calculation part 36 Output Probability Calculation Unit 37. Uncertainty Calculation Unit 38 Judgment section

Claims

1. A learning device for learning an AED-type model capable of streaming speech recognition, An acoustic feature extraction unit extracts acoustic features from training audio data and generates an acoustic feature vector in which multiple elements are consecutive in the time direction, A replication unit that replicates the aforementioned acoustic feature vector a predetermined number of times, A masking processing unit sets a predetermined number of elements that are consecutive from the end of the aforementioned acoustic feature vector to be masked, such that the number of masked elements included in each acoustic feature vector decreases, and masks the masked elements. A control signal insertion unit inserts a streaming control signal indicating that streaming speech recognition should be performed for each masked acoustic feature vector of the aforementioned mask target, An encoder that outputs a first vector matrix by encoding each acoustic feature vector into which the aforementioned streaming control signal is inserted, A language feature extraction unit extracts language features from the character / word sequence corresponding to the aforementioned audio data and generates a language feature vector in which multiple elements are consecutive in the time direction. A decoder that outputs a second vector matrix by decoding the language feature vector and the first vector matrix, An input value calculation unit calculates input values ​​representing evidence in EDL from the second vector matrix, A Dirichlet distribution calculation unit calculates a Dirichlet distribution from the input value, A streaming loss calculation unit calculates the streaming loss, which is the loss function during streaming speech recognition, from the Dirichlet distribution. An output probability calculation unit calculates the probability of outputting a word using the softmax function from the input value, An offline loss calculation unit calculates the offline loss, which is the loss function during offline speech recognition, from the output probability of the aforementioned word. A learning unit that learns the AED type model based on the sum of the streaming loss and the offline loss, A learning device characterized by being equipped with the following features.

2. The streaming loss includes a loss for learning the process by which the Dirichlet distribution changes from an ambiguous state to a steep state, and a loss for promoting the Dirichlet distribution during streaming speech recognition to approach the Dirichlet distribution during offline speech recognition. The learning device according to feature 1.

3. The mask processing unit does not set the mask target for one of the acoustic feature vectors duplicated by the duplication unit. The control signal insertion unit inserts an offline control signal indicating that offline speech recognition should be performed for acoustic feature vectors for which the mask target has not been set. The learning device according to feature 1.

4. A speech recognition device that performs streaming speech recognition using an AED-type model learned with the learning device described in claim 1, An acoustic feature extraction unit that extracts acoustic features from speech data for recognition and generates acoustic feature vectors, A control signal insertion unit inserts a streaming control signal into the aforementioned acoustic feature vector, which indicates that streaming speech recognition should be performed. An encoder that encodes the acoustic feature vector into which the aforementioned streaming control signal has been inserted, A language feature extraction unit extracts language features from the character / word sequence resulting from speech recognition by the aforementioned AED-type model and generates a language feature vector. A decoder that decodes the language feature vector and the output of the encoder, An input value calculation unit calculates an input value representing evidence in EDL from the output of the decoder, An output probability calculation unit calculates the probability of outputting a word using the softmax function from the input value, An uncertainty calculation unit calculates the uncertainty of the AED-type model using the output probability of the aforementioned word, A determination unit that determines whether the uncertainty is less than a preset threshold, outputs the result of streaming speech recognition by the AED type model if the uncertainty is less than the threshold, and causes the acoustic feature extraction unit to process the next audio data if the uncertainty is not less than the threshold. A voice recognition device characterized by being equipped with the following features.

5. A program for causing a computer to function as the learning device described in claim 1.

6. A program for causing a computer to function as the speech recognition device described in claim 4.

Images