Voice recognition device, voice recognition method and program

Abstract

To provide a voice recognition device, a voice recognition method and a program by which voice recognition by an E2E model can improve robustness.SOLUTION: A voice recognition device performs voice recognition using an End-to-End model, and comprises: an encoder as a model which converts an input voice signal into a featured expression; a decoder as a model which converts voice data into a text by using the output of the encoder; and a learning part which selects a block length of the voice signal input to the encoder at random and lets the encoder and decoder learn.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a voice recognition device, a voice recognition method, and a program. [Background technology]

[0002] Two types of speech recognition technology have been developed and put into practical use: online recognition, which performs speech recognition sequentially for each speech block, and offline recognition, which performs speech recognition using the entire utterance (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-92836 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, although online recognition has a fast response, its performance is often lower than that of offline recognition. However, depending on the utterance, offline recognition may have a lower performance than online recognition. As such, speech recognition using conventional E2E models had problems with robustness.

[0005] The present invention has been made in consideration of the above problems, and aims to provide a speech recognition device, a speech recognition method, and a program that can improve the robustness of speech recognition using an E2E model. [Means for solving the problem]

[0006] (1) In order to achieve the above-mentioned object, a speech recognition device according to one embodiment of the present invention is a speech recognition device that performs speech recognition using an end-to-end model, and includes an encoder which is a model that converts an input speech signal into a characteristic expression, a decoder which is a model that converts speech data into text using the output of the encoder, and a learning unit that randomly selects a block length of the speech signal input to the encoder and trains the encoder and the decoder.

[0007] (2) In addition, in the speech recognition device according to one aspect described in (1) above, the encoder may include a first encoder that performs processing online and a second encoder that performs processing offline, the decoder may include a first decoder that performs processing online and a second decoder that performs processing offline, and the device may further include an integration unit that outputs a final recognition result by rescoring an output of the first decoder and an output of the second decoder during inference.

[0008] (3) In addition, in the speech recognition device according to the embodiment described in (2) above, the second decoder may be configured to integrate an output of the first encoder and an output of the second encoder during training to learn the second decoder.

[0009] (4) In the speech recognition device according to the aspect described above in (3), the second decoder may stack the output of the first encoder and the output of the second encoder vertically.

[0010] (5) In addition, in the speech recognition device according to the aspect described in (2) above, the integration unit may perform inference on the speech data acquired using the second decoder to calculate a first score, calculate a second score for the same character string as the character string inferred by the second decoder using the first encoder, and re-score the first score and the second score by calculating a weighted sum to determine the text to be inferred.

[0011] (6) In order to achieve the above-mentioned object, a speech recognition method according to one embodiment of the present invention is a speech recognition method for a speech recognition device that performs speech recognition using an end-to-end model, in which an encoder converts an input speech signal into a characteristic expression, a decoder converts speech data into text using an output of the encoder, and a learning unit randomly selects a block length of the speech signal input to the encoder to train the encoder and the decoder.

[0012] (7) In order to achieve the above object, a program according to one embodiment of the present invention is a program that causes a computer of a speech recognition device that performs speech recognition using an end-to-end model having an encoder and a decoder to execute a first conversion step of converting an input speech signal into a characteristic expression, a second conversion step of converting the speech data into text using the output of the first conversion step, and a step of randomly selecting a block length of the speech signal input in the first conversion step to train the encoder and the decoder. Effect of the Invention

[0013] According to (1) to (7), speech recognition using the E2E model can improve robustness. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing an overview of a voice recognition process and configuration of the prior art. [Diagram 2] FIG. 1 is a diagram showing an overview of a voice recognition process and a configuration according to an embodiment. [Diagram 3] 1 is a diagram illustrating an example of the configuration of a voice recognition device according to an embodiment. [Figure 4] Figure 1 shows an example of hypotheses and scores generated by the offline decoder. [Diagram 5] This figure shows an example of hypotheses and scores generated by the online decoder. [Figure 6] 1 is a flowchart of a training process of the voice recognition device according to the embodiment. [Figure 7] 4 is a flowchart of a voice recognition process of the voice recognition device according to the embodiment. [Figure 8] FIG. 11 is a diagram illustrating an example program of a processing procedure performed by an integration unit according to the embodiment. [Figure 9] FIG. 13 is a diagram showing an example of RNN-T scoring. [Figure 10] FIG. 13 shows the evaluation results when using the CSJ corpus. [Figure 11] 1A and 1B are diagrams illustrating examples of texts speech-recognized by a conventional method and the method of the present embodiment. [Figure 12] FIG. 13 is a diagram showing an example of the effect of online DB learning. [Figure 13] FIG. 13 is a diagram showing the speech recognition rate and ED of the model of the present embodiment with DB learning and a separation model. [Figure 14] This figure shows the relationship between block size and CER in the offline mode of CSJ. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component is of a recognizable size. In addition, in all the drawings for explaining the embodiments, the same reference numerals are used for the parts having the same functions, and the repeated explanation is omitted. In addition, "based on XX" in this application means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0016] (E2E model) First, we will explain the End-to-End (E2E) model of speech recognition. In recent years, End-to-End (E2E) models have been attracting attention in speech recognition technology, replacing the conventional DNN-HMM (Deep Neural Network-Hidden Markov model) model. Deep learning using the E2E model is a system in which machine learning systems, which previously required multiple stages of processing from when input data was given until the results were output, are replaced with a single large neural network equipped with multiple layers and modules that perform various processes.

[0017] In speech recognition using such E2E models, there are online and offline speech recognition. Online models can perform speech recognition with minimal latency, and are used, for example, for real-time transcription. Offline models wait until the speech is finished before processing, and are used, for example, for generating transcriptions.

[0018] In recent years, there have been several attempts to combine online and offline models to improve performance. Furthermore, conventional speech recognition devices were trained with a fixed block size, but because the length of actual speech varies from case to case, conventional speech recognition devices trained with a block size of the same length were not able to adequately handle the situation.

[0019] For this reason, in this embodiment, as described below, the results of online and offline recognition are integrated, and further, a block length is randomly selected for learning.

[0020] (Outline of this embodiment) Here, an overview of the conventional technology and an overview of this embodiment will be described. FIG. 1 is a diagram showing an overview of speech recognition processing and configuration in the prior art. In the prior art, online speech recognition and offline speech recognition are performed separately. A speech signal X is input to an online encoder 901 and an offline encoder 911. In the online processing, learning and recognition are performed with a fixed block length, as in code g901. In the offline processing, processing is performed, for example, for each sentence, as in code g911.

[0021] In online processing, a speech signal X of a fixed block length is input to an online encoder 901, and the encoded signal is input to an online decoder 902. Then, the online decoder 902 sequentially outputs the text processed online. In the offline processing, a speech signal X is input to an offline encoder 911, and the encoded signal is input to an offline decoder 912. Then, the offline decoder 912 outputs text that has been subjected to offline processing.

[0022] FIG. 3 is a diagram showing an outline of the voice recognition process and configuration according to this embodiment. In this embodiment, online processing and offline processing are performed simultaneously. The first encoder 11 for online processing performs encoding processing on the input speech signal X for each phrase of a predetermined length, and outputs the processing result to the first decoder 12 for online processing and the second decoder 14 for offline processing. During training, the first encoder 11 changes the block size of the speech signal and learns, as in code g11. The first decoder 12 sequentially outputs the results of the decoding processing (code g12), and also outputs them to the integration unit 15.

[0023] The second encoder 13 for offline use performs encoding processing on the input speech signal X for each sentence, for example, and outputs the processing result to the second decoder 14 for offline processing. During training, the second encoder 13 learns using the speech signal of, for example, one sentence. During training, the second decoder 14 learns using the output of the first encoder 11 and the output of the second encoder 13. The second decoder 14 holds the output of the first encoder 11, and after holding the output for, for example, one sentence, it learns using the output of the first encoder 11 and the output of the second encoder 13. The second decoder 14 performs a decoding process on the input result, and outputs the decoded result to the integration unit 15. The integrating unit 15 integrates the output of the first decoder 12 and the output of the second decoder 14, and outputs the integrated result (code g22). The processing of each unit and the reference symbols in FIG. 2 will be described later.

[0024] Generally, in the E2E model, an encoder is a model that converts voice data into a characteristic expression. For example, the encoder extracts frequency components in the time domain or frequency domain of the voice data and converts them into a vector (e.g., 256th order) that expresses the voice characteristics. Generally, in an E2E model, a decoder converts voice data into text using the output of an encoder. In this way, in the E2E model, the encoder and decoder work together to perform speech recognition. However, each model needs to be trained in advance using a corpus, etc.

[0025] In this embodiment, the online model and the offline model are integrated as one model for learning. That is, the second decoder 14 for offline use uses the output of the first encoder 11 and the output of the second encoder 13. This integration is performed both during learning and during speech recognition (inference). The integration is performed, for example, when one utterance is completed, that is, when the offline processing is completed. For this reason, the second decoder 14 holds the output of the first encoder 11 until the offline processing is completed.

[0026] In this embodiment, the block size is changed randomly when inputting to the online first encoder 11. The block size is, for example, 5, 10, or 20. During speech recognition, the block size is fixed to a predetermined size.

[0027] Furthermore, in this embodiment, integration is also performed during speech recognition. That is, the integration unit 15 integrates the output of the first decoder 12 and the output of the second decoder 14 to perform inference (text estimation).

[0028] (Voice recognition device) Next, the voice recognition device 1 of this embodiment will be described. 1 is a diagram showing an example of the configuration of a voice recognition device according to the present embodiment. As shown in FIG. 1, the voice recognition device 1 includes, for example, an acquisition unit 2, an extraction unit 3, a processing unit 4, a storage unit 5, and an output unit 6. The processing unit 4 includes, for example, a first encoder 11, a first decoder 12, a second encoder 13, a second decoder 14, an integration unit 15, and a learning processing unit 16.

[0029] The acquisition unit 2 acquires an audio signal. The audio signal may be recorded data or may be an audio signal collected in real time. In the case of collecting audio, the acquisition unit 2 is a microphone.

[0030] During training, the extraction unit 3 extracts (cuts out) a block length from the acquired voice signal, for example, by randomly changing the block length, and outputs the extracted voice signal to the first encoder 11. During speech recognition, the extraction unit 3 extracts (cuts out) a signal for each predetermined block length from the acquired voice signal, and outputs the extracted voice signal to the first encoder 11. During training and recognition, the extraction unit 3 extracts a voice signal for, for example, one sentence from the acquired voice signal, and outputs the extracted voice signal to the second encoder 13. The extraction unit 3 performs speech section detection using a well-known method, and extracts a voice signal for, for example, one sentence.

[0031] The processing unit 4 trains (learns) each unit using the input voice signal, and performs voice recognition processing using each unit that has already been trained.

[0032] The first encoder 11 is an encoder for online processing, for example, a blockwise conformer encoder. The first encoder 11 outputs a hidden state H on to the first decoder 12 and the second decoder 14. Note that the training method of the first encoder 11, the hidden state H on etc. will be discussed later.

[0033] The first decoder 12 is a decoder for online processing, for example, a transducer decoder. A transducer is one of the offline-based processing techniques. The first decoder 12 outputs the result of online recognition y on The first decoder 12 outputs the recognition result (text) and a score α on to the integrating unit 15. The training method of the first decoder 12 will be described later. The score is, for example, a value between 0 and 1.00.

[0034] The second encoder 13 is an encoder for offline processing, for example, a full-context conformer encoder. Note that a conformer is one of offline-based processing techniques. The second encoder 13 outputs a hidden state H off to the first decoder 12 and the second decoder 14. Note that the training method of the second encoder 13, the method of calculating the score, and the hidden state H off etc. will be discussed later.

[0035] The second decoder 14 is a decoder for offline processing, for example a transducer decoder. The second decoder 14 also receives the recognition result (text) and a score α off The second decoder 14 outputs the beam search result, hypothesis hyps, to the integration unit 15. Note that the beam search is a process of searching for character candidates for each character in recognition and calculating the probability of their likelihood. At this time, the top four character candidates, for example, are selected. Note that the training method of the second decoder 14 and the method of calculating the score will be described later.

[0036] The integrating unit 15 integrates the output result of the first decoder 12 and the output result of the second decoder 14 to generate a speech recognition result y off The processing method of the integration unit 15 will be described later.

[0037] During training, the learning processing unit 16 trains each unit. During recognition, the learning processing unit 16 performs speech recognition processing using each unit that has already been trained.

[0038] The storage unit 5 stores programs, thresholds, formulas, etc. used during training and speech recognition.

[0039] The output unit 6 outputs the recognition result output by the integration unit 15 to an external device. The external device is, for example, an image display device, a personal computer, a mobile terminal, a dedicated terminal, or the like.

[0040] (Encoder) Next, the processing of the first encoder 11 and the second encoder 13 will be further explained. The second encoder 13 is, for example, a full-context conformer encoder. A conformer encoder, for example, consists of two convolutional layers for downsampling, a linear projection layer, and a positional encoding layer, followed by N conformer blocks. The convolutional layers receive a sequence of speech features X and subsample it. The conformer blocks convert the sampled feature sequence into a hidden state H of length T using the following equation (1): off =[h 1, … , h T ]. Note that what is input to the encoder in this way is not the speech signal itself, but the acoustic feature sequence X extracted from the speech signal. As shown in FIG. 2, the second encoder 13 converts this hidden state H off The acquisition unit 2 calculates, for example, a static Mel-Scale Log Spectrum (MSLS), a delta MSLS, and one delta power every predetermined time (for example, 10 ms) to calculate an acoustic feature. The MSLS is obtained by using a spectral feature as a feature of the audio signal and performing an inverse discrete cosine transform on MFCC (Mel Frequency Cepstrum Coefficient).

[0041]

number

[0042] Note that each conformer block includes, for example, two feedforward layers, a multi-headed self-attention layer, a convolutional layer, a layer normalization layer, and a residual connection.

[0043] The first encoder 11 can compute block-wise in a streaming scenario. bThe b-th block is defined as follows:

[0044]

number

[0045] In formula (2), L block represents the lock size, and L hop represents the hop length, which is the length by which the window function is shifted. For each block including a hidden state of L block length, the hidden state corresponding to the b-th block is encoded. This process is repeated sequentially, and finally, a hidden state of T block length is obtained as shown in the following equation (3). As shown in FIG. 2, the first encoder 11 encodes this hidden state H on Output.

[0046]

number

[0047] (decoder) Next, the processing of the first decoder 12 and the second decoder 14 will be further explained. The first decoder 12 and the second decoder 14 are, for example, transducer decoders. A transducer-decoder, for example, consists of a prediction network and a combination network. The prediction network calculates the number of tokens g s-1 By conditioning on the high-level representation g s where s represents the index of the non-whitespace token, written as follows:

[0048]

number

[0049] The joint network is composed of the t-th hidden state vector h tand g output from equation (4) s This is a feed-forward network that combines the above and is described as the following equation (5).

[0050]

number

[0051] The transducer model marginalizes a potential alignment z given a text string y as follows:

[0052]

number

[0053] In (6), S is the total length of the complete token sequence, i is the position of the alignment path of length (T+S), and t i The th decoder state and s i th token, and Π is the product of P(*) from i=1 to (T+S). For both online and offline transducers, the model parameters are optimized by minimizing the negative log-likelihood as shown in Eqs. (7) and (8).

[0054]

number

[0055]

number

[0056] In equation (7), y on is the output (text) of the first decoder 12 by online processing. y off is the output (text) of the second decoder 14 by offline processing. on are the model parameters of the first decoder 12. offare the model parameters of the second decoder 14.

[0057] As described above, the speech recognition device 1 of this embodiment includes a block-wise encoder and a full-context encoder that can operate in both online and offline modes. The speech feature sequence X is supplied to the block-wise encoder and the full-context encoder in parallel, as shown in FIG.

[0058] In online mode, the block-wise encoder generates a hidden state vector H that is fed to the online decoder, similar to a conventional cascaded-encoder structure (see, for example, Ref. 1). on Generate.

[0059] Reference 1; A. Narayanan, Tara N. Sainath, et al., “Cascaded encoders for unifying streaming and non-streaming ASR”, in Proc. ICASSP, 2020, pp.5629-5633

[0060] In the offline mode, the hidden state vector H on , H off Unlike conventional cascade encoders, H is stacked vertically as shown in the following equations (8) and (9). on , H off , H off para All have the same length. Since it is time × dimension, for example, if the dimensions before concat are each 256, they are stacked vertically. off para becomes a 512-dimensional hidden state vector.

[0061]

number

[0062]

number

[0063] Note that casc stands for cascade connection and para stands for parallel connection. Also, in equation (10), concat(H on ,H off ) is the hidden state vector H on and H off The equation (10) is a process for integrating the output of the first encoder 11 and the output of the second encoder 13. The first encoder 11 and the second encoder 13 each output a hidden state vector, which is, for example, a 256-dimensional vector, at each time. For example, when 10 frames are output, 10 256-dimensional vectors are output. The second decoder 14 vertically stacks, for example, the 256-dimensional × 10 vector output by the first encoder 11 and the 256-dimensional × 10 vector output by the second encoder 13, and integrates them into a 512-dimensional × 10 vector.

[0064] The structure of this embodiment can avoid the accumulation of errors that often occur in cascade structures. In addition, in this embodiment, the block-wise encoder and the full-context encoder are designed with the concept that they can better extract local features and global features, respectively. That is, in this embodiment, the hidden state vector H on and H off Combining the hidden state vectors of both encoders enables an offline mode that uses both local and global features.

[0065] In addition, multi-task learning of online output and offline output can improve the robustness of the block-wise encoder, as described later. Furthermore, in this embodiment, a one-pass beam search is used to further improve the performance of the offline mode, as described later. In this embodiment, separate transducer decoders are used to avoid the online mode and the offline mode from interfering with each other. Depending on the conditions of use, the transducer decoder may be shared.

[0066] (Integrated learning method) The model (encoder, decoder) of this embodiment is optimized by multi-task learning using a weighted sum of losses using the following equation (11), as shown in equations (7) and (8).

[0067]

number

[0068] In formula (11), λ represents the weight of learning. block In this embodiment, we used dynamic block (DB) learning, which dynamically changes the block size of each batch during training. min -L max ] (e.g., a predetermined range) at random.

[0069] By this process, according to the present embodiment, the robustness of the offline mode can be improved by increasing the variance of the output of the block-wise encoder during training. Furthermore, according to the present embodiment, the block size can be flexibly selected during inference without compromising performance. That is, according to the present embodiment, the block size can be flexibly adjusted according to the requirements of accuracy and latency.

[0070] Next, the one-pass beam search performed by the integrating unit 15 will be described. The integration unit 15 calculates a score using the offline model (offline second decoder 14) and generates a hypothesis for the speech recognition result. Furthermore, the integration unit 15 calculates the probability (score) of a character string that is the same as the character string hypothesized by the offline model using the online model (online first decoder 12).

[0071] In this embodiment, we perform a one-pass beam search that combines online and offline transducer decoders to improve the performance of the offline mode. Unlike conventional one-pass beam search methods (e.g., see Reference 2), this embodiment uses two RNN-T decoders to score hypotheses. In addition, conventional label-synchronized one-pass beam search (e.g., see Reference 3) calculates the probability of all possible alignment paths in the Connectionist Temporal Classification (CTC) prefix scoring. In contrast, this embodiment uses time synchronization to avoid this inefficient scoring.

[0072] Reference 2; Brian Yan, Siddharth Dalmia, et al., “CTC Alignments Improve Autoregressive Translation”, arXiv preprint arXiv:2210.05200, 2022 Reference 3; S. Watanabe, T. Hori, et al., “Hybrid CTC / Attention Architecture for End-to-End Speech Recognition”, IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240-1253, 2017

[0073] Figure 4 shows an example of hypotheses and scores generated by the offline decoder. Figure 5 shows an example of hypotheses and scores generated by the online decoder. In this example, the offline decoder gives "ohayougo" a score of 0.40, which is lower than the score of "ohayouga" at 0.50. However, the online decoder gives "ohayougo" a score of 0.60, which is lower than the score of "ohayouga" at 0.30. Note that the examples shown in Figures 4 and 5 are just examples.

[0074] (Training Procedure) Next, an example of a training process procedure will be described with reference to Fig. 6. The training process of the voice recognition device according to this embodiment is shown in a flowchart.

[0075] (Step S11) The processing unit 4 initializes parameters of each model (encoder, decoder, and integrating unit).

[0076] (Step S12) The acquisition unit 2 acquires a voice signal.

[0077] (Step S13) The acquisition unit 2 extracts acoustic features from the acquired acoustic signal using a known method.

[0078] (Step S14) The learning processing unit 16 changes the block size for extracting the audio signal, for example, randomly.

[0079] (Step S15) The learning processing unit 16 performs speech recognition and performs training processing of the model (the first encoder 11, the first decoder 12, the second encoder 13, and the second decoder 14) using the calculated acoustic feature sequence X. The processing unit 4 performs online processing and offline processing simultaneously.

[0080] (Step S16) The learning processing unit 16 determines whether the training process has ended. For example, the learning processing unit 16 determines that the training has ended when the training has been performed a predetermined number of times. If the training process has ended (Step S16; YES), the learning processing unit 16 ends the training process. If the training process has not ended (Step S16; NO), the learning processing unit 16 returns to the process of Step S12.

[0081] (Voice recognition processing procedure) Next, an example of a voice recognition process procedure will be described with reference to Fig. 7. The voice recognition process of the voice recognition device according to this embodiment is shown in a flowchart.

[0082] (Step S21) The processing unit 4 sets the block size used in online processing to a predetermined value.

[0083] (Step S22) The acquisition unit 2 acquires a voice signal.

[0084] (Step S23) The acquisition unit 2 extracts acoustic features from the acquired acoustic signal by a known method.

[0085] (Step S24) The processing unit 4 performs speech recognition processing using the trained model. Note that the processing unit 4 performs online processing and offline processing simultaneously.

[0086] (Step S25) The first decoder 12 sequentially outputs the recognition results obtained by the online processing via the output unit 6.

[0087] (Step S26) The processing unit 4 determines whether or not the offline processing has ended. If the offline processing has ended (Step S26; YES), the processing unit 4 proceeds to the processing of Step S27. If the offline processing has not ended (Step S26; NO), the processing unit 4 returns to the processing of Step S22.

[0088] (Step S27) The integrating unit 15 integrates the output of the first decoder 12 and the output of the second decoder 14. The process performed by the integrating unit 15 will be described later.

[0089] (Step S27) The processing unit 4 outputs, via the output unit 6, the recognition result obtained by integrating the results of the online processing and the offline processing by the integrating unit 15.

[0090] Here, the text that the voice recognition device 1 outputs during voice recognition will be described. The online first decoder 12 of the speech recognition device 1 sequentially performs speech recognition on speech data for each block input for each predetermined block size and outputs the result. In parallel with this, the offline second decoder 14 performs speech recognition on each sentence when an utterance of one sentence is captured. In response to this, the combining unit 15 combines one sentence's worth of text processed online (e.g., code g104 in FIG. 11) with one sentence's worth of text processed offline (e.g., code g105 in FIG. 11), and outputs the final recognition result (e.g., code g106 in FIG. 11).

[0091] The above-mentioned steps of the training method and steps of the voice recognition method are merely examples, and the present invention is not limited to these. For example, the voice recognition device 1 may perform several processes in parallel.

[0092] (Example of integration process) Next, a description will be given of an example of the processing procedure performed by the integrating unit 15 in step S27. Fig. 8 is a diagram showing an example of a program of the processing procedure performed by the integrating unit according to this embodiment.

[0093] The integration unit 15 initializes the hypothesis hyps (step S51).

[0094] The integration unit 15 processes the 3rd to 15th lines from t=1 to T (step S52). In this process, for example, if it is possible to predict up to "Good morning" at time t=t-1, it predicts the characters after "Good morning" at the next time t=t.

[0095] In the process of step S52, the integration unit 15 assigns the hypothesis hyps obtained at time t-1 to A, and initializes ext_hyps, which is a hypothesis obtained at the current time (step S521). The hypothesis hyps obtained at time t-1 is, for example, "Good morning." The hypothesis ext_hyps obtained at the current time is, for example, a hypothesis that adds the next character to "Good morning."

[0096] In the process of step S52, the integration unit 15 uses the offline transducer decoder as the main decoder to generate initial hypotheses and calculate the score of each hypothesis (step S522). In this example, the top four hypothesis candidates are used. The top four are, for example, in the example of FIG. 4, the candidates for the characters following "ohayou" at the previous time t=t-1, which are "go", "ga", "gaji", and "ro", which have high probabilities. Through this process, hypotheses are generated by the offline decoder of FIG. 6, for example, and the scores are calculated.

[0097] In the process of step S522, the integration unit 15 selects the top k (for example, 4) candidates from among the candidates (top-k_pre(P off (z t ))) and calculate the score of each candidate as shown in FIG. 6 (step S5221). t is the current hypothetical text.

[0098] In the process of step S52, after step S522 is completed, the integration unit 15 calculates the probability of outputting the candidate hypothesis character string obtained by the offline model for each character string by using the online model as well (step S523). Offline score alpha off is expressed as the following equation (12), and the online score α is calculated using the following equation (13). on Calculate.

[0099]

number

[0100]

number

[0101] In this way, the online transducer decoder is used to score the generated hypotheses and search for more probable hypotheses in OnScore in equation (13). Specifically, in equation (13), the probability Pon(z t-1 ) to generate ext_hyps from the previous hypothesis. In the transducer model, the character at each time is predicted.

[0102] Fig. 9 is a diagram showing an example of RNN-T scoring. In Fig. 9, the horizontal axis represents time and the vertical axis represents characters, and it is determining which token (character) is to be output at each time. For example, at time t=1, each point indicates that there is a pattern of outputting nothing ("0"), a pattern of outputting one character ("1"), and a pattern of outputting two characters ("2"). The probability of each point is stored. The probability at time t2 indicates that it is necessary to add up (arrow g52) all the probabilities moving from the three points at the previous time, t=1, to the point at time t=2 as indicated by arrow g51.

[0103] Note that since the transducer-decoder may emit multiple tokens for each time frame, as mentioned above, it is necessary to sum all possible paths (the three arrows from time t=1 to time t=2 in Figure 9). For this reason, the integration unit 15 calculates an integrated score α joint is calculated using the decoding weight μ described as in the following equation (14). Note that the value of the weight μ may be set according to the usage environment, the number of learning times, the corpus, and the like.

[0104]

number

[0105] In step S52, after step S523 is completed, the integration unit 15 adopts the more plausible hypothesis based on the weighted sum (step S53).

[0106] In this manner, in this embodiment, a score is calculated for each text for which a hypothesis was generated offline, and then the score is recalculated for the same strings using the online model. The weighted average of the scores calculated by the online model and the scores calculated offline is calculated to select the text with the highest probability.

[0107] The process (algorithm) of the integration unit 15 described with reference to FIG. 8 is just an example, and the present invention is not limited to this.

[0108] (evaluation) An example of the results of evaluating the above-described voice recognition device 1 will be described. The input features consist of 80-dimensional mel-scale filterbank features with a window size of 512 samples and a hop length of 160 samples, sampled at 16 kHz. The second encoder 13 (full context encoder) and the first encoder 11 (block-wise encoder) used in the evaluation were configured with two convolutional layers, a 256-dimensional linear projection layer, and 12 conformer layers with 1024 linear units. During learning, the learning processing unit 16 performs the learning process using the block size L block was randomly selected from [5-50] using DB learning.

[0109] In both transducer decoders (first decoder 12, second decoder 14), one LSTM layer with hidden size 256 and one linear layer with joint size 320 were used in the prediction network and the combination network. The joint model was trained for 50 epochs using the Adam optimizer with a learning rate of 0.0015 and a learning weight, λ, in equation (11) of 0.5. The decoder weight, μ, in equation (12) of the online decoder (first decoder 12), was set to 0.3. The speech recognition device 1 of this embodiment was evaluated using two data sets, the Corpus of Spontaneous Japanese (CSJ) (Reference 4) and LibriSpeech 960h (Reference 5).

[0110] Reference 4: K. Maekawa, “Corpus of spontaneous Japanese: Its design and evaluation”, in ISCA & IEEE Workshop on SSPR, 2003 Reference 5; Vassil Panayotov, Guoguo Chen, et al., “Librispeech: an asr corpus based on public domain audio books”, in Proc. ICASSP, 2015, pp. 5206-5210.

[0111] The CSJ corpus consisted of 299 hours of speech data from academic presentations. The character error rate / word error rate (CER / WER) was calculated for each of the CSJ and LibriSpeech. To evaluate the delay in the online mode, we calculated the emission delay (ED) defined in Reference 6. The evaluation was performed using the ESPnet (Reference 7) toolkit.

[0112] Reference 6; Jay Mahadeokar, Yuan Shangguan, et al., “Alignment restricted streaming recurrent neural network transducer”, in Proc. SLT, 2020, pp. 52-59 Reference 7;Shinji Watanabe, Takaaki Hori, et al., “ESPnet: End-to-end speech processing toolkit”, in Proc. INTERSPEECH, 2018, pp. 2207-2211.

[0113] FIG. 10 is a diagram showing the evaluation results when the CSJ corpus, which is a spontaneous Japanese corpus, is used. In FIG. 10, "Separated" is a comparative example, and is an evaluation result using independent processing. "Cascaded encoder" is a comparative example, and is an evaluation result using a cascaded encoder. "Parallel encoder" is an evaluation result using an encoder connected in parallel (with an integrated encoder) according to the embodiment. "+DBT" is an evaluation result when the block size is changed during learning in the present embodiment. "+2-pass rescoring" is an evaluation result when rescoring is performed using the conventional two-pass method (see Reference 1). "+DBT+1-pass" is an evaluation result when learning is performed with a different block size in the present embodiment, and processing is also performed in the integration unit 15. Note that each value is the speech recognition rate CER (%), eval1, eval2, and eval3 are the names of evaluation datasets provided by the CSJ dataset, and ave is an average value.

[0114] Thus, we compared our approach with other approaches, such as separately trained online or offline transducers and cascade encoders. To make a fair comparison, we implemented the cascade encoder proposed in Reference 1 and used the same number of encoder layers as proposed in our architecture.

[0115] As shown in Figure 10, the present embodiment, which utilizes the three techniques of parallel structure, DB learning, and one-pass beam search, showed the best performance in both online and offline modes. Furthermore, the evaluation results of the present embodiment outperformed the cascade encoder in both modes. In particular, the conventional cascade encoder did not show any improvement in online mode. In contrast, the present embodiment achieved performance improvement in both modes. Furthermore, the one-pass beam search of the present embodiment, which tightly combines online and offline modes, brought about a larger performance improvement than the two-pass rescoring.

[0116] FIG. 11 is a diagram showing examples of texts speech-recognized by a conventional method and the method of this embodiment. The symbol g101 indicates the correct data. Reference symbol g102 is the output text when online processing is performed using the conventional method. Code g103 is the output text when offline processing is performed using the conventional method. Symbol g104 is the output text when online processing is performed using the trained model of this embodiment. Symbol g105 is the output text when offline processing is performed using the trained model of this embodiment. Reference numeral g106 denotes the output text resulting from the integration performed by the integration unit 15. Additionally, single underlining indicates incorrect answers, and double underlining indicates that text that was correct in the online process was incorrect in the offline process.

[0117] As shown by symbols g104 to g106 in FIG. 11, according to this embodiment, by integrating online processing and offline processing, it is possible to utilize the advantages of both and improve performance (correct answer).

[0118] FIG. 12 is a diagram showing an example of the effect of DB learning in online mode. The horizontal axis is block size, and the vertical axis is speech recognition rate (%). Line g201 is the result when online processing and offline processing are separated (hereinafter referred to as "separated model") and learning is performed with a fixed block size of 20. Line g202 is the result when online processing and offline processing are integrated as described above, and DB learning is not performed and learning is performed with a fixed block size of 20. Line g203 is the result when online processing and offline processing are integrated as described above and DB learning is performed. The values ​​are the average values ​​of three evaluation results.

[0119] 12, the model of this embodiment without DB learning has a slightly better CER than the separation model. And, the model of this embodiment with DB learning has a trade-off between block size and CER, and outperforms the baseline at all block sizes.

[0120] Fig. 13 is a diagram showing the speech recognition rate and ED of the model of this embodiment with DB learning and the separated model. As shown in Fig. 13, this embodiment significantly outperforms the separated model in all block sizes and maintains the same ED in almost all block sizes. That is, this embodiment does not increase the calculation time or the calculation cost.

[0121] FIG. 14 is a diagram showing the relationship between block size and CER in offline mode of CSJ. The horizontal axis is block size, and the vertical axis is speech recognition rate CER (%). The values ​​are average values ​​of three evaluation results. Line g211 is the evaluation result of the offline model of the separation model. Line g212 is the evaluation result when the block size is fixed at 20 and training is performed with processing by the integration unit 15 of this embodiment and without DB training. Line g213 is the evaluation result when the integration unit 15 of this embodiment is processed and with DB training. When the block size was 20, the model of this embodiment trained with a block size of 20 slightly outperformed the separated model. And when DB training was performed with this embodiment, the model achieved a CER equal to or higher than that of the separated model for all block sizes. In particular, when DB training was also performed with the model of this embodiment, the robustness of the offline mode improved, and when the block size was 10 or more, it exceeded the baseline.

[0122] As described above, in this embodiment, the following three operations are performed. I. Integrate the encoder output during training. Online encoders excel at local feature extraction, whereas offline encoders excel at global feature extraction. Therefore, in this embodiment, we were able to improve the performance of both models by integrating the encoder outputs and performing multi-task learning. II. Randomly change the block size and perform training. According to this embodiment, the robustness of the decoder can be improved by randomly allocating the block size of the online encoder during training. The block size can be freely changed by the user (the trade-off between accuracy and response can be adjusted). III. The decoder outputs are integrated. Online and offline models have their strengths and weaknesses. For this reason, in this embodiment, the decoding results are also integrated. Specifically, the hypotheses output by the offline model are rescored by the online model.

[0123] By using these techniques, the present embodiment improves the performance in both online and offline modes, i.e., the present embodiment improves the robustness of speech recognition using the E2E model. In addition, the DB learning of this embodiment improves the robustness of the offline mode while enabling flexible selection of block sizes in the online mode. Furthermore, in this embodiment, we further improve the performance of the offline mode by performing a one-pass beam search using online and offline transducer decoders.

[0124] Furthermore, in this embodiment, since the circuits are connected in parallel rather than in cascade, it is possible to avoid error accumulation that often occurs in the cascade structure of the conventional method.

[0125] In the above-described voice recognition device 1, the model may be provided via a cloud or a server. In the above example, the model is trained using a Japanese corpus and speech recognition is performed on Japanese speech, but the present invention is not limited to this. Even if the speech is a language other than Japanese, the present embodiment can be applied by training the model using a corpus of the language to be recognized.

[0126] Although the above-mentioned DB learning, encoder integration, decoder integration, and one-pass beam search are performed as an example, the effect can be obtained without performing all of these processes, as shown in the evaluation results. That is, the speech recognition device 1 may perform at least one of DB learning, encoder integration, decoder integration, and one-pass beam search.

[0127] In addition, a program for realizing all or part of the functions of the voice recognition device 1 of the present invention may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform all or part of the processes performed by the voice recognition device 1. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage providing environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" also refers to storage devices that hold a program for a certain period of time, such as volatile memory (RAM) inside a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0128] The above program may also be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be for realizing part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0129] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0130] Reference Signs List 1... speech recognition device, 2... acquisition unit, 3... extraction unit, 4... processing unit, 5... storage unit, 6... output unit, 11... first encoder, 12... first decoder, 13... second encoder, 14... second decoder, 15... integration unit, 16... learning processing unit

Claims

1. A speech recognition device that performs speech recognition using an end-to-end model, An encoder is a model that converts an input speech signal into a characteristic representation. a decoder that uses the output of the encoder to convert audio data into text; a learning unit that randomly selects a block length of the audio signal input to the encoder and learns the encoder and the decoder; A voice recognition device comprising:

2. The encoder includes a first encoder that performs online processing and a second encoder that performs offline processing; The decoder comprises a first decoder that performs online processing and a second decoder that performs offline processing; and an integration unit that outputs a final recognition result by rescoring the output of the first decoder and the output of the second decoder during inference.

2. The speech recognition device according to claim 1.

3. During training, the second decoder integrates an output of the first encoder and an output of the second encoder to learn the second decoder.

3. The speech recognition device according to claim 2.

4. the second decoder vertically stacks the output of the first encoder and the output of the second encoder; 4. The speech recognition device according to claim 3.

5. the integration unit performs inference on the speech data acquired using the second decoder to calculate a first score, calculates a second score on the same character string as the character string inferred by the second decoder using the first encoder, and re-scores the first score and the second score by calculating a weighted sum to determine a text to be inferred; 3. The speech recognition device according to claim 2.

6. A speech recognition method for a speech recognition device that performs speech recognition using an end-to-end model, comprising: The encoder converts the incoming audio signal into a characteristic representation. a decoder converts the audio data into text using the output of the encoder; a learning unit randomly selecting a block length of the audio signal input to the encoder and learning the encoder and the decoder; Speech recognition methods.

7. A computer of a speech recognition device that performs speech recognition using an end-to-end model having an encoder and a decoder, a first conversion step of converting an input speech signal into a characteristic representation; a second conversion step of converting the voice data into text using the output of the first conversion step; a step of randomly selecting a block length of the speech signal input in the first conversion step and training the encoder and the decoder; A program that executes the following.

Images