Speech recognition device, speech recognition method, and speech recognition program

Abstract

To enable ASR streaming processing to be efficiently performed.SOLUTION: A speech recognition device 10 comprises an acquisition unit 151, an encoding unit 152, and a decoding unit 153. The acquisition unit 151 acquires a speech signal for each chunk. The encoding unit 152 calculates intermediate feature quantity based on the speech signal of a first chunk acquired by the acquisition unit 151 and the speech signals of a certain number of chunks acquired in the past from the first chunk. The decoding unit 153 calculates a sequence of tokens for identifying text based on the intermediate feature quantity calculated by the encoding unit 152.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a voice recognition device, a voice recognition method, and a voice recognition program. [Background technology]

[0002] Conventionally, voice AI using ASR (Automatic Speech Recognition), a technology that converts voice to text, is known. Voice AI requires streaming processing for purposes such as real-time speech-to-text transcription, video subtitling, real-time speech translation, voice assistants operated by voice, and device operation by voice commands while driving.

[0003] The RNN-Transducer model is known as a model for performing ASR using streaming processing. However, the RNN-Transducer model has problems such as not working well for long sequences (sequences of speech signals), the recursive nature of the RNN state, and the inability to operate in parallel, making training difficult, and gradient vanishing and explosion occurring during training.

[0004] In response to this, a causal approach using the Convolution Augmented Transformer (Conformer) model, which is superior for non-streaming ASR tasks (see, for example, Non-Patent Document 1), and a chunk-based approach (see, for example, Non-Patent Document 2) have been proposed. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] TNSainath, et al., "Improving the latency and quality of cascaded encoders," in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp.8112-8116

[0006] [Non-Patent Document 2] S. Zhang, Z. Gao, H. Luo, M. Lei, J. Gao, Z. Yan, and L. Xie, "Streaming chunk-aware multihad attention for online end-to-end speech recognition,", Interspeech, 2020, pp.2142-2146 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the conventional techniques have a problem in that the ASR streaming process may not be executed efficiently.

[0008] For example, the causal approach described in Non-Patent Document 1 calculates the attention score of the current element in a sequence only on past frames and masks future frames. However, the causal approach suffers from the problem that the computational cost increases quadratically as the length of the speech increases, because new and past frames are processed repeatedly.

[0009] Also, for example, the chunk-based approach described in Non-Patent Document 2 processes audio frames in the form of audio chunks when a new speech frame arrives. However, the approach described in Non-Patent Document 2 does not take context into account, which may result in poor performance (e.g., recognition accuracy). [Means for solving the problem]

[0010] In order to solve the above-mentioned problems and achieve the object, the speech recognition device is characterized by having an acquisition unit that acquires a speech signal for each chunk, an encoding unit that calculates intermediate features based on the speech signal of a first chunk acquired by the acquisition unit and the speech signals of a certain number of chunks acquired prior to the first chunk, and a decoding unit that calculates a string of tokens for identifying text based on the intermediate features calculated by the encoding unit. Effect of the Invention

[0011] According to the present invention, ASR streaming processing can be efficiently executed. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a voice recognition device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram for explaining a model according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating the attention mask according to the first embodiment. [Figure 4] FIG. 4 is a flowchart showing a flow of processing of the voice recognition device according to the first embodiment. [Diagram 5] FIG. 5 shows the results of the experiment. [Figure 6] FIG. 6 shows the results of the experiment. [Figure 7] FIG. 7 shows the results of the experiment. [Figure 8] FIG. 8 is a diagram illustrating the prior art. [Figure 9] FIG. 9 is a diagram illustrating the prior art. [Figure 10] FIG. 10 is a diagram illustrating an example of a computer that executes a voice recognition program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A voice recognition device, a voice recognition method, and a voice recognition program according to the present application will be described in detail below with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments described below.

[0014] [Configuration of the first embodiment] First, the configuration of a voice recognition device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a voice recognition device according to the first embodiment. The voice recognition device 10 receives an input of a voice signal and outputs text. The voice recognition device 10 also performs streaming processing. That is, the voice recognition device 10 outputs text in real time while receiving an input of a voice signal.

[0015] As shown in FIG. 1, the voice recognition device 10 includes a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, and a control unit 15.

[0016] The communication unit 11 performs data communication with other devices via a network. For example, the communication unit 11 is a network interface card (NIC).

[0017] The input unit 12 receives input of data and is an interface connected to input devices such as a mouse, a keyboard, and a microphone.

[0018] The output unit 13 outputs data. The output unit 13 is an interface connected to an output device such as a display and a speaker.

[0019] The storage unit 14 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc. The storage unit 14 may be a semiconductor memory in which data can be rewritten, such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM). The storage unit 14 stores an operating system (OS) and various programs executed by the voice recognition device 10.

[0020] The voice recognition device 10 stores model information 141 and retention information 142 .

[0021] The model information 141 is information such as parameters for constructing a model, etc. For example, the model information 141 is weights and biases of a neural network.

[0022] The retained information 142 is information obtained in an already executed process. The information obtained in an already executed process is retained as the retained information 142 and may be used in a subsequent process.

[0023] The control unit 15 controls the entire voice recognition device 10. The control unit 15 is, for example, an electronic circuit such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 15 also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory.

[0024] In addition, the control unit 15 functions as various processing units by the operation of various programs. For example, the control unit 15 functions as an acquisition unit 151, an encoding unit 152, a decoding unit 153, and a storage unit 154.

[0025] The acquisition unit 151, the encoding unit 152, the decoding unit 153, and the storage unit 154 execute the ASR using the model. The model is a so-called machine learning model, and is constructed based on the model information 141.

[0026] Before describing the model of this embodiment, the Conformer model described in Non-Patent Document 1 will be described. The Conformer model is a type of self-attention based Transformer model. The Transformer model is known to be superior to a Recurrent Neural Network (RNN) in non-streaming ASR.

[0027] The Conformer model is a so-called encoder-decoder based model (see Reference 1). In addition, the Conformer model can be combined with a decoder such as Connectionist Temporal Classification (CTC) (Reference 2) or Traceducer (Reference 3) to become an encoder that can be used for end-to-end ASR tasks. Reference 1: A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in INTERSPEECH, 10 2020, pp. 5036-5040. Reference 2: A. Graves, Connectionist Temporal Classification. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012. Reference 3: F. Boyer, Y. Shinohara, T. Ishii, H. Inaguma and S. Watanabe, "A Study of Transducer Based End-to-End ASR with ESPnet: Architecture, Auxiliary Loss and Decoding Strategies," 2021, ASRU, Cartagena, Colombia, 2021, pp. 16-23.

[0028] The Conformer model has a multi-headed self-attention layer that computes the relationship between each element in a sequence (a sequence of audio signals), a Query (Q), and all other elements in the sequence, a Key (K).

[0029] The computational cost of the self-attention layer is proportional to the dimension D of the conformer model and is a quadratic function of the length X of the speech signal. The computational cost of the self-attention layer is expressed as in equation (1).

[0030]

number

[0031] For this reason, the causal method using the conformer model described in Non-Patent Document 1 has a problem in terms of computational cost. Also, as described above, the chunk-based approach in Non-Patent Document 2 has a problem in that it does not take context into account.

[0032] In this embodiment, the problems of the conventional technology are solved by performing ASR using a method that takes into account the context while suppressing an increase in calculation costs.

[0033] The model of this embodiment will be described with reference to Fig. 2, and the processes of the acquisition unit 151, the encoding unit 152, the decoding unit 153, and the storage unit 154 will be described. Fig. 2 is a diagram for explaining the model according to the first embodiment.

[0034] As shown in FIG. 2, the Conformer model functions as an encoder. The acquisition unit 151 acquires a voice signal for each chunk and inputs it to the encoder. For example, the voice signal of the Nth chunk is N 1 …x N t where t is the number of frames included in one chunk. In other words, a chunk is a certain number (e.g., t) of consecutive frames when an audio signal is divided into frames in a time series. Note that an audio signal is, for example, a frequency over time.

[0035] The encoding unit 152 encodes the audio signal of the current chunk and the audio signal of the context chunk acquired by the acquisition unit 151, and outputs intermediate features. Here, the current chunk is the latest chunk acquired. On the other hand, the context chunk is a combination of the current chunk and a certain number (n-1) of chunks acquired in the past. If the number of context chunks is n, the number of current chunks is 1, and the number of the certain number of chunks acquired in the past is n-1. The chunks acquired in the past may be stored as the stored information 142. Also, the value of n (an integer equal to or greater than 1) may be determined in advance by the user.

[0036] The encoding unit 152 calculates an attention score from the context chunk. That is, the encoder functions as a self-attention layer. At this time, the encoding unit 152 uses the information of the context chunk as an attention mask.

[0037] When calculating the attention score, the encoding unit 152 calculates the relationship between each of the speech signals Q (Query) to be calculated and K (Key). The combination of Q and K to be calculated in the conformer model is represented as an attention mask. Fig. 3 is a diagram for explaining the attention mask according to the first embodiment.

[0038] Here, n=3. Also, each of the 3×3 squares in FIG. 3 is an audio signal corresponding to a chunk. When calculating the attention score for a current chunk, the encoding unit 152 refers to a context chunk that includes the current chunk and two (n-1) chunks (two 3×3 chunks) in the past (opposite direction to the arrow of K). Even if Q advances (moves in the direction of the arrow) according to streaming processing, the number of current chunks is a maximum of 3 (three 3×3 chunks).

[0039] In this way, the encoding unit 152 can perform encoding while minimizing the number of chunks to be referenced and taking the context into consideration.

[0040] If the encoder is a function “encoder”, the encoding unit 152 performs the calculation shown in formula (2). N is the intermediate feature of the N-th chunk. The encoding unit 152 encodes the s-dimensional intermediate feature h N Calculate.

[0041]

number

[0042] Pre-processing in Fig. 2 is a process of converting a voice signal, which is, for example, a one-dimensional frequency, into MFCC (Mel Frequency Cepstrum Coefficients). Subsampling is a process of reducing the size of the data. Subsampling is followed by a self-attention layer. The self-attention layer takes the context chunk as input and calculates the attention score. Note that the encoder configuration shown in Fig. 2 is just one example.

[0043] The decoding unit 153 decodes the intermediate features calculated by the encoding unit 152 and past calculation results to calculate a label string. The elements of the label string are called tokens. The past calculation result is the last token of the label string calculated for the previous chunk.

[0044] A token is information that identifies, for example, a subword, and thus words and texts can be reproduced by arranging tokens.

[0045] If the decoder is a function “decoder”, the decoding unit 153 performs the calculation shown in formula (3). N is the number of tokens in the N-th chunk. Also, y N k is the k-th token of the N-th chunk.

[0046]

number

[0047] The decoder may be a Transducer (Reference 4). The Transducer includes two RNN layers and one fully connected layer. Reference 4: F. Boyer, Y. Shinohara, T. Ishii, H. Inaguma, and S. Watanabe, “A study of transducer based end-to-end ASR with espnet: Architecture, auxiliary loss and decoding strategies,” CoRR, vol. abs / 2201.05420, 2022. [Online]. Available: https: / / arxiv.org / abs / 2201.05420

[0048] The retaining unit 154 retains the last token calculated by the decoding unit 153 in the retained information 142. The last token retained here is used in the calculation of the next chunk.

[0049] [Processing of the first embodiment] 4 is a flowchart showing a flow of processing of the voice recognition device according to the first embodiment. As shown in Fig. 4, first, the voice recognition device 10 sets N to 1 (step S101). Next, the speech recognition device 10 acquires the speech signal of the Nth chunk (step S102).

[0050] Next, the speech recognition device 10 encodes the speech signal from the Nn-th chunk to the N-th chunk, and calculates the N-th intermediate feature (step S103). The speech recognition device 10 performs encoding using, for example, a conformer model. At this time, the speech recognition device 10 calculates an attention score based on the speech signal from the Nn-th chunk to the N-th chunk.

[0051] Then, the speech recognition device 10 decodes the N-th intermediate feature and the last element (token) of the (N-1)-th label string (step S104). It is assumed that the (N-1)-th label string is stored in the storage unit 14.

[0052] Furthermore, the speech recognition device 10 holds the N-th label string obtained by decoding (step S105), and outputs the N-th label string (step S106). Note that the speech recognition device 10 may convert tokens included in the N-th label string into words and output them.

[0053] If the end condition is not satisfied (No at step S107), the voice recognition device 10 increments N by 1 (step S108) and returns to step S102 to repeat the process.

[0054] On the other hand, if the end condition is satisfied (Yes in step S107), the voice recognition device 10 ends the process. The end condition is, for example, that the voice signal is no longer input.

[0055] [Advantages of the First Embodiment] As described above, the speech recognition device 10 includes an acquisition unit 151, an encoding unit 152, and a decoding unit 153. The acquisition unit 151 acquires a speech signal for each chunk. The encoding unit 152 calculates intermediate features based on the speech signal of a first chunk acquired by the acquisition unit 151 and speech signals of a certain number of chunks acquired prior to the first chunk. The decoding unit 153 calculates a string of tokens for identifying text based on the intermediate features calculated by the encoding unit 152.

[0056] Here, the attention mask in the causal method described in Non-Patent Document 1 is expressed as shown in Figure 8. Figure 8 shows that when calculating the attention score, all past audio signals (in the opposite direction to the arrow of K) are referenced. For this reason, the amount of calculation becomes enormous as Q progresses.

[0057] Moreover, the attention mask in the chunk-based approach described in Non-Patent Document 2 is expressed as shown in Fig. 9. Fig. 9 shows that when calculating the attention score of a certain chunk, past chunks are not referenced. Therefore, the context is not taken into account.

[0058] In contrast, as shown in Fig. 3, in this embodiment, a certain number of past chunks are referenced when calculating the attention score, so that it is possible to suppress an increase in the amount of calculation while taking into account the context. As a result, the speech recognition device 10 can efficiently execute the ASR streaming process.

[0059] Conventionally, ASR streaming processing requires a large amount of calculations, and therefore may be implemented on the cloud. On the other hand, by using this embodiment, the amount of calculations can be reduced, and ASR using edge computing can be realized.

[0060] For example, performing ASR on user devices can reduce the rapidly growing demand for network bandwidth, while edge computing provides benefits such as reduced latency, improved privacy and security, increased reliability through local processing, and superior scalability compared to cloud-based infrastructure.

[0061] In addition, by integrating the Transducer decoder with the Conformer model (especially the Conformer-Small encoder) as shown in Figure 2, the Transducer decodes only the latest chunk, which reduces the overall decoding time.

[0062] In this embodiment, when calculating the attention score, past contexts are referenced at the chunk level, whereas it is possible to reference a fixed past context for each frame of audio. However, in this case, the size of the context to be stored becomes large, which causes a problem of increased storage capacity.

[0063] [experiment] An experiment conducted to verify the effect of the embodiment will be described. Figures 5, 6, and 7 are diagrams showing the results of the experiment. In the experiment, the causal method (Casual) described in Non-Patent Document 1, the chunk-based approach (Chunk) described in Non-Patent Document 2, and the embodiment (Chunked-Context) were compared.

[0064] Figure 5 shows the computational cost of each method in terms of FLOPS (Floating Point Operations). In this embodiment (Chunked of n chunks), the upper limit of the computational cost is limited to a fixed value in equation (4), where C is the number of frames in one chunk, and D is the dimension of the Conformer model.

[0065]

number

[0066] As shown in FIG. 5, the longer the audio signal, the more effective this embodiment is.

[0067] Fig. 6 shows the RTF (Real-Time Factor). The RTF becomes smaller as the processing time of the encoder and decoder relative to the length of one chunk of audio signal is shorter. Fig. 6 shows that the processing time of this embodiment (Chunked-Context) is shorter than that of the causal method (Casual).

[0068] Fig. 7 shows the WER (Word Error Rate) of each method. As shown in Fig. 7, in this embodiment (Limited), sufficient accuracy is obtained while suppressing the calculation cost.

[0069] [System configuration, etc.] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed or integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware by wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

[0070] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0071] [program] As an embodiment, the voice recognition device 10 can be implemented by installing a voice recognition program that executes the above voice recognition processing as package software or online software on a desired computer. For example, the above voice recognition program can be executed by an information processing device, thereby making the information processing device function as the voice recognition device 10. The information processing device here includes desktop and notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), and further slate terminals such as PDA (Personal Digital Assistant), etc.

[0072] The voice recognition device 10 may also be implemented as a voice recognition server device that provides a service related to the learning process to a client, the client being a terminal device used by a user. For example, the voice recognition server device is implemented as a server device that provides a voice recognition service that receives a voice signal as input and outputs a text. In this case, the voice recognition server device may be implemented as a Web server, or may be implemented as a cloud that provides a service related to the voice recognition process by outsourcing.

[0073] 10 is a diagram showing an example of a computer that executes a voice recognition program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0074] The memory 1010 includes a read only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0075] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the voice recognition device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing the same process as the functional configuration of the voice recognition device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0076] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0077] Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0078] 10 Voice recognition device 11 Communications Department 12 Input section 13 Output section 14 Storage section 15 Control section 141 Model Information 142 Retention information 151 Acquisition Department 152 Encoding section 153 Decoding section 154 Holding part

Claims

1. an acquisition unit that acquires an audio signal chunk by chunk; an encoding unit that calculates intermediate features based on the audio signal of a first chunk acquired by the acquisition unit and audio signals of a certain number of chunks acquired prior to the first chunk; a decoding unit that calculates a string of tokens for identifying text based on the intermediate features calculated by the encoding unit; A speech recognition device comprising:

2. 2. The speech recognition device according to claim 1, wherein the encoding unit performs encoding using a conformer model.

3. 2. The speech recognition device according to claim 1, wherein the decoding unit performs the decoding using a transducer.

4. A speech recognition method executed by a speech recognition device, comprising: An acquisition step of acquiring an audio signal chunk by chunk; an encoding step of calculating intermediate features based on the audio signal of a first chunk acquired by the acquisition step and audio signals of a certain number of chunks acquired prior to the first chunk; a decoding step of calculating a string of tokens for identifying text based on the intermediate features calculated by the encoding step; 13. A speech recognition method comprising:

5. An acquisition step of acquiring an audio signal chunk by chunk; an encoding step of calculating intermediate features based on the audio signal of a first chunk acquired by the acquisition step and audio signals of a certain number of chunks acquired prior to the first chunk; a decoding step of calculating a string of tokens for identifying text based on the intermediate features calculated by the encoding step; A speech recognition program that causes a computer to execute the above steps.

Images