Speech recognition device, speech recognition method, and program

Abstract

A speech recognition device according to one aspect of the present disclosure comprises: a first generation unit that uses, as input, a feature amount indicating a feature of a chunk unit of speech to generate a first intermediate representation indicating an intermediate representation of the speech by an encoder; a second generation unit that uses, as input, a series of a processing unit of characters or character strings to generate a second intermediate representation indicating an intermediate representation of the processing unit by a series model; and a calculation unit that uses, as input, the first intermediate representation and the second intermediate representation to calculate the generation probability of the next processing unit by a model including linear transformation. The encoder is a model in which a relative multi-head attention mechanism and Mamba, which is a selective state space model, are disposed in parallel between a first feedforward network and a second feedforward network.

Description

Speech recognition device, speech recognition method, and program

[0001] The present disclosure relates to a speech recognition device, a speech recognition method, and a program.

[0002] An RNN-Transducer (Non-Patent Document 1) is known as a machine learning model used for speech recognition. Also, a Conformer (Non-Patent Document 2) is known as a machine learning model used for the encoder of an RNN-Transducer. An RNN-Transducer using a Conformer as an encoder can generally achieve higher speech recognition performance than a Transformer.

[0003] Alex Graves, "Sequence Transduction with Recurrent Neural Networks", ICML, 2012.Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu and Ruoming Pang, "Conformer: Convolution-augmented Transformer for Speech Recognition", Interspeech, 2020.

[0004] However, the speech recognition performance of the prior art is not sufficient, and further improvement in performance is required.

[0005] The present disclosure has been made in consideration of the above points, and aims to achieve high-performance speech recognition.

[0006] A speech recognition device according to one aspect of the present disclosure includes a first generation unit that receives, as input, feature quantities that represent characteristics of chunk-unit speech and generates a first intermediate representation that represents an intermediate representation of the speech using an encoder; a second generation unit that receives, as input, a sequence of processing units of characters or character strings and generates a second intermediate representation that represents an intermediate representation of the processing unit using a sequence model; and a calculation unit that receives, as input, the first intermediate representation and the second intermediate representation and calculates the generation probability of the next processing unit using a model including linear transformation, wherein the encoder is a model in which a relative multi-head attention mechanism and a selective state space model called Mamba are arranged in parallel between a first feedforward network and a second feedforward network.

[0007] High-performance speech recognition can be achieved.

[0008] FIG. 1 is a diagram showing an example of a conformer; FIG. 2 is a diagram showing an example of a proposed model; FIG. 3 is a diagram showing an example of the hardware configuration of a speech recognition device according to an embodiment; FIG. 4 is a diagram showing an example of the functional configuration of a speech recognition device according to an embodiment; and FIG. 5 is a flowchart showing an example of speech recognition processing according to an embodiment.

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0010] <Conformer> Conformer (Non-Patent Document 2) is known as a machine learning model used in the encoder of RNN-Transducer (Non-Patent Document 1). The conformer described in Non-Patent Document 2 will be described below with reference to Fig. 1. Fig. 1 is a diagram showing an example of a conformer.

[0011] As shown in Figure 1, the Conformer includes a first feed-forward network (FFN) layer, a multi-head attention (MHA) layer, a convolutional layer, and a second FFN layer. That is, the Conformer adds a convolutional layer to the previous Transformer, which is better at modeling local dependencies in the input.

[0012] The function or algorithm that performs the processing of the first FFN layer is called the FFN. １ (·), a function or algorithm that executes the processing of the MHA layer is called MHA(·), a function or algorithm that executes the processing of the convolution layer is called Conv(·), and a function or algorithm that executes the processing of the second FFN layer is called FFN(·). ２ (·). Also, the input to the first FFN layer (i.e., the input to the conformer) is x １ , the input to the MHA layer is x ２ , the input to the convolution layer is x ３ , the input to the second FFN layer is x ４ In this case, input x = x １ Given y, the Conformer sequentially computes the output y by:

[0013] x ２ = x １ +FFN １ (x １ ) x ３ = x ２ + MHA(x ２ ) x ４ = x ３ + Conv(x ３ ) y = x ４ +FFN ２ (x ４ In this way, in the Conformer, the input to the next layer is the sum of the output of each layer and its input (this is called "residual connection"). Generally, both the input x and the output y are represented by vectors.

[0014] <Proposed Model> There are many fields in which speech recognition accuracy is insufficient, and further improvement is necessary. In particular, conventional streaming speech recognition encoders cut off references to past information at a length of approximately 800 to 1,600 ms, which means that information of longer spans (e.g., speaker, noise type, channel characteristics, etc.) may not be fully utilized.

[0015] On the other hand, by using a state space model, it is possible in principle to store all information in past input speech, making it possible to perform speech recognition using more clues.

[0016] Therefore, in the following, we propose a model (hereinafter also referred to as the "proposed model") in which a model called Mamba (Reference 1), which is a type of Selective State Space Model (Selective SSM), is introduced into Conformer. Mamba dynamically changes model parameters according to the input, making it possible to forget unnecessary information in the input and selectively focus on only effective parts. For details about Mamba, please refer to Reference 1, etc.

[0017] An example of the proposed model will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the proposed model.

[0018] As shown in FIG. 2, the proposed model 1000 includes a first FFN layer 1100, an MHA layer 1200, a Mamba layer 1300, a combined linear transformation layer 1400, and a second FFN layer 1500.

[0019] The first FFN layer 1100 and the second FFN layer 1500 are both layers or modules realized by a feedforward network. The MHA layer 1200 is a layer or module realized by relative multi-head attention. The Mamba layer 1300 is a layer or module realized by Mamba. The combination and linear transformation layer 1400 is a layer or module realized by stacking (combining) vectors in the dimensional direction and linear transformation.

[0020] The function or algorithm that performs the processing of the first FFN layer 1100 is called an FFN. １ (·), a function or algorithm that executes the processing of the MHA layer is MHA(·), a function or algorithm that executes the processing of the Mamba layer 1300 is Mamba(·), a function or algorithm that executes the processing of the combined linear transformation layer 1400 is L(·), and a function or algorithm that executes the processing of the second FFN layer 1500 is FFN(·). ２ Let the input to the first FFN layer 1100 (i.e., the input to the proposed model) be x １, the input to the MHA layer 1200 and the Mamba layer 1300 is x ２ , the input to the second FFN layer 1500 is x ３ In this case, input x = x １ Given y, the proposed model 1000 sequentially computes the output y by:

[0021] x ２ = x １ +FFN １ (x １ ) x ３ = x ２ + L(MHA(x ２ ), Mamba (x ２ )) where L(MHA(x ２ ), Mamba (x ２ ))=W ｍｅｒｇｅ concat(MHA(x ２ ), Mamba (x ２ )) y = x ３ +FFN ２ (x ３ ) where W ｍｅｒｇｅ is a weight matrix of the linear transformation, and cancat is a combination operation of two vectors (sequences) in the vector dimension direction. ｍｅｒｇｅ is concat(MHA(x ２ ), Mamba (x ２ )) to the dimension of the original vector (i.e., the input x ２ is the weight matrix for the linear transformation to transform

[0022] In this way, the proposed model 1000 arranges the MHA layer 1200 and the Mamba layer 1300 in parallel, and merges their outputs using a linear transformation. This effectively combines detailed information within each chunk of speech with long-range memory across chunks, enabling improved recognition accuracy.

[0023] The following describes a speech recognition device 10 that uses the proposed model 1000 as an encoder of an RNN-Transducer to realize speech recognition for speech represented by given speech data.

[0024] <Example of Hardware Configuration of Speech Recognition Device 10> An example of the hardware configuration of the speech recognition device 10 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the hardware configuration of the speech recognition device 10 according to this embodiment.

[0025] 3, the speech recognition device 10 according to this embodiment includes an input device 101, a display device 102, an external I / F 103, a communication I / F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other via a bus 109 so as to be able to communicate with each other.

[0026] The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the voice recognition device 10 does not necessarily have to include at least one of the input device 101 and the display device 102, for example.

[0027] The external I / F 103 is an interface with an external device such as a recording medium 103a. Examples of the recording medium 103a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0028] The communication I / F 104 is an interface for connecting to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a non-volatile storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The processor 108 is one of various arithmetic devices such as a central processing unit (CPU) or a graphics processing unit (GPU).

[0029] 1 is an example and is not intended to be limiting. The speech recognition device 10 may have multiple auxiliary storage devices 107 or multiple processors 108, may not have some of the hardware shown in the figure, or may have various hardware other than the hardware shown in the figure.

[0030] <Example of Functional Configuration of Speech Recognition Device 10> An example of the functional configuration of the speech recognition device 10 according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the functional configuration of the speech recognition device 10 according to this embodiment.

[0031] 4, the speech recognition device 10 according to this embodiment includes an input unit 201, a feature extraction unit 202, a first expression sequence generation unit 203, a second expression sequence generation unit 204, a posterior probability calculation unit 205, and an output unit 206. Each of these units is realized by, for example, processing executed by the processor 108 or the like of one or more programs installed in the speech recognition device 10.

[0032] The input unit 201 receives given audio data. The audio data may be, for example, audio data representing streaming audio.

[0033] The feature extraction unit 202 receives the speech data input by the input unit 201 and extracts an acoustic feature sequence from the speech waveform of the speech data. The acoustic feature sequence is expressed as a sequence of vectors (feature vectors) that represent the acoustic feature sequence for each chunk in the speech waveform. A chunk is a section of a certain time width in the speech waveform. The feature extraction unit 202 may extract the acoustic feature sequence from the speech waveform using existing tools or libraries (e.g., TorchAudio library, etc.). Hereinafter, the acoustic feature sequence extracted by the feature extraction unit 202 will be referred to as {o １ , ..., o ｔ} where o ｎ is the acoustic feature of the n-th chunk.

[0034] The first expression sequence generation unit 203 generates the acoustic feature sequence {o １ , ..., o ｔ} is input, and a speech expression sequence (hereinafter also referred to as a "first expression sequence") is output by the proposed model 1000. That is, the first expression sequence generation unit 203 outputs the acoustic feature o ｔ is input to the proposed model 1000, and the first intermediate representation v is output. ｔ This generates the first representation sequence {v １ , ..., v ｔ} is obtained. The first expression sequence generation unit 203 is a functional unit equivalent to the encoder of an RNN-Transducer.

[0035] The second expression sequence generation unit 204 generates a token sequence {e １ , ..., e ｉ－１} and outputs a text expression sequence (hereinafter also referred to as a "second expression sequence"). ｉ－１ is input to the existing sequence model, and the second intermediate representation w ｉ－１ This generates the second representation sequence {w １ , ..., w ｉ－１} is obtained. A token is a processing unit of a character or a string, and typical examples include a word, a phrase, and a subword. The second expression sequence generation unit 204 is a functional unit equivalent to the Prediction Network (sometimes called a "Predictor") of an RNN-Transducer. The second expression sequence generation unit 204 is realized by an existing sequence model (e.g., LSTM (Reference 2)).

[0036] The posterior probability calculation unit 205 calculates the first expression sequence {v １ , ..., v ｔ} and the second expression sequence {w １ , ..., w ｉ－１} is input, and the generation probability (posterior probability) of the i-th token is calculated. The posterior probability calculation unit 205 is a functional unit equivalent to the Joint Network of the RNN-Transducer. The Joint Network is realized by, for example, a linear transformation layer and a Softmax function.

[0037] The output unit 206 generates the i-th token in accordance with the posterior probability calculated by the posterior probability calculation unit 205. As a result, the token sequence {e １ , ..., e ｉ} is obtained.

[0038] <Speech Recognition Processing> An example of speech recognition processing according to this embodiment will be described with reference to FIG. 5. FIG. 5 is a flowchart showing an example of speech recognition processing according to this embodiment. In streaming speech recognition, the following steps S101 to S107 are executed each time speech data representing streaming speech in a certain predetermined time interval is provided. For simplicity, it is assumed below that speech data representing streaming speech is provided in chunk units. Note that, although the following speech recognition processing will be described assuming streaming speech recognition, the following speech recognition processing can also be similarly applied to speech recognition other than streaming speech recognition by making appropriate necessary modifications.

[0039] The input unit 201 receives given audio data (step S101). Hereinafter, the chunk number of the streaming audio represented by this audio data is assumed to be t.

[0040] The feature extraction unit 202 receives the speech data input in step S101 and extracts an acoustic feature o from the speech waveform of the speech data. ｔ (step S102). １ , ..., o ｔ} is obtained.

[0041] The first expression sequence generation unit 203 generates the acoustic feature o obtained in step S102. ｔ The first intermediate representation v is generated by the proposed model 1000. ｔ (Step S103). Here, in the proposed model 1000, the acoustic feature o ｔ Enter x １ The following are calculated sequentially to obtain the first intermediate representation v as the output y: ｔ is generated.

[0042] x ２ = x １ +FFN １ (x １ ) x ３ = x ２ + L(MHA(x ２ ), Mamba (x ２ )) where L(MHA(x ２ ), Mamba (x ２ ))=W ｍｅｒｇｅ concat(MHA(x ２ ), Mamba (x ２ )) y = x ３ +FFN ２ (x ３ ) This gives the first representation sequence {v １ , ..., v ｔ} is obtained.

[0043] For simplicity, let i = t and let the token sequence consisting of tokens up to t-1 be {e １ , ..., e ｉ－１} is obtained. Also, a second representation sequence {w １ , ..., w ｔ－２} is assumed to be obtained.

[0044] The second expression sequence generation unit 204 generates the t-1-th token e ｉ－１ is used as input, and the second intermediate representation w is generated using the existing sequence model. ｔ－１ (step S104). As a result, the second expression sequence {w １ , ..., w ｔ－１} is obtained.

[0045] The posterior probability calculation unit 205 calculates the first expression sequence {v １ , ..., v ｔ} and the second expression sequence {w １ , ..., w ｔ－１} is input, and the posterior probability of the t-th token is calculated (step S105).

[0046] The output unit 206 outputs the t-th token e according to the posterior probability calculated in step S105. ｔ is generated (sampled) (step S106).

[0047] The output unit 206 outputs the token e generated in step S106. ｔ to a predetermined output destination (step S107). As a result, the token sequence {e １ , ..., e ｉ} is output to the output destination. The output destination is not limited to a specific output destination, but may be, for example, the display device 102 such as a display, a storage area such as the auxiliary storage device 107, another program, another device, etc.

[0048] <Others> In the above embodiment, it has been described that each model (including the proposed model 1000) constituting the RNN-Transducer has been trained, but all or some of these models, including the proposed model 1000, may be trained by a known training method using training data. Examples of training data include data represented as a pair of speech data and text data representing the correct speech recognition result for the speech data. Also, as a known training method, for example, a cross-entropy error with respect to the posterior probability of a token may be used as an error function, and the parameters of the model to be trained may be updated so as to minimize the error function using a stochastic gradient descent method or the like.

[0049] The speech recognition device 10 according to this embodiment may have a learning unit as a functional unit that executes the above-described learning method. The speech recognition device 10 having such a learning unit may be called, for example, a "model learning device" or a "learning device."

[0050] <Summary> As described above, the speech recognition device 10 according to this embodiment performs speech recognition using the proposed model 1000, which is a type of selective state space model that introduces Mamba into Conformer, as an encoder. This makes it possible for the speech recognition device 10 according to this embodiment to realize speech recognition based on long-distance dependencies across chunks that could not be adequately captured by conventional encoders for streaming speech recognition.

[0051] While there are conventional techniques using a time-invariant state space model (e.g., Reference 3, etc.), the proposed model 1000 uses Mamba, a type of selective state space model that operates more similarly to an attention mechanism. Furthermore, the proposed model 1000 does not simply replace the convolutional layer with the Mamba layer 1300, but instead arranges the MHA layer 1200 and the Mamba layer 1300 in parallel. This makes it possible to more efficiently utilize the characteristics of each layer, namely, the strength of the MHA layer 1200, which can flexibly model information within a chunk, and the strength of the Mamba layer 1300, which can effectively model long-distance dependencies between chunks.

[0052] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0053] [References] Reference 1: Albert Gu and Tri Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces", 2023, arXiv:2312.00752 Reference 2: Sepp Hochreiter and Jurgen Schmidhuber, "LONG SHORT-TERM MEMORY", Neural Computation, 1997. Reference 3: George Saon, Ankit Gupta and Xiaodong Cui, "DIAGONAL STATE SPACE AUGMENTED TRANSFORMERS FOR SPEECH RECOGNITION", ICASSP, 2023

[0054] REFERENCE SIGNS LIST 10 Speech recognition device 101 Input device 102 Display device 103 External I / F 103a Recording medium 104 Communication I / F 105 RAM 106 ROM 107 Auxiliary storage device 108 Processor 109 Bus 201 Input unit 202 Feature extraction unit 203 First expression sequence generation unit 204 Second expression sequence generation unit 205 Posterior probability calculation unit 206 Output unit

Claims

1. A speech recognition device comprising: a first generation unit that takes as input feature quantities that represent the characteristics of a chunk of speech and generates a first intermediate representation that represents an intermediate representation of the speech using an encoder; a second generation unit that takes as input a series of processing units of characters or strings and generates a second intermediate representation that represents an intermediate representation of the processing units using a series model; and a calculation unit that takes as input the first intermediate representation and the second intermediate representation and calculates the generation probability of the next processing unit using a model including linear transformation, wherein the encoder is a model in which a relative multi-head attention mechanism and a selective state space model called Mamba are arranged in parallel between a first feedforward network and a second feedforward network.

2. The speech recognition device of claim 1, wherein the encoder is a model that generates the first intermediate representation as the output of the second feedforward network by inputting the sum of the output of the first feedforward network, which has the features as input, and the residual connection of the first feedforward network to the relative multi-head attention mechanism and the Mamba, respectively, and inputting the sum of the result of combining and linearly transforming the outputs of the relative multi-head attention mechanism and the Mamba and the residual connection of the relative multi-head attention mechanism and the Mamba to the second feedforward network.

3. A speech recognition method in which a computer executes the following steps: a first generation procedure in which, using an encoder, an encoder takes as input features that represent the characteristics of a chunk of speech; a second generation procedure in which, using a sequence of processing units of characters or strings as input, an encoder generates a second intermediate representation that represents an intermediate representation of the processing units using a sequence model; and a calculation procedure in which, using the first intermediate representation and the second intermediate representation as input, an encoder calculates the probability of generating the next processing unit using a model including a linear transformation; and the encoder is a model in which a relative multi-head attention mechanism and a selective state space model, Mamba, are arranged in parallel between a first feedforward network and a second feedforward network.

4. A program that causes a computer to execute the following steps: a first generation procedure that takes as input features that represent the characteristics of a chunk of audio and generates a first intermediate representation that represents an intermediate representation of the audio using an encoder; a second generation procedure that takes as input a series of processing units of characters or strings and generates a second intermediate representation that represents an intermediate representation of the processing units using a series model; and a calculation procedure that takes as input the first intermediate representation and the second intermediate representation and calculates the generation probability of the next processing unit using a model including linear transformation, wherein the encoder is a model in which a relative multi-head attention mechanism and a selective state space model called Mamba are arranged in parallel between a first feedforward network and a second feedforward network.

Images