Long context end-to-end speech recognition system

Abstract

The present invention relates generally to speech processing, and more particularly to end-to-end automatic speech recognition (ASR) with long context information. Some embodiments of the present invention provide systems and methods for end-to-end ASR suitable for recognizing long audio recordings, such as lecture and conversational speech. The present disclosure includes a transformer-based ASR system that utilizes context information, where the transformer takes in multiple sentences at once and predicts the transcription of the last sentence. This is repeated with a sliding window shifting by one sentence at a time to recognize the entire recording. Furthermore, when a long audio recording includes multiple speakers, some embodiments of the present invention can use acoustic and / or textual features obtained only from previous utterances spoken by the same speaker as the last sentence.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to speech processing, and in particular, to end-to-end automatic speech recognition (ASR) with long context information. BACKGROUND

[0002] Recent advances in deep learning techniques have opened up a new paradigm for automatic speech recognition (ASR), namely, so-called end-to-end ASR, which includes training and using a single deep network that directly converts speech signals or spectral features into target text. Unlike typical hybrid ASR systems, end-to-end systems rely only on paired acoustic and linguistic data (i.e., training data) and not on linguistic knowledge, and the entire system is trained using a single algorithm. Thus, this approach makes it feasible to build ASR systems without expert knowledge. Several end-to-end models have been proposed and applied to ASR, such as connectionist temporal classification (CTC), attention-based encoder-decoder (AED), recurrent neural network (RNN) transducer (RNN-T), transformer, and combinations thereof. In particular, among the most prominent sequence-to-sequence tasks including machine translation and ASR, transformers have recently provided a significant performance gain over RNN-based end-to-end models (e.g., AED and RNN-T). As a result, transformer-based approaches are being increasingly investigated to further improve ASR accuracy.

[0003] On the other hand, ASR systems suffer from speaker and topic diversity. For those speakers who have different characteristics than those commonly observed in the training data, the recognition accuracy is usually not high. For utterances on topics that are not common in the training data, the accuracy is also not high. This is because the training of ASR systems is to improve the accuracy of the training data, so they are not good at recognizing utterances of features that are not common or unseen in the training data.

[0004] To overcome this problem, previous approaches extract speaker vectors or topic vectors from previous utterances spoken by the same speaker or utterances with the same topic as the current utterance as context information and feed them into the neural network for ASR along with the speech signal of the current utterance. With the speaker vector or topic vector, the network is adapted to the current speaker or topic, and the recognition accuracy is improved. However, this previous approach summarizes the information of previous utterances into a single fixed-dimensional vector, so the vector can not retain enough information of the previous utterances. In general, a fixed-dimensional vector is not sufficient to represent the sequential patterns of acoustic or linguistic features of multiple utterances. This can hinder the network from better adapting to the speaker / topic.

[0005] As mentioned above, there are some previous studies on incorporating contextual information in end-to-end ASR. For example, speaker vectors are fed into the encoder network of the AED along with the speech features, and topic vectors are fed into the decoder network of the AED. However, there is no effective method for transformer-based ASR systems. Therefore, ASR systems cannot utilize contextual information to recognize long speech. Moreover, unlike previous methods, it is desirable to be able to utilize contextual information without summarizing it into a fixed-dimension vector. SUMMARY

[0006] It is an object of some embodiments of the present invention to provide a system and method of end-to-end ASR suitable for recognizing long audio recordings, such as lecture and conversational speech.

[0007] The present disclosure includes a long-context end-to-end ASR system that utilizes contextual information, where a context-expanded transformer takes in multiple sentences at once and predicts the text transcription of the last sentence. This is repeated in a sliding window fashion shifting by one sentence at a time to recognize long audio recordings. During training and decoding, the context-expanded transformer utilizes contextual information to improve recognition accuracy for long audio recordings, such as lecture and conversational speech.

[0008] The present invention includes the steps of making a speech segment consisting of connected multiple sentences and training a context-expanded transformer to minimize a loss function for the last sentence of the speech segment, where the previous utterances are used to adapt the hidden vectors representing acoustic and linguistic features at each encoder / decoder layer of the transformer to recognize the last sentence of the speech segment. Unlike previous methods that summarize the information of the previous utterances into a single fixed-dimension vector, the present invention directly uses the previous utterances to adapt the hidden vectors through the self-attention mechanism of the transformer without losing information. The present invention provides better adapted hidden vectors, thus improving speech recognition accuracy.

[0009] The present invention also includes the steps of decoding a long audio, where (1) updating the speech segment by appending the next sentence to the speech segment based on a sliding window technique, (2) decoding the last sentence of the updated speech segment using the context-expanded transformer, and (3) stopping the decoding when there is no next sentence in the long audio.

[0010] Furthermore, when the long audio recording includes multiple speakers, some embodiments of the present invention can use acoustic and / or textual features obtained only from previous utterances spoken by the same speaker as the last sentence.

[0011] Further, some embodiments of the present disclosure provide an end-to-end automatic speech recognition (ASR) system. The ASR system can include an interface configured to obtain a sequence of acoustic features comprising an utterance; a memory configured to store a computer executable ASR program module, a beam search decoder, and a speech segment updating module, the ASR program module comprising a context expansion transformer network, the context expansion transformer network comprising an encoder network and a decoder network; and a processor connected with the memory and configured to repeatedly decode the utterance. The processor is configured to perform the following steps: arrange a speech segment and a text segment using the sequence of acoustic features and a token sequence provided by the beam search decoder; update the speech segment by appending the sequence of acoustic features to a last one of the speech segment and update the text segment by appending a token sequence of a recognition result of a previous utterance to a last one of the text segment; receive the updated speech segment, the updated text segment, and a partial token sequence from the beam search decoder; estimate token probabilities for the beam search decoder based on the speech segment and the text segment; and find a most probable token sequence from the estimated token probabilities as a speech recognition result using the beam search decoder.

[0012] Another embodiment of the present disclosure provides an end-to-end automatic speech recognition method. The method includes the following steps: obtaining a sequence of acoustic features comprising an utterance; arranging a speech segment and a text segment using the sequence of acoustic features and a token sequence provided from a beam search decoder; updating the speech segment by appending the sequence of acoustic features to a last one of the speech segment and updating the text segment by appending a token sequence of a recognition result of a previous utterance to a last one of the text segment; receiving the updated speech segment, the updated text segment, and a partial token sequence from the beam search decoder; estimating token probabilities for the beam search decoder based on the speech segment and the text segment; finding a most probable token sequence from the estimated token probabilities as a speech recognition result using the beam search decoder; and generating text data based on the most probable token sequence.

[0013] Further, another embodiment of the present disclosure provides a non-transitory computer readable medium including program instructions to cause a computer to perform an end-to-end automatic speech recognition method. The method includes the following steps: obtaining a sequence of acoustic features comprising an utterance; arranging a speech segment and a text segment using the sequence of acoustic features and a token sequence provided from a beam search decoder; updating the speech segment by appending the sequence of acoustic features to a last one of the speech segment and updating the text segment by appending a token sequence of a recognition result of a previous utterance to a last one of the text segment; receiving the updated speech segment, the updated text segment, and a partial token sequence from the beam search decoder; estimating token probabilities for the beam search decoder based on the speech segment and the text segment; finding a most probable token sequence from the estimated token probabilities as a speech recognition result using the beam search decoder; and generating text data based on the most probable token sequence.

[0014] The presently disclosed implementations will be further explained with reference to the drawings. The illustrated drawings are not necessarily drawn to scale, but are generally intended to illustrate the principles of the presently disclosed implementations. BRIEF DESCRIPTION OF DRAWINGS

[0015] [ Figure 1 ]

[0016] Figure 1 is a block diagram illustrating a long context end-to-end ASR system according to an implementation of the present application.

[0017] [ Figure 2 ]

[0018] Figure 2 is a block diagram illustrating a context expansion transformer according to an implementation of the present application.

[0019] [ Figure 3 ]

[0020] Figure 3 is a schematic diagram illustrating a speech recognition process performed on a first utterance using a context expansion transformer according to an implementation of the present application.

[0021] [ Figure 4 ]

[0022] Figure 4 is a schematic diagram illustrating a detailed computation flow in each encoder block according to an implementation of the present application.

[0023] [ Figure 5 ]

[0024] Figure 5 is a schematic diagram illustrating a detailed computation flow in each decoder block according to an implementation of the present application.

[0025] [ Figure 6A ]

[0026] Figure 6A is a schematic diagram illustrating some steps of a long context end-to-end ASR system with a context expansion transformer according to an implementation of the present application.

[0027] [ Figure 6B ]

[0028] Figure 6B is a schematic diagram illustrating some steps of a long context end-to-end ASR system with a context expansion transformer according to an implementation of the present application.

[0029] [ Figure 6C ]

[0030] Figure 6Cis a schematic diagram showing some steps of a long context end-to-end ASR system with context expansion transducers according to an embodiment of the application.

[0031] [ Figure 7A ]

[0032] Figure 7A Monologue ASR results on CSJ and TED-LIUM3 datasets according to an embodiment of the application are shown.

[0033] [ Figure 7B ]

[0034] Figure 7B Dialogue ASR results on SWITCHBOARD and HKUST datasets according to an embodiment of the application are shown. DETAILED DESCRIPTION

[0035] While the above figures set forth embodiments of the disclosure, other embodiments can also be contemplated as discussed in the Discussion. The disclosure presents illustrative embodiments by way of representation and not limitation. Those skilled in the art can devise numerous other variations and embodiments without departing from the principles of the presently disclosed embodiments.

[0036] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability or configuration of the disclosure. Rather, the following description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the subject matter disclosed in the appended claims.

[0037] In the following description, specific details are set forth to provide a thorough understanding of the embodiments. However, persons of ordinary skill in the art will understand that the embodiments can be practiced without these specific details. For example, the systems, processes, and other elements in the disclosed subject matter can be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known processes, structures and techniques can be shown without unnecessary detail in order to avoid obscuring the embodiments. Furthermore, like reference numerals and names are used to designate like elements in the various drawings.

[0038] Additionally, various embodiments can be described as a process depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or a block diagram. Although a flowchart can describe operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be re-arranged. A process can be terminated when its operations are completed, but could also occur intermittently during the process without a terminal shutdown of the process. When the process corresponds to a function, its termination can correspond to a return of the function to the calling function or the main function.

[0039] Furthermore, embodiments of the subject matter disclosed can be implemented, at least in part, in either manual or automated ways. Manual or automated implementations can be performed or at least assisted with the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks can be stored in a machine-readable medium. A processor(s) can perform the necessary tasks.

[0040] The modules and networks exemplified in the present disclosure can be computer programs, software, or instruction codes that can execute instructions using one or more processors. The modules and networks can be stored in one or more storage devices, or otherwise stored into a computer-readable medium, such as a storage medium, a computer storage medium, or a data storage device (removable and / or non-removable) such as a magnetic disk, an optical disk, or a magnetic tape, wherein the computer-readable medium can be accessed from the one or more processors to execute the instructions.

[0041] Computer storage media can include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media can be RAM, ROM, EEPROM, or flash memory, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by an application, module, or both, using one or more processors. Any such computer storage media can be part of the device or accessible or connectable thereto. Any application or module described herein can be implemented using computer readable / executable instructions that can be stored or otherwise held by such computer-readable media.

[0042] Long-context end-to-end ASR system 100

[0043] Figure 1A block diagram of a long-context end-to-end ASR system (ASR system) 100 according to some embodiments of the present application is shown. The ASR system 100 can include a human-machine interface (HMI) 110 connectable with a keyboard 111 and a pointing device / media 112, one or more processors 120, a storage device 130, a memory 140, a network interface controller 150 (NIC) connectable with a network 190 including a local area network and an internetwork, a display interface 160, an audio interface 170 connectable with an input device / microphone device 175, a printer interface 180 connectable with a printing device 185. The memory 140 can be one or more memory units. The ASR system 100 can receive electronic audio waveform / spectral data 195 via the network 190 connected with the NIC 150. The storage device 130 includes a long-context end-to-end ASR module 200, a context expansion transformer module 300, a speech segment update module 201, a beam search decoding module 202, a voice activity detection (VAD) module 132, and an acoustic feature extraction module 134. The pointing device / media 112 can include a module that reads a program stored on a computer-readable recording medium. The context expansion transformer module 300 can be formed of neural network parameters.

[0044] The VAD module 132 is a program for extracting spoken utterances from the electronic acoustic waveform of 195 or an electronic acoustic waveform received via the audio interface 170 from the microphone device 175. The VAD module 132 can extract spoken utterances by detecting the start and end positions of each spoken utterance in a given waveform and cutting out each spoken utterance from the waveform according to the start and end positions. The start and end positions can be decided based on the signal power, zero-crossing frequency, and / or frequency pattern of the waveform. If a given waveform is already an utterance that can be manually extracted by pressing a button or automatically obtained by other devices or computers (not shown), the VAD module 132 can not be included.

[0045] The acoustic feature extraction module 134 is a program for extracting an acoustic feature sequence from each spoken utterance determined by the VAD module 132. The acoustic feature sequence can be a sequence of mel-scale filter bank coefficients with its first and second order time derivatives and / or pitch features.

[0046] To perform long context end-to-end ASR, the keyboard 111, the pointing device / media 112, or instructions can be transmitted to the long context end-to-end ASR system 100 via a network 190 connected to other computers (not shown). The system 100 receives the instructions via the HMI 110 and executes the instructions for performing long context end-to-end ASR using the processor 120 connected to the memory 140 by loading the long context end-to-end ASR module 200, the speech segment update module 201, the beam search decoding module 202, the context expansion transformer module 300, and the acoustic feature extraction module 134 stored in the storage 130.

[0047] The long context end-to-end ASR module 200 outputs a token sequence as a recognition result for a given acoustic feature sequence and sends the token sequence to the display device 165 via the display interface 160, to the printing device 185 via the printer interface 180, or to other computers (not shown) via the network 190. The network 190 can be a wireless network or a wired network. In addition, the network 190 can be connected to the Internet. Each token in the token sequence can be a single word, a single letter, a single character, or a piece of text in the form of a word.

[0048] The long context end-to-end ASR module 200

[0049] Figure 2 A block diagram of a long context end-to-end ASR module (ASR module) 200 according to some embodiments of the present application is shown. The speech segment update module 201 arranges speech segments and text segments using the acoustic feature sequence 203 and the token sequence 204, which is given by the beam search decoding module (beam search decoder) 202. The context expansion transformer module 300 receives the speech segments and the text segments from the speech segment update module 201 and outputs / generates token probabilities for the beam search decoding module 202. The decoding module 202 finds the most likely token sequence based on the token probabilities, wherein the beam search decoding module 202 sends the beam search decoding module 202 a partial token sequence hypothesized in the beam search process of the beam search decoding module 202 to the transformer module 300 in order to receive from the transformer module 300 a probability of the next token of the partial token sequence.

[0050] The speech segment update module 201 updates the speech segment by appending an acoustic feature sequence 203 to the last part of the speech segment. The speech segment can be an acoustic feature sequence comprising one or more sentences. Initially, the speech segment can be an empty sequence. If the length of the speech segment exceeds a predetermined threshold, it can be shortened by deleting the oldest utterance from the speech segment through appending the acoustic feature sequence 203 to avoid making the speech segment too long. This length can be determined by the duration of the speech segment or the number of acoustic features in the speech segment. The speech segment update module 201 also arranges a text segment including a token sequence. The text segment can be updated by appending a token sequence 204 to the last part of the text segment, where the token sequence 204 is obtained from the beam search decoding module 202 as a recognition result of the previous utterance. Initially, the text segment can be an empty sequence. The text segment can also be shortened in the same way as shortening the speech segment by deleting the oldest token. The speech segment update module 201 outputs the updated speech segment and the updated text segment to the context expansion transformer module 300.

[0051] Voice segment update module 201

[0052] As one embodiment of the present invention, the voice segment update module 201 can be performed as follows. Let X... u and Y u X represents the acoustic feature sequence and token sequence of the u-th utterance in a long audio recording, respectively, where 1 ≤ u ≤ U, and U is the number of utterances in the audio recording. Acoustic feature sequence X u It is of length T u d a 3D vector sequence, i.e. in Token sequence Y u It is of length M u The token sequence, that is, Regarding the vocabulary list V, y u，m ∈V, serving as a system-defined set of tokens. In this disclosure, sequences are sometimes represented by a series of indices, for example, a utterance sequence X. v , ..., X u It was also written as X v：u .

[0053] The voice segment update module 201 includes voice segment S X and text segment S Y Before processing long audio recordings, the speech segment S is divided into empty sequences. X and text segment S Y Initialization (referred to as the initial phase). Given an acoustic feature sequence X. u The voice segment update module 201 updates via the attached X u Update voice segment S X That is, SX ← S X X u The update module 201 also receives Y u-1 from the beam search decoding module 202 and updates the text segment S u-1 by appending Y Y , i.e., S Y ← S Y Y u-1 Note that Y u-1 is the ASR result obtained for X u-1 , X u-1 is the sequence of acoustic features of the previous utterance.

[0054] If the length of the speech segment S X | S X | is greater than a predetermined threshold value Γ, the speech segment S X may be shortened. Assuming S X = X v:u , such that 1 ≤ v < u, S X may be shortened to

[0055]

[0056] where v' is the smallest index satisfying v < v' ≤ u and |X v′:u | ≤ Γ. The segment length is computed as The text segment S Y may also be shortened to S Y ← Y v′:u-1 so as to be consistent with the speech segment.

[0057] Contextual expansion transformer 300

[0058] Figure 3 A contextual expansion transformer module 300 is shown as one embodiment of the present invention. The network architecture is similar to the conventional transformer of the prior art, which includes an encoder, a decoder, and self-attention layers. The conventional transformer of the prior art accepts essentially / only a single sentence at a time and predicts the token probabilities of the single sentence, where no contextual information is used.

[0059] In contrast, the contextual expansion transformer of the present invention is configured to accept a speech segment 301 and a text segment 302 comprising multiple sentences, and predict the token probabilities 303 of the last sentence 304 of the speech segment, where the token sequence of the text segment and the sequence of acoustic features of the speech segment excluding the last sentence are used as contextual information to adapt the hidden vectors in the transformer module 300.

[0060] The context expansion transformer module 300 receives the speech segment 301 and the text segment 302 from the speech segment update module 201, and also receives the partial token sequence 305 from the beam search decoding module 202. From these input sequences, the transformer module 300 estimates the token probabilities 303. The token probabilities 303 are used in the beam search decoding module 202 to find the most likely token sequence as the speech recognition result of the last utterance 304.

[0061] The goal of the long-context end-to-end ASR according to the present application is to find the most likely token sequence v:u and the text segment Y v:u-1 as the speech recognition result of the speech segment X

[0062]

[0063] where in equation (2), the probability p(y u，m | Y v:u-1 , x u，1:m-1 , X v:u ) is computed using the context expansion transformer module 300.

[0064] The context expansion transformer module 300 comprises an encoder 311 and a decoder 321. The encoder 311 first applies a two-dimensional convolution (Conv2D) 312 to the speech segment X v:u 301. In Figure 3 , it is assumed that X v:u = X u-2:u , i.e., v = u - 2.

[0065] The Conv2D 312 computes a sequence of hidden vectors

[0066]

[0067] where is the sequence of hidden vectors, is a convolution weight tensor, is a bias vector, and “*” represents a convolution operation applied to both the time dimension and the feature dimension of X v:u , C is the number of channels, s t and s f are the span sizes of the time dimension and the feature dimension, and K t and K f are the kernel sizes of the time dimension and the feature dimension, respectively. ReLU(·) is a rectified linear unit used as an activation function. The Conv2D 312 can be repeated one or more times.

[0068] The positional encoding (PosEnc) 313 makes the positional encoding vector of the sequence of vectors ​​

[0069] PosEnc(H) = [PE1,..., PE i ,..., PE L ], (4)

[0070] where PE i is the position encoding vector at position i, and its 2jth element and (2j+1)th element are computed as

[0071]

[0072]

[0073] L is the length of the vector sequence H, and d is the dimension of each column vector in H.

[0074] The output of PosEnc 313 is then added with respect to to obtain the first hidden vector sequence i.e.

[0075]

[0076] The encoder 311 has N enc encoder blocks 314, which are vertically stacked, with the first hidden vector being fed into the first encoder block of 314.

[0077] Each encoder block has a self-attention layer 315, which is based on a multi-head attention mechanism. A scaled dot-product attention can be computed, i.e.

[0078]

[0079] where Q is a matrix of packed query vector sequences, K and V are matrices of key and value vectors in the packed sequences, respectively. d k is the dimension of each key vector.

[0080] Multi-head attention allows the model to jointly attend to information at different positions in the input sequence from multiple representation subspaces:

[0081] MHA(Q, K, V) = Concat(head1,..., head M )W o , (9)

[0082] where

[0083]

[0084] and by parameter matrices and​ to apply the projection.d v is the dimension of each value vector, d h is the dimension of each hidden vector in the encoder or decoder, and in a typical transformer architecture d k = d v = d h / M. Concat(·) denotes the concatenation of matrices.

[0085] The self-attention layer 315 in the nth encoder block computes a sequence of hidden vectors from a multi-head attention computation where the query, key, and value vectors all come from the sequence of hidden vectors obtained in the previous encoder block or a Conv2D with PosEnc. Thus, each column vector in is adapted by the attention mechanism from the other vectors in

[0086] Then, a sequence of hidden vectors is obtained, i.e.

[0087]

[0088] where ξ(·) represents a layer normalization 316, which linearly normalizes a given vector using learned mean and standard deviation parameters.

[0089] Then, a feed-forward network 317 and another layer normalization 318 are applied to i.e.

[0090]

[0091] where FFN(·) represents the feed-forward network 317. The encoder 311 finally outputs an encoder state as the sequence of hidden vectors generated by the last encoder block.

[0092] Figure 4 The detailed computation flow in each encoder block is shown, where from an input sequence of hidden vectors 401 an output sequence of hidden vectors 402

[0093] In Figure 4 and and are decomposed into a sequence of hidden vectors

[0094] and

[0095] The self-attention 415 performs self-attention of equation (11), where Each hidden vector of Y is obtained by a weighted sum of all hidden vectors of the multi-sentence utterance of the segment This enables each hidden vector to adapt to the speaker of the segment.

[0096] Figure 3 The decoder 321 in Y accepts a text segment Y v:u-1 , a partial token sequence y u，1:i-1 , and an encoder state and estimates the probability distribution of y u，i in equation (2). The text segment Y v:u-1 is extended by appending the partial token sequence y u，1:i-1 , and the extended text segment is written as y' u，1:k-1 , which represents all previous tokens up to index k-1, where |Y v:u-1 | < k < |Y v:u | and k = |Y v:u-1 | + i. |Y * | denotes the number of tokens in the sequence Y * .

[0097] The decoder 321 first applies token embeddings and position encoding, i.e.

[0098]

[0099] where Embed(·) represents the token embedding, which converts each token into a d d -dimensional vector. Next, the decoder computes the hidden vector in each nth decoder block 324, i.e.

[0100]

[0101]

[0102]

[0103] and outputs the decoder state of the last decoder block, i.e., where N dec denotes the number of decoder blocks. Equation (14) applies self-attention 325 and layer normalization 326. Equation (15) applies source attention 327 to the encoder state and then applies layer normalization 328, where is used for the query vector. Equation (16) applies a feed-forward network 329 and layer normalization 330. Finally, the transformer token probability distribution is obtained by applying a linear transformation 331 and a softmax function 332, i.e.

[0104]

[0105] Figure 5 The detailed computation flow in each decoder block 500 is shown, where self-attention 525, layer normalization 526, source attention 527, layer normalization 528, feedforward network 529, and layer normalization 530 are used to transform the input hidden vector sequence 501 into the output hidden vector sequence 502

[0106] In the Figure 5 , the self-attention 525 performs the self-attention of equation (14), where each hidden vector of is obtained by a weighted sum of all hidden vectors of , which enables each hidden vector to adapt to the topic of the text segment. The source attention 527 performs the source attention of equation (15), which takes the encoded speech segment from the last encoder block 503, where the hidden vector sequence of the encoded segment in the multi-sentence utterance is used to adapt the hidden vector sequence of the decoder

[0107] Beam search decoding module 202

[0108] The beam search decoding module 202 finds the most likely token sequence based on equation (2), where the token probability p(y u，i |Y v:u-1 , y u，1:i-1 , X v:u ) is given by the context expansion transformer module 300.

[0109] The beam search decoding module 202 of the present invention finds the most likely token sequence in equation (2) However, it is difficult to enumerate all possible token sequences of Y u and to compute , because the number of possible token sequences increases exponentially to the length of the sequence. Therefore, the beam search technique is usually used to find where shorter token sequence hypotheses are first generated, and only limited hypotheses with scores higher than other hypotheses are expanded to obtain longer hypotheses. Finally, the best token sequence hypothesis is selected among the complete hypotheses that reach the end of the sequence.

[0110] The beam search decoding module 202 finds Let Ω l be a set of partial hypotheses with length l. At the beginning of the beam search, Ω0only contains the hypothesis with the start token <sos>of the hypothesis, its score a( <sos>The value is set to 0. For l = 1 to L max Ω can be expanded by attaching a possible single token. l-1 Each part of the hypothesis is assumed, and new hypotheses are stored in Ω. l In the middle, L max This is the maximum length of the hypothesis to be searched. A score is calculated for each new hypothesis in the logarithmic field, i.e.

[0111] i.α(h)=α(g)+log p(y|Y v:u-1 g, X v:u ), (18)

[0112] Where g is Ω l-1 Partial assumptions in the hypothesis are that y is a single token attached to g, and h is a new hypothesis, i.e., h = g·y. The probability p(y|Y) is... v:u-1 g, X v:u ) can be calculated using equation (17).

[0113] If y represents a sequence <eos>the special token that marks the end of the utterance, then h is added to instead of Ω l where denotes the complete hypothesis set.

[0114] Finally, the complete hypothesis set is obtained as

[0115]

[0116] In the beam search process, Ω l is allowed to keep only a limited number of hypotheses with higher scores, and other hypotheses are pruned to improve search efficiency.

[0117] Figure 6A , 6B and 6C show schematic diagrams of some steps of a long context end-to-end ASR system with context expansion transducers according to embodiments of the application.

[0118] In Figure 6A , the system recognizes a speech segment X 1:1 that includes only the first sentence X1 without relying on the previous context, and outputs the first token sequence hypothesis In Figure 6B , the system reads the speech segment X 1:2 obtained from the speech segment update module. Then, the system recognizes X2 conditioned on the previous speech context X1 and the text context , and outputs the second token sequence hypothesis In this way, the system can recognize utterances along a long audio input. Due to time and memory space constraints, the speech segment update module can reduce the size of the segment by truncating the oldest utterances in the segment if the length of the speech segment exceeds a predetermined threshold Γ. When a speech segment is truncated, the corresponding text segment can also be truncated. Figure 6C An example is shown in which the system recognizes X5 for a given speech segment X 3:5 and text segment to obtain

[0119] In some embodiments of the application, various ways of constructing the speech segment and the text segment are also considered. Typically, only a single speaker is included in each speech segment of an utterance. However, different speakers can be included in a conversational speech segment. In Figure 6C , the speech segment X 3:5 In this case, X3 can be spoken by speaker A, X4 is spoken by speaker B, and X5 is spoken by speaker C. With our method, speech features from different speakers can negatively affect the transformer in terms of speaker adaptation. To avoid such segments (segments with negative effects), we propose to make each segment have only a single speaker. For example, if X1, X3, and X5 are spoken by the same speaker, then the input segment is constructed as (X1, X3, X5) to identify X5. The output context can be made to be or the output context is simply based on the initial way to avoid breaking the conversation context. Depending on whether the speaker information is used or not, we call these input and output contexts speaker-dependent (SD) and speaker-independent (SI), respectively. The same way is applied in the training phase.

[0120] To construct the SD segments, we need to know the speaker ID of each utterance. This is a limitation of the SD method, but we can usually obtain the utterances of each speaker from the channel associated with the speaker in a telephone conversation or a meeting recorded by a worn microphone. Speaker classification techniques can also be used to recognize the speaker recorded with a distant microphone.

[0121] Training procedure

[0122] In the context expansion transformer 300, the model parameters are optimized to minimize the transformer loss

[0123]

[0124] where X and U are the training data including speech segments and text segments, i.e., Q is the number of long audio recordings in the training data, and I q is the number of utterances (or segments) in the qth audio recording. v u is the utterance index representing the start position of the u-th segment, which is determined based on the length of the speech segment and a threshold value Γ.

[0125] Θ represents the set of model parameters of the Conv2D, self-attention network, layer normalization, and feedforward network in the encoder 311, and the token embedding network, self-attention network, source attention network, layer normalization, and linear network in the decoder 321.

[0126] The set of network parameters Θ can be optimized by the stochastic gradient descent method, where each element of the set of parameters Θ is repeatedly updated to

[0127]

[0128] until converges, where η is the learning rate.

[0129] X and Y can also be divided into M small subsets such that and and update the parameters by repeating the following for m = 1,..., M

[0130]

[0131] By updating the parameters with small subsets, the parameters are updated more frequently and the loss function converges faster.

[0132] In some embodiments of the invention, the Connectionist Temporal Classification (CTC) loss can also be used for training and decoding to improve recognition accuracy. The CTC sequence probability can be computed as

[0133]

[0134] where CTC(P, Y) is an operation that marginalizes the posterior probabilities of all possible alignments between P and Y using the forward-backward algorithm.

[0135] For training, the CTC loss can be added to the transducer loss in equation (20), i.e.

[0136] where λ is a scaling factor that balances the transducer and CTC losses, and Θ includes the linearly transformed parameters of CTC in equation (23), which are jointly optimized with the parameters of the context expansion transducer.

[0137] Evaluation

[0138] Some experiments were conducted using the monologue benchmarks in the CSJ [1] and TED-LiUM3 [2] corpora, and the conversation benchmarks in the SWITCHBOARD [3] and HKUST [4] corpora.

[0139] [1] K. Maekawa, H. Koiso, S. Furui, and H. Isahara, "Spontaneous speech corpus of Japanese," in Proc. LREC, vol. 2, pp. 947-952, 2000.

[0140] [2] F. Hernandez, V. Nguyen, S. Ghannay, N. Tomashenko, Y. Esteve, O. Jokisch and R. Potapova, "TED-LIUM 3: Twiceas much data and corpus repartition for experiments on speaker adaptation", pp. 198-208, Speech and Computer, 2018.

[0141] [3] J. Godfrey, E. Holliman and J. McDaniel published "SWITCHBOARD: telephone speech corpus for research and development" in Proc. IEEE ICASSP, Vol. 1, pp. 517-520 in 1992.

[0142] [4] Y. Liu, P. Fung, Y. Yang, C. Cieri, S. Huang and D. Graff published "HKUST / MTS: A very large scale mandarin telephonespeechcorpus" in Chinese Speech and Language Processing, pp. 724-735, in 2006.

[0143] An 80-dimensional log-mel-filter bank acoustic feature was extracted from the speech data in the corpus, along with 3-dimensional pitch features (d). a =83). Basically, the hidden vectors in the encoder and decoder have 256 dimensions (d). h =256). The encoder has one Conv2D module, followed by 12 encoder blocks (N). enc =12). Conv2D consists of two layers of 2D convolutions with 256 channels (C=256) and a kernel size of 3×3 (K). t =K f =3), the span is 2 (s) t =s f =2), and has ReLU activation, its output is a 256-dimensional hidden vector sequence, reducing the sequence length to 1 / 4. A 64-dimensional four-head (M=4) multi-head attention (d) is employed. k = d v = 64). The feed-forward network has one hidden layer with 2,048 units and ReLU nonlinearity. The decoder has a token embedding layer followed by 6 decoder blocks (N dec = 6). The dimensions of the self-attention, source attention, and feed-forward layers in the decoder are the same as in the encoder, i.e., d e = d d = 256. The output dimension depends on the number of unique tokens in the task. There are 3260 characters in CSJ, 652 wordpieces in TED-LIUM3, 1996 wordpieces in SWITCHBOARD, and 3653 characters in HKUST. All models are trained with the end-to-end speech processing toolkit ESPnet. We generally follow the default configuration for each task in the ESPnet recipe, where speed perturbation is applied to all datasets we use, but SpecAugment is only applied to TED-LIUM3 and SWITCHBOARD. The baseline transducer (state-of-the-art) is trained on independent utterances without context. In addition, speaker adaptation based on i-vectors is also tested for comparison, where the speaker i-vectors are concatenated to each frame vector immediately before the first encoder block. To train the transducer with the proposed method, each utterance is extended into segments by concatenating with the previous utterance, where we set the segment length threshold Γ to 2000, which corresponds to 20 seconds, since the acoustic feature vectors are extracted with a period of 10 milliseconds.

[0144] Finally, the parameters of the top 5 models based on the validation loss are averaged to obtain the final model for recognition. The LSTM-based RNN language model is also trained using the transcriptions of CSJ and HKUST, as well as the external text data of TED-LIUM3. The transcriptions are concatenated in the same way as the context-extended transducer training. No LM is used for SWITCHBOARD. The ASR performance is measured by the character error rate (CER) or word error rate (WER).

[0145] Figure 7A Monologue ASR results are shown for the CSJ and TED-LIUM3 datasets. For each dataset, the proposed method greatly reduces the ASR errors from the baseline, with a reduction in relative error rate of 5% to 15%. The i-vector method does not consistently reduce the error rate. Figure 7B Dialogue ASR results are shown for the SWITCHBOARD and HKUST datasets.

[0146] For each dataset, the proposed method greatly reduces ASR errors from the baseline, especially in the SD context, with a relative error rate reduction of 7% to 13.5%. We also demonstrate that the SD context is better than the SI context in conversational ASR.

[0147] Embodiments of the invention can be realized as a method, for which examples have been provided. The acts performed as part of the method can be ordered in any suitable way. Accordingly, embodiments can be constructed in which acts are performed in an order different than illustrated, which can include acts being performed at the same time even though this is not shown in illustrative embodiments.

[0148] The use of ordinal terms such as "first", "second", etc., in the claims to modify a claim element does not by itself connote any priority, precedence, or order of one claim element over another, or the temporal sequence of acts of a method, but are used merely as labels to distinguish one claim element having a certain name from another element having a same name (except for a hardware circuit claim whereby a first and second means can connote circuit elements having certain prioritization or order of acts in operations).

[0149] While the present invention has been described by examples of preferred embodiments, it is understood that various other modifications and changes can be made by those skilled in the art which follow in the spirit and scope of the present invention.

[0150] Accordingly, it is the intention of the appended claims to cover all such variations and modifications as falling within the true spirit and scope of the present invention.< / eos> < / sos> < / sos>

Claims

1. An end-to-end automatic speech recognition, ASR, system (100), the ASR system (100) comprising: an interface (170) configured to obtain a sequence of acoustic features comprising an utterance; a memory (140) configured to store a computer executable ASR program module comprising a context expansion transducer network, the context expansion transducer network comprising an encoder network and a decoder network, a beam search decoder (202), and a speech segment update module (201); a processor (120) connected with the memory (140) configured to repeatedly decode the utterance by performing the following steps: arranging a speech segment and a text segment using the sequence of acoustic features and a sequence of tokens provided from the beam search decoder (202); updating the speech segment by appending the sequence of acoustic features to the last of the speech segment and updating the text segment by appending a sequence of tokens of a recognition result of a previous utterance to the last of the text segment; receiving the updated speech segment, the updated text segment, and a partial sequence of tokens from the beam search decoder (202); estimating token probabilities for the beam search decoder (202) based on the speech segment and the text segment, wherein the sequence of acoustic features excluding the last utterance in the speech segment and the sequence of tokens of the text segment are used as context information through the context expansion transducer network; and finding a most probable sequence of tokens from the estimated token probabilities using the beam search decoder as a speech recognition result.

2. The ASR system of claim 1, further comprising an output interface configured to generate text data from the most probable sequence of tokens.

3. The ASR system of claim 1, wherein, the processor is configured to stop decoding when no more sequence of acoustic features is provided, or wherein the processor is configured to detect when each utterance comprises a speaker dependent, SD, context.

4. The ASR system of claim 1, wherein, The SD context is determined by a speaker identification data, ID.

5. The ASR system of claim 4, wherein, The speaker ID is determined by a channel associated with a recording device storing utterances of the speaker or a microphone arranged for the speaker.

6. The ASR system of claim 1, wherein, The step of updating the speech segment is performed by appending the sequence of acoustic features, wherein the step of updating the text segment is performed by appending the sequence of tokens of the recognition result of a previous utterance to the last of the text segment.

7. The ASR system of claim 1, wherein, The text segment is initially an empty sequence.

8. The ASR system of claim 1, wherein, Each of the encoder network and the decoder network comprises a deep feedforward architecture having a repetition of blocks of self-attention layers and feedforward layers.

9. The ASR system of claim 8, wherein, The decoder network is characterized by having a source attention layer in each of the repetition of blocks to read output from the encoder.

10. The ASR system of claim 1, wherein, The self-attention mechanism and the source attention mechanism exploit mutual dependencies between input frames and output tokens.

11. The ASR system of claim 1, wherein, The transducer is configured to accept multiple utterances at a time and exploit previous utterances to predict output tokens for the last utterance.

3. The ASR system of claim 1 or 2, wherein the processor is configured to perform the steps of: receiving a sequence of acoustic features comprising an utterance; arranging a speech segment and a text segment using the sequence of acoustic features and a sequence of tokens provided from the beam search decoder (202); updating the speech segment by appending the sequence of acoustic features to the last of the speech segment and updating the text segment by appending a sequence of tokens of a recognition result of a previous utterance to the last of the text segment; receiving the updated speech segment, the updated text segment, and a partial sequence of tokens from the beam search decoder (202); estimating token probabilities for the beam search decoder (202) based on the speech segment and the text segment, wherein the sequence of acoustic features excluding the last utterance in the speech segment and the sequence of tokens of the text segment are used as context information through the context expansion transducer network; finding a most probable sequence of tokens from the estimated token probabilities using the beam search decoder as a speech recognition result.

4. The ASR system of claim 3, wherein the processor is configured to perform the steps of: receiving a sequence of acoustic features comprising an utterance; arranging a speech segment and a text segment using the sequence of acoustic features and a sequence of tokens provided from the beam search decoder (202); updating the speech segment by appending the sequence of acoustic features to the last of the speech segment and updating the text segment by appending a sequence of tokens of a recognition result of a previous utterance to the last of the text segment; receiving the updated speech segment, the updated text segment, and a partial sequence of tokens from the beam search decoder (202); estimating token probabilities for the beam search decoder (202) based on the speech segment and the text segment, wherein the sequence of acoustic features excluding the last utterance in the speech segment and the sequence of tokens of the text segment are used as context information through the context expansion transducer network; finding a most probable sequence of tokens from the estimated token probabilities using the beam search decoder as a speech recognition result.

12. The ASR system of claim 1, wherein, In case the duration of a segment exceeds a predetermined constant length, the size of the segment is determined by truncating the oldest utterance in the segment.

13. The ASR system of claim 12, wherein, When an input segment is truncated, an output context corresponding to the input segment is truncated.

14. The ASR system of claim 1, wherein, The transformer comprises a self-attention layer, a source attention layer and a feed-forward layer in the decoder.

15. An end-to-end automatic speech recognition (ASR) method, the ASR method comprising the steps of: obtaining an acoustic feature sequence comprising utterances; arranging a speech segment and a text segment using the acoustic feature sequence and a token sequence provided from a beam search decoder; updating the speech segment by appending the acoustic feature sequence to the last of the speech segment, and updating the text segment by appending a token sequence of a recognition result of a previous utterance to the last of the text segment; receiving the updated speech segment, the updated text segment and a partial token sequence from the beam search decoder; estimating token probabilities for the beam search decoder based on the speech segment and the text segment, wherein the acoustic feature sequence excluding the last utterance in the speech segment and the token sequence of the text segment are used as context information by a context expansion transformer network; finding a most probable token sequence as a speech recognition result from the estimated token probabilities using the beam search decoder; and generating text data based on the most probable token sequence.

16. The ASR method according to claim 15, further comprising the step of completing the decoding when no more acoustic feature sequence is provided, or the step of detecting whether each utterance comprises a speaker-dependent (SD) context.

17. A non-transitory computer-readable medium comprising program instructions to cause a computer to perform a method, the method comprising the operations of: obtaining an acoustic feature sequence comprising utterances; arranging a speech segment and a text segment using the acoustic feature sequence and a token sequence provided from a beam search decoder; updating the speech segment by appending the acoustic feature sequence to the last of the speech segment, and updating the text segment by appending a token sequence of a recognition result of a previous utterance to the last of the text segment; receiving the updated speech segment, the updated text segment and a partial token sequence from the beam search decoder; estimating token probabilities for the beam search decoder based on the speech segment and the text segment, wherein the acoustic feature sequence excluding the last utterance in the speech segment and the token sequence of the text segment are used as context information by a context expansion transformer network; finding a most probable token sequence as a speech recognition result from the estimated token probabilities using the beam search decoder; and generating text data based on the most probable token sequence.

Images