Transformer-based automatic speech recognition system with time reduction layer

By introducing time reduction and self-attention operations into the transformer-based ASR system, computational efficiency is optimized, solving the problem of low computational efficiency for long input sequences and achieving higher speech recognition accuracy and prediction precision.

CN116324973BActive Publication Date: 2026-11-10HUAWEI TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202180068843.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-10-21
Filing Date
2021-06-29
Publication Date
2026-11-10
Estimated Expiration
2041-06-29

AI Technical Summary

Technical Problem

Transformer-based automatic speech recognition systems are computationally inefficient when processing long input sequences, and self-attention mechanisms struggle to effectively learn the temporal dependencies of frame-level acoustic features.

Method used

We employ time reduction and self-attention operations, using an encoder neural network (NN) to process speech frame feature vectors to reduce computation, and a decoder neural network (NN) to predict label sequences, combined with gradient descent to optimize model parameters.

Benefits of technology

It improves computational efficiency, reduces computational costs, and enhances the accuracy and prediction precision of speech recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116324973B_ABST
    Figure CN116324973B_ABST
Patent Text Reader

Abstract

A computer-implemented method and system for automatic speech recognition are provided. A first speech sequence is processed into a second speech sequence comprising a second set of speech frame feature vectors using a time-shortening operation of an encoder neural network (NN), each of the speech frame feature vectors being concatenated with information from corresponding plurality of speech frame feature vectors included in the first set, wherein the second speech sequence comprises fewer speech frame feature vectors than the first speech sequence. The second speech sequence is then transformed into a third speech sequence comprising a third set of speech frame feature vectors using a self-attention operation of the encoder NN. The third speech sequence is processed using a probabilistic operation of the encoder NN to predict a first label sequence corresponding to the third set of speech frame feature vectors. The third speech sequence is further processed using a decoder neural network (NN) to predict a second label sequence corresponding to the third set of speech frame feature vectors.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application claims priority to U.S. Patent Application Serial No. 17 / 076,794, filed on October 21, 2020, entitled “Transfermer-Based Automatic Speech Recognition System Incorporating Time-Reduction Layer,” which is incorporated herein by reference in its entirety. Technical Field

[0002] This invention relates to automatic speech recognition using neural networks. Background Technology

[0003] End-to-end automatic speech recognition (ASR) systems are becoming increasingly important because they rely on relatively simple training and inference processes compared to traditional systems based on hidden Markov models (HMMs). One such ASR system employs a transformer architecture that uses multi-head self-attention to model temporal contextual information. This type of ASR system is often referred to as a transformer-based ASR system. A transformer-based ASR system performs ASR on a sequence of frame-level acoustic features from an audio file. A transformer-based ASR consists of a transformer encoder and a transformer decoder, each implemented using a corresponding set of neural network layers. The transformer encoder receives a sequence of frame-level acoustic features (x1, ..., xT) as input and maps it to a high-level representation sequence (h1, ..., hN). The decoder generates a transcription (y1, ..., yL) of the audio file one token at a time (word-level tokens or sub-word units, such as characters generated by byte-pair encoding, or sentence fragments). Each tag yl is conditioned on its representation (h1, ..., hN) and the previously generated tag (y1, ..., yl–1) through an attention mechanism. The basic fact transcription of an audio file (i.e., the target transcription or target sequence) is represented by word-level tags or sub-word units, such as characters generated by byte-pair encoding, or sentence fragments. Transformer-based ASR systems have been shown to achieve significantly lower word error rates (WERs) compared to recurrent neural network (RNN)-based system architectures. An example of such a transformer-based ASR system is described in: S. Karita et al., A Comparative Study on Transformers vs. RNNs in Speech Applications, ASRU 2019, arXiv:1909.06317v2[cs.CL], September 28, 2019.

[0004] Self-attention mechanisms apply an attention matrix to a sequence of frame-level acoustic features to learn temporal dependencies within the input sequence. Applying self-attention to speech recognition is challenging because individual speech frames, which include acoustic features, are not lexical units like words. Individual speech frames cannot convey different meanings, which can make it difficult for automatic self-attention mechanisms to compute appropriate attention weights. Transformer-based ASR systems suffer from reduced computational efficiency and learn longer but meaningless dependencies for longer input sequences.

[0005] Therefore, it is necessary to improve the converter-based ASR system. Summary of the Invention

[0006] This invention provides a transformer-based ASR system that optimizes computational efficiency and enriches input feature-level acoustic features for longer sequences of frame-level acoustic features input to the transformer-based ASR system. It also provides a method for performing ASR using the transformer-based ASR system.

[0007] According to a first exemplary aspect of the present invention, a computer-implemented method for automatic speech recognition is provided, comprising: acquiring a first speech sequence including a first set of speech frame feature vectors, each of the speech frame feature vectors representing a corresponding speech frame corresponding to a corresponding time step; processing the first speech sequence into a second speech sequence including a second set of speech frame feature vectors using a time reduction operation of an encoder neural network (NN), each of the speech frame feature vectors being concatenated with information from a corresponding plurality of speech frame feature vectors included in the first set, wherein the second speech sequence includes fewer speech frame feature vectors than the first speech sequence; transforming the second speech sequence into a third speech sequence including a third set of speech frame feature vectors using a self-attention operation of the encoder NN; processing the third speech sequence using a probabilistic operation of the encoder NN to predict a first label sequence corresponding to the third set of speech frame feature vectors; and processing the third speech sequence using a decoder neural network (NN) to predict a second label sequence corresponding to the third set of speech frame feature vectors.

[0008] In at least some scenarios, applying time reduction operations before self-attention operations can reduce the number of computational operations that the self-attention operations need to perform.

[0009] In at least some examples of the foregoing aspects, the method includes, during the training phase of the encoder NN and the decoder NN: calculating a loss function based on the predicted first label sequence and the predicted second label sequence; and performing backpropagation using gradient descent to update the learnable parameters of the decoder NN and the encoder NN to reduce the loss function.

[0010] In at least some examples of one or more of the foregoing aspects, the method includes, during the inference phase, calculating a label sequence of the third speech sequence based on the predicted first label sequence and the predicted second label sequence.

[0011] In at least some examples of one or more of the foregoing aspects, an input speech sequence is obtained comprising an input set of speech frame feature vectors, each of which represents a corresponding speech frame at a corresponding time step; the input speech sequence is processed into the first speech sequence using a subsampling operation of the encoder NN, wherein the first speech sequence comprises fewer speech frame feature vectors than the input speech sequence.

[0012] In at least some examples of one or more of the foregoing aspects, the time reduction operation is performed using one or more linear NN layers of the encoder NN.

[0013] In at least some examples of one or more of the foregoing aspects, obtaining the first speech sequence includes: obtaining an initial speech sequence comprising an initial set of speech frame feature vectors, each of the speech frame feature vectors representing a corresponding speech frame corresponding to a corresponding time step; and processing the first speech sequence into the first speech sequence using another self-attention operation of the encoder NN prior to the time reduction operation.

[0014] In at least some examples of one or more of the foregoing aspects, obtaining the initial speech sequence includes: obtaining an input speech sequence comprising an input set of speech frame feature vectors, each of the speech frame feature vectors representing a corresponding speech frame corresponding to a corresponding time step; and processing the input speech sequence into the initial speech sequence using a subsampling operation of the encoder NN, wherein the initial speech sequence comprises fewer speech frame feature vectors than the input speech sequence.

[0015] In at least some examples of one or more of the above aspects, the self-attention operation and the other self-attention operation are each performed by a corresponding sub-network of the self-attention layer.

[0016] In at least some examples of one or more of the foregoing aspects, the method includes using a corresponding number of self-attention layers for each of the self-attention operation and the other self-attention operation based on the obtained hyperparameters.

[0017] According to another exemplary aspect, an automatic speech recognition computing system is disclosed, comprising: a memory storing executable instructions; and a processing device communicating with the memory. The processing device is configured to execute the instructions to cause the computing system to: acquire a first speech sequence including a first set of speech frame feature vectors, each of the speech frame feature vectors representing a corresponding speech frame corresponding to a corresponding time step; process the first speech sequence using a time reduction operation into a second speech sequence including a second set of speech frame feature vectors, each of the speech frame feature vectors being concatenated with information from a corresponding plurality of speech frame feature vectors included in the first set, wherein the second speech sequence includes fewer speech frame feature vectors than the first speech sequence; transform the second speech sequence using a self-attention operation into a third speech sequence including a third set of speech frame feature vectors; process the third speech sequence using a probabilistic operation to predict a first label sequence corresponding to the third set of speech frame feature vectors; and process the third speech sequence using another self-attention operation and another probabilistic operation to predict a second label sequence corresponding to the third set of speech frame feature vectors.

[0018] In some exemplary embodiments of the system, the processing device is configured to execute the instructions to enable the computing system to implement an encoder neural network (NN) and a decoder NN, wherein the time reduction operation, the self-attention operation, and the probabilistic operation are each executed using a corresponding sub-network of the encoder NN, and the other self-attention operation and the other probabilistic operation are each executed using a corresponding sub-network of the decoder NN.

[0019] According to another exemplary aspect, a computer-readable medium is provided that stores computer instructions, which, when executed by a processing device of a computer system, cause the computer system to: acquire a first speech sequence including a first set of speech frame feature vectors, each of the speech frame feature vectors representing a corresponding speech frame corresponding to a corresponding time step; process the first speech sequence into a second speech sequence including a second set of speech frame feature vectors using a time reduction operation of an encoder neural network (NN), each of the speech frame feature vectors being concatenated with information from a corresponding plurality of speech frame feature vectors included in the first set, wherein the second speech sequence includes fewer speech frame feature vectors than the first speech sequence; transform the second speech sequence into a third speech sequence including a third set of speech frame feature vectors using a self-attention operation of the encoder NN; process the third speech sequence using a probabilistic operation of the encoder NN to predict a first label sequence corresponding to the third set of speech frame feature vectors; and process the third speech sequence using a decoder neural network (NN) to predict a second label sequence corresponding to the third set of speech frame feature vectors.

[0020] According to another exemplary aspect, an automatic speech recognition system is provided, comprising: an encoder neural network capable of acquiring a first speech sequence including a first set of speech frame feature vectors, each of the speech frame feature vectors representing a corresponding speech frame at a corresponding time step; the encoder neural network performing: a time reduction operation to transform the first speech sequence into a second speech sequence including a second set of speech frame feature vectors, each of the speech frame feature vectors being concatenated with information from a corresponding plurality of speech frame feature vectors included in the first set, wherein the second speech sequence includes fewer speech frame feature vectors than the first speech sequence; a self-attention operation to transform the second speech sequence into a third speech sequence including a third set of speech frame feature vectors using a self-attention mechanism; and a probabilistic operation to predict a first label sequence corresponding to the third set of speech frame feature vectors. The system further comprises a decoder neural network to process the third speech sequence to predict a second label sequence corresponding to the third set of speech frame feature vectors. Attached Figure Description

[0021] The accompanying drawings, which now illustrate exemplary embodiments of this application, are shown by way of example, wherein:

[0022] Figure 1 This is a block diagram of the structure of an automatic speech recognition (ASR) system provided in an exemplary embodiment;

[0023] Figure 2 This is provided by an exemplary embodiment. Figure 1 A block diagram of the subsampling operation of the ASR system;

[0024] Figure 3 This is provided by an exemplary embodiment. Figure 1 A flowchart of the time reduction operation of the ASR system;

[0025] Figure 4 This is provided by an exemplary embodiment. Figure 1 A flowchart of the training phase process of the ASR system;

[0026] Figure 5 This is a block diagram of the structure of an ASR system provided in another exemplary embodiment;

[0027] Figure 6 This is provided by an exemplary embodiment. Figure 5 A flowchart of the encoder training phase process of the ASR system;

[0028] Figure 7 This is a block diagram of the structure of an ASR system provided in another exemplary embodiment;

[0029] Figure 8 This is a block diagram of the structure of an ASR system provided in another exemplary embodiment;

[0030] Figure 9 This is a block diagram of a computing system that can be used to host an ASR system, which is an exemplary embodiment.

[0031] The same reference numerals may be used to denote the same components in different figures. Detailed Implementation

[0032] Exemplary embodiments will be described more fully below with reference to the accompanying drawings, which illustrate some, but not all, embodiments of the invention. The features and aspects presented in this invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Where possible, any term expressed herein in the singular is intended to include the plural as well, and vice versa, unless explicitly stated otherwise. In this invention, the terms “a” or “the” are also intended to include the plural unless the context clearly indicates otherwise. Furthermore, the terms “comprising” or “having” as used in this invention specify the presence of the said element but do not exclude the presence or addition of other elements.

[0033] This invention relates to a converter-based ASR system that, in some applications, can use fewer computational resources and / or improve prediction accuracy compared to known converter-based ASR schemes. For example, implementing the converter-based ASR system and method disclosed in this document may require fewer computational operations and / or less system memory compared to previous schemes.

[0034] In at least some applications, the disclosed transformer-based ASR system and method can improve speech recognition performance by reducing the frame rate used in the subnetworks of the transformer-based ASR system. In exemplary embodiments, this is achieved by combining convolutional subsampling operations performed on the input speech frames with time reduction operations. In some examples, the computational cost for training the encoder and decoder of the transformer-based ASR system and for performing the operations of the trained transformer-based ASR system during inference can be reduced while achieving accurate ASR results. For example, the computational cost can be reduced to nk. 2 The value is k times the frame rate reduction ratio, and n is the number of self-attention layers after the time reduction operation.

[0035] Figure 1This is a block diagram of a converter-based end-to-end ASR system 100 provided in an exemplary embodiment. As used herein, end-to-end can refer to a system that can simultaneously learn learnable neural network parameters for all neural networks and neural network layers from system input to output during system training. As used in this invention, "learning" can refer to operations or values ​​that have been adjusted during the training of the neural network.

[0036] A transducer-based ASR system 100 receives audio input in the form of a sound source sequence X representing spoken speech, and predicts the corresponding sequence (i.e., transcription) Y representing the utterance's pixels, characters, words, or sentence segments. The sound source sequence X comprises a sequence of speech frames, each represented as a corresponding speech frame feature vector: X = [x1, ..., x...]. T (i.e., frame-level acoustic feature sequence). In an exemplary embodiment, each speech frame feature vector x1, ..., x... T It includes a set of elements that embed information representing their corresponding speech frames, with each speech frame corresponding to a time step. In an exemplary embodiment, d att This refers to the number of elements (i.e., dimensions) included in the feature vector of each speech frame. For example, each speech frame feature vector x1, ..., x... T The elements included in the vector can collectively represent the power spectral envelope of their corresponding speech frames. Each speech frame has the same predefined duration (e.g., a 10ms or 15ms time step duration, as a non-limiting example). In an exemplary embodiment, each speech frame feature vector x i (where i is the source frame index, 1≤i≤T) is the log-Mel filter bank eigenvector, including a set of d values ​​called the log-Mel frequency cepstral coefficients (MFCCs). att Each element.

[0037] The number of source speech frames included in the source sequence X is typically much greater than the number of text characters or sentence fragments included in the target sequence Y (i.e., the target transcription). For example, a target sequence Y corresponding to a 15-second utterance could include 150 text characters (e.g., 30 words, with an average of 5 characters per word), which would be predicted based on 1500 speech frames (assuming 10ms per frame). Therefore, frame rate reduction is an important aspect of successfully training a transformer-based ASR system100.

[0038] The transformer-based ASR system 100 includes two neural networks (NNs): an encoder NN 102 and a decoder NN 104. During training, the encoder NN 102 and the decoder NN 104 predict the frame-by-frame posterior distribution of the target sequence Y, i.e., p, given a corresponding sound source sequence X.ctc (Y|X) and p s2s (Y|X).

[0039] The encoder NN 102 comprises multiple layers, where groups of consecutive layers form NN subnetworks that perform task-specific operations. Figure 1 In the encoder NN 102, the task-specific operations implemented by the NN subnetworks include: subsampling operation 106, temporal reduction operation 108, self-attention operation 110, and linear / softmax probability operation 114. The decoder NN 104 also includes multiple layers forming NN subnetworks that perform the corresponding task-specific operations. The task-specific operations implemented by the corresponding NN subnetworks of the decoder NN 102 include: embedding operation 116, source attention + self-attention operation 118, and linear / softmax probability operation 120. These operations will now be explained in more detail below.

[0040] The subsampling operation 106 of encoder NN 102 serves as a preprocessing mechanism for capturing the embeddings of a set of source speech frames using convolution by applying frame stacking and frame skipping techniques (examples of which are shown below). Figure 2 As shown), to reduce the frame rate of the input sequence X. In some cases, combining information from multiple source speech frames can form a transformed frame, which more meaningfully represents units such as phonemes compared to the source speech frames. In an exemplary embodiment, subsampling operation 106 applies a convolution-based method to reduce the frame rate, as described below: Y. Wang et al., Transformer-based Acoustic Modeling for Hybrid Speech Recognition, ICASSP2020, arXiv:1910.09799v2 [cs.CL] April 30, 2020.

[0041] In an exemplary embodiment, the subsampling operation 106 can be represented by equation (1), as follows:

[0042] X0 = EncPre(X) (1)

[0043] In an exemplary embodiment, the EncPre(X) subsampling operation 106 transforms the sound source sequence X into a subsampled sequence using a subnetwork. This subnetwork comprises two convolutional neural network (CNN) blocks 202, each with a stride of 2 and a kernel size of 3. Together, they reduce the frame rate of the output sequence X0 by a factor of 4 (n) compared to the source speech frame rate of X. sub d is the number of frame feature vectors in the output sequence X0.att (This refers to the number of elements in the feature vector of each frame). Each CNN block 202 includes a CNN layer and a ReLU layer, and is configured with a corresponding set of learnable parameters learned during end-to-end training of the transformer-based ASR system 100, the learnable parameters including a weight matrix and a bias vector.

[0044] In an exemplary embodiment, a time reduction operation 108 is included to apply another frame rate reduction technique using a different frame rate reduction technique than that applied by the subsampling operation 106. In an exemplary embodiment, the time reduction operation 108 is used to concatenate adjacent frames in the subsampling sequence X0 to output another time-reduced sequence. Where, n sub2 The output sequence X R The frame length (e.g., the number of feature vectors in the output frame), and n sub2 <n sub .

[0045] Figure 3 An example of time reduction operation 108, represented by equation (2), is shown:

[0046] X R =TR(X0) (2)

[0047] In an exemplary embodiment, the time reduction operation 108 is implemented using at least one linear neural network layer of the encoder NN 102, which applies the learned linear transformation to the sequence. To generate a time-reduced sequence X R =(h1,h2,…,h nsub2 In one example, time reduction operation 108 concatenates two or more frames (e.g., speech feature vectors) from sequence X0 to corresponding frames (e.g., speech feature vectors) based on the learned weight matrix 302. In an exemplary embodiment, the temporal resolution can be reduced by a factor of k, where k is the number of concatenated frames. For example, the temporal resolution can be reduced by k = 2. l The time-reduction sequence X is generated by the time-reduction operation of the subsampled sequence X0. Here, l is the number of layers in the time-reduction operation 108. R It has a reduced length, that is, n sub2 <n sub In an exemplary embodiment, the number of layers l can be a hyperparameter. In an exemplary embodiment where l = 1 and k = 2, when k = 2, an example of the TR of the j-th layer at the output of the i-th time step can be described by the following equation (4):

[0048]

[0049] One or more neural network layers of time reduction operation 108 are configured via corresponding learnable parameter sets learned during end-to-end training of ASR system 100, the learnable parameters including weight matrix 302 and bias vector. In various exemplary embodiments, computation is performed in layers of time reduction operation 108. It is not limited to cascading of consecutive steps. It can be any technique that reduces the time step size in the transformer layer, such as attention, cascading, dense combination, etc.

[0050] As described above, the transformer-based ASR system receives a sequence of frame-level feature vectors as input and maps the sequence of frame-level feature vectors to a sequence of high-level feature vector representations. The decoder generates a transcription sequence (y1, ..., yL) one tag at a time. Each tag yl is conditioned on the high-level feature vector representation and the previously generated tag (y1, ..., yl–1) through an attention mechanism. In this respect, the encoder NN 102 includes another set of layers forming a subnetwork for performing a self-attention operation 110. The self-attention operation 110 includes one or more self-attention mechanisms that time-reduced sequence X R Another sequence transformed into high-level encoded feature vectors As shown in the following equation (5):

[0051] X e =EncBody(X R (5)

[0052] Where e is the number of NN layers included in the self-attention operation 110.

[0053] In an exemplary embodiment, the self-attention operation 110 can be implemented using the techniques described below: S. Karita et al., A COMPARATIVE STUDY ON TRANSFORMER VS RNNIN SPEECH APPLICATIONS, ASRU 2019, arXiv:1909.06317v2[cs.CL], September 28, 2019.

[0054] In this respect, self-attention operation 110 is used to learn sequence information using a self-attention mechanism. The point attention layer can be defined by the following equation (6) in context:

[0055]

[0056] in, and It is the input to the attention layer, d att It is the number of feature dimensions, n q It is Xq The length of n k It is X k The length of q. In an exemplary embodiment, q equals k.

[0057] In this invention, X q X kT This is called the "attention matrix". Input X q This can be considered a query; enter X. k and X v It is a set of key-value pairs.

[0058] The layer represented by equation (6) can be extended to multi-head attention (MHA) for parallel processing of multiple attentions, as shown in equations (7) and (8) below:

[0059]

[0060]

[0061] in, It is the input to the MHA layer. It is the output of the h-th self-attention layer (h = 1, ..., d) head ), It is a learnable weight matrix, d head It is the number of self-attention operations 110 performed by the MHA layer.

[0062] In an illustrative example, the self-attention operation 110(X) e =EncBody(X R It includes e NN layers, and each NN layer can be defined as follows:

[0063]

[0064]

[0065] Where i = 0, ..., e-1 are layer indices, f is the layer norm operation, and FF i For the i-th two-layer feedforward network:

[0066]

[0067] in, The input sequence X R The t-th frame, It is a learnable weight matrix. It is a learnable bias vector. The component MHA in equation (10) i (X i ,X i ,Xi This can be called "self-attention".

[0068] In an exemplary embodiment, the linear / softmax probability operation 114 is performed by a linear layer, followed by a softmax layer. The linear layer applies the learned linear transformation to the high-level sequence X generated by the self-attention operation 110. e In order to make the high-level sequence X e The projection is onto a suitable space for processing by the softmax layer. The softmax layer then applies the softmax function to the output of the linear layer to generate a probability distribution of possible network outputs (e.g., the probabilities of all possible labels). Therefore, the encoder NN102 outputs a frame-by-frame posterior distribution of the target sequence Y (i.e., p...). ctc (Y|X)). In an exemplary embodiment, the linear / softmax probability operation 114 is used to generate connectionist temporal classification (CTC) outputs and scores.

[0069] In an exemplary embodiment, the decoder NN 104 is implemented using the decoder architecture described in the aforementioned document: S. Karita et al., A COMPARATIVE STUDY ONTRANSFORMER VS RNN IN SPEECH APPLICATIONS, ASRU 2019, arXiv:1909.06317v2[cs.CL], September 28, 2019.

[0070] As described above, the task-specific operations implemented by the corresponding NN subnetwork of the decoder NN 102 include: embedding operation 116, source attention + self-attention operation 118, and cross-entropy (CE) operation 120.

[0071] Embedding operation 116 can be represented by equation (12):

[0072] Y0[1:t-1]=DecPre(Y[1:t-1-) (12)

[0073] Where t is the target frame index, and Y[1:t-1- is the context of the target Y[t].

[0074] Embedding operation 116 is used to embed a token into a context sequence Y0[1:t-1] using known embedding techniques. For example, the token can be a word, character, word, or sentence fragment. The subnetwork of the NN layer providing embedding operation 116 is configured by a corresponding set of learnable parameters learned during end-to-end training of the ASR system 100, the learnable parameters including a weight matrix and a bias vector.

[0075] During the training phase of the transformer-based ASR system 100, the input to the decoder NN 104 is a ground truth context sequence Y[1:t-1-] corresponding to the ground truth labels transcribed from the source sequence X. During the post-training inference phase, the decoder input is a target context sequence Y[1:t-1-] and start tags s, which is the output generated by the encoder network 102.

[0076] Source attention operations and self attention operations 118 can be respectively generated by and This means that its output can be represented by Equation 13:

[0077] Y d [t-=DecBody(X e ,Y0[1:t-1]) (13)

[0078] Where d is the number of NN layers included in the source attention + self attention operation 118.

[0079] In the exemplary embodiment, the source attention + self attention operation 118(DecBody(X) e Y0[1:t-1])) is used to encode sequence X e (Generated by self-attention operation 110) and the target context Y[1:t-1] generate the next target frame. Source attention + self-attention operation 118(DecBody(X e Y0[1:t-1])) can be described using two attention modules (self-attention and source attention), which are represented by the following equation:

[0080]

[0081]

[0082]

[0083] Where j = 0, ..., d-1 are the indices of the decoder layers. This is described as encoder-decoder attention. For generating a unidirectional sequence using the decoder, the attention matrices at the t-th target frame are masked so that they are not concatenated with future frames after t.

[0084] The linear CE operation 120 can be represented by equation (17):

[0085] Y post [1:t] = DecPost(Y d [1:t]) (17)

[0086] Linear / softmax probability operation 120(DecPost(Y) d [1:t])) is used to predict the next label prediction Y post The posterior distribution of [1:t]. In an exemplary embodiment, the linear / softmax probability operation 120 is performed by a linear layer, followed by a softmax layer. In an exemplary embodiment, the linear / softmax probability operation 114 is used to generate cross-entropy (CE) outputs and scores.

[0087] Therefore, the decoder NN 104 receives the encoded sequence X of the tag ID. e Given the target context sequence Y[1:t-1]. First, the embedding operation 116 (DecPre(.)) embeds the labels into the learnable vector Y0[1:t-1]. Then, the source attention + self-attention operation (DecBody(.)) and the linear / softmax operation 120 (DecPost(.)) are applied to the target context sequence Y[1:t-1]. e Predicting the next label Y under Y[1:t-1]. post [t- posterior distribution.

[0088] The NN sub-network layers forming the embedding operation 116, the source attention + self attention operation 118, and the probability operation 120 are each configured by a corresponding set of learnable parameters learned during end-to-end training of the ASR system 100, the learnable parameters including weight matrices and bias vectors.

[0089] In an exemplary embodiment, the converter subnetwork of the ASR system 100 (i.e., encoder self-attention operation 110 and decoder source attention + self-attention operation 118) does not include convolutional or recurrent layers. Therefore, the positional information in the input sequence provided to the encoder self-attention operation 110 and decoder source attention + self-attention operation 118 can be represented using sinusoidal position coding, as shown in Equation 18:

[0090]

[0091] Sequences X0 and Y0 are concatenated with (PE[1], PE[2], ...) before being applied to self-attention operation 110 and source attention + self-attention operation 118, respectively. In some exemplary embodiments, subsampling operation 106 may also apply VGG-like convolutional subsampling, in which case positional encoding of sequence X0 is not required.

[0092] During the training of the ASR system, the sequence-to-sequence (S2S) frame-by-frame posterior distribution of the target sequence Y is predicted by the decoder NN 104 (i.e., p...). s2s (Y|X)), and the frame-by-frame posterior distribution of the target sequence Y predicted by the linear / softmax operation 114 of encoder NN 102 (i.e., p) of connectionist temporal classification (CTC). ctc (Y|X)). The training loss based on CTC and CE, computed by loss function operation 122, can be described as:

[0093] L ASR =-αlogp s2s (Y|X)-(1-α)logp ctc (Y|X) (19)

[0094] Here, α is a hyperparameter.

[0095] In at least some exemplary embodiments, the post-training inference phase (also known as the decoding phase) uses beam search to predict the next tag given a speech feature x and a previously predicted tag. This beam search combines scores from S2S (e.g., decoder NN 104 output), CTC (e.g., encoder NN 102 output), and the output of the trained RNN language model (lm).

[0096]

[0097] Among them, y * Let Y be the set of hypotheses for the target sequence Y, and λ and γ be hyperparameters. In these examples, the RNN lm can be implemented using known architectures and training techniques. In some examples, the RNN lm can be omitted.

[0098] In one exemplary embodiment, the overall operation of the ASR system 100 during the training phase can be referenced. Figure 4 Described as:

[0099] Input: Speech frame sequence X and target output label Y

[0100] Output: Predicted output labels

[0101] The following process 400 is performed by encoder NN 102:

[0102] Step 402: Subsampling operation 108 (X0 = EncPre(X)) transforms sequence X into sequence To reduce the frame rate. In addition, when convolutional subsampling is applied in subsampling operation 106, sequence X0 is concatenated with (PE[1-, PE[2-, ...) described in equation (18).

[0103] Step 404: Reduce the time operation 108(X) R =TR(X0)) is applied to sequence X0 to further reduce the sequence length by a factor of k.

[0104] Step 406: Perform self-attention operation 110(X) e =EncBody(X R Sequence X applied to time reduction R e times, to generate another embedded sequence X e .

[0105] Step 408: Apply probability operation 114 to another embedded sequence X e The steps are to compute the frame-by-frame posterior distribution p of the target sequence Y given the source sequence X. ctc (Y|X).

[0106] The following procedure 410 is performed by decoder NN 104:

[0107] Step 412: Embedding operation 116 (Y0[1:t-1] = DecPre(Y[1:t-1])) is applied to the input target sequence Y to embed the previous context of the target sequence Y[1:t-1- labeled with ID into the learnable vector Y0[1:t-1-. Furthermore, the sequence Y0 is concatenated with (PE[1], PE[2], ...) as described in equation (18).

[0108] Step 414: Apply source attention + self-attention operation 118 (Y) d [t] = DecBody(X) e ,Y0[1:t-1])) and probability operation 120(Y post [1:t] = DecPost(Y d [1:t])), so that given X e And Y[1:t-1- predict the next label prediction Y post The posterior distribution p of [t] s2s (Y|X). DecBody(.) was applied d times.

[0109] Step 416: Apply loss function operation 122 to calculate

[0110] L ASR =-αlogp s2s (Y|X)-(1-α)logp ctc (Y|X)

[0111] Step 418: Perform backpropagation to update the ASR system 100 parameters (e.g., weight matrix and bias vector) using gradient descent, with the aim of reducing the loss.

[0112] Repeat steps 402 to 418 until the threshold loss criterion is reached or the maximum number of training iterations defined has been performed.

[0113] After the ASR system 100 has been trained, it can be deployed on one or more devices and used for real-world speech recognition activities. As mentioned above, in some examples, the ASR system 100 can be supplemented with a trained RNN lm network (not shown) and the output label can be predicted using Equation 20. Given speech features x from sequence X and the previously predicted tag, use beam search to predict the next tag. The search beam combines scores from S2S, CTC, and RNN language models.

[0114] Figure 5 Another exemplary embodiment of a converter-based E2E ASR system 500 is shown, which is the same as ASR system 100, except that in the NN encoder 102, the time reduction operation 108 is inserted into the stack of e converter NN layers included in the self-attention operation 110. Figure 5 As shown, the NN layer of the time reduction operation 108 has been divided into a first sub-network and a second sub-network implementing the first self-attention operation 110-1 and the second self-attention operation 110-2, with the time reduction operation 108 located between the first and second self-attention operations. In an exemplary embodiment, the first self-attention operation 110-1 includes e1 layers, and the second self-attention operation 110-2 includes e2 layers, where e = e1 + e2. The NN encoder 102 of the ASR system 500 can be represented as follows:

[0115] X0 = EncPre(X)

[0116] X1 = Enc1Body(X0)

[0117] X R =TR(X1)

[0118] X e =Enc2Body(XR )

[0119] Another difference is the number of layers; Enc1Body(.) and Enc2Body(.) each have the same configuration as described above regarding EncBody(.). In at least some scenarios, applying some self-attention layers before reducing execution time can improve the accuracy of ASR System 500 relative to ASR System 100.

[0120] Figure 6 This is a flowchart of the modified encoder NN process 400A executed by the NN encoder 102 of the ASR system 500. The encoder NN process 400A is the same as the encoder NN process 400A, except that the time reduction operation 108 moves between layers of self-attention operations, as described below:

[0121] Step 402: Subsampling operation 108 (X0 = EncPre(X)) transforms sequence X into sequence To reduce the frame rate. In addition, when convolutional subsampling is applied in subsampling operation 106, sequence X0 is concatenated with (PE[1], PE[2], ...) described in equation (18).

[0122] Step 406A: Apply the first self-attention operation 110-1 (X1 = Enc1Body(X0)) to the frame reduction sequence X0e1 times to generate another embedding sequence X1.

[0123] Step 404: Reduce the time operation 108(X) R =TR(X1)) is applied to sequence X1 to further reduce the sequence length by a factor of k.

[0124] Step 406B: Perform the second self-attention operation 110-2(X) e =Enc2Body(X R Sequence X applied to time reduction R e2 times, to generate another embedded sequence X e .

[0125] Step 408: Apply probability operation 114 to another embedded sequence X e The steps are to compute the frame-by-frame posterior distribution p of the target sequence Y given the source sequence X. ctc (Y|X).

[0126] Despite their growing popularity, previously known transformer-based ASR systems in the field of speech and semantic recognition experience a decrease in computational efficiency if the input sequence of length l is increased by m steps. The exemplary embodiments described herein can, in some cases, alleviate the computational burden on the converter by applying a time-reducing subnetwork. In certain applications, the proposed method can reduce the computational cost of the encoder's self-attention operation by nk. 2 The time step size is multiplied by k, where k is the frame rate reduction and n is the number of transformer layers following the time-shrunk subnetwork layer. The described embodiment combines time steps in the transformer, which can be applied to speech and semantic domains not limited to text classification, and can also be applied to tasks such as neural machine translation, automatic speech recognition, speaker verification, and keyword discovery.

[0127] Figure 7 Another exemplary embodiment of a transducer-based E2E ASR system 700 is shown. This transducer-based E2E ASR system 700 is identical to the ASR system 100, except that the subsampling operation 106 is omitted in the NN encoder 102, and the time reduction operation 108 operates directly on the input speech sequence X (e.g., X...). R =EncPre(X)). In this type of example, additional NN layers can be added to the time reduction operation 108 to provide additional levels of frame cascading.

[0128] Figure 8 Another exemplary embodiment of a transformer-based E2E ASR system 800 is shown, which is identical to ASR system 100 except that subsampling operation 106 is replaced by a reconnaissance neural network 806, and time reduction operation 108 is replaced by an adaptive time reduction operation 808. The reconnaissance neural network 806 (e.g., can be a low-latency end-to-end streaming speech recognition system using a reconnaissance network, as described by C. Wang et al.) https: / / arxiv.org / abs / 2003.10369 2020

[12] The reconnaissance NN 806, as described in the diagram, is used to detect word boundaries with the aid of a forced aligner. The reconnaissance NN 806 determines the start and stop boundaries of words in the speech. An adaptive time reduction operation 808 is used to concatenate a different number of frames relative to the word boundaries output by the reconnaissance NN 806 network. This embodiment may require a separate neural network to apply adaptive time reduction in the ASR system encoder.

[0129] Figure 9This is a block diagram of a simplified exemplary implementation of the computing system 100 described herein. Examples of the invention can be implemented in other computer systems, which may include components different from those discussed below. For example, in some examples, computing system 100 may be an artificial intelligence (AI) chip, an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA) including circuitry for a compressed embedding matrix as described in the examples herein.

[0130] Although Figure 9 A single instance of each component is shown, but multiple instances of each component may exist in the computing system 100.

[0131] Computing system 100 may be a server-side device including one or more servers located in a data center, or a cloud computing system, etc., wherein the embedding matrix may be compressed according to the examples described herein. In some examples, computing system 100 may be a user device, such as a client device / terminal, user equipment (UE), mobile station (STA), smartphone, laptop, computer, tablet, smart device, machine-type communication device, smart (or connected) vehicle, or consumer electronics device, etc. In some examples, computing system 100 may be an edge computing device.

[0132] The computing system 100 may include one or more processing devices 114, such as processors, microprocessors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), dedicated logic circuits, dedicated artificial intelligence processing units, tensor processing units, neural processing units, hardware accelerators, or combinations thereof. The computing system 100 may also include one or more optional input / output (I / O) interfaces 116, which may support connection to one or more optional input devices 118 and / or optional output devices 120.

[0133] In the example shown, one or more input devices 118 (e.g., keyboard, mouse, microphone, touchscreen, and / or keypad) and one or more output devices 120 (e.g., display, speaker, and / or printer) are shown as optional and external to the server. In other exemplary embodiments, no input devices 118 and output devices 120 may be present, in which case I / O interface 116 may not be required.

[0134] The computing system 100 may include one or more network interfaces 122 for wired or wireless communication with other computing devices in the network. The one or more network interfaces 122 may include wired links (e.g., Ethernet cables) and / or wireless links (e.g., one or more antennas) for intra-network and / or inter-network communication.

[0135] The computing system 100 may also include one or more storage units 124, wherein the one or more storage units 124 may include high-capacity storage units such as solid-state drives, hard disk drives, disk drives and / or optical disk drives.

[0136] The computing system 100 may include one or more memories 128, wherein the one or more memories 128 may include volatile or non-volatile memories (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). The one or more non-transitory memories 128 may store instructions executed by one or more processing devices 114, for example, to perform the exemplary embodiments described in this invention. The one or more memories 128 may include other software instructions, such as software instructions for implementing an operating system and other applications / functions.

[0137] In some exemplary embodiments, one or more storage units 124 (or one or more memories 128) may include software instructions 1101 executable by processing device 1114 to implement ASR system 100, 500, 700, or 800. In some exemplary embodiments, or additionally, one or more memories 128 may include software instructions executable by processing device 114 to implement ASR system 100, 500, 700, or 800. In some exemplary embodiments, additionally or otherwise, computing system 100 may execute instructions from external memory (e.g., an external drive connected to a server via wired or wireless communication), or may have executable instructions provided by transient or non-transient computer-readable media. Examples of non-transient computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable storage.

[0138] Certain adjustments and modifications can be made to the described embodiments. Therefore, the embodiments discussed above should be considered illustrative rather than restrictive.

[0139] All published papers identified in this invention are incorporated herein by reference.

Claims

1. A computer-implemented method for automatic speech recognition using a transformer-based automatic speech recognition (ASR) system, characterized in that, include: Obtain a first speech sequence including a first set of speech frame feature vectors, wherein each speech frame feature vector represents a corresponding speech frame at a corresponding time step. The first speech sequence is processed into a second speech sequence using a time reduction operation of an encoder NN, which includes a second set of speech frame feature vectors, each of which is concatenated with information from a corresponding plurality of speech frame feature vectors included in the first set. The second speech sequence includes fewer speech frame feature vectors than the first speech sequence. The time reduction operation is performed using one or more linear NN layers of the encoder NN. Using the self-attention operation of the encoder NN, the second speech sequence is transformed into a third speech sequence including a third set of speech frame feature vectors; The third speech sequence is processed using the probabilistic operations of the encoder NN to predict a first label sequence corresponding to a third set of feature vectors of the speech frames; The third speech sequence is processed using a decoder NN to predict a second label sequence corresponding to a third set of feature vectors of the speech frames.

2. The method according to claim 1, characterized in that, The method includes, during the training phase of the encoder NN and the decoder NN: Calculate the loss function based on the predicted first label sequence and the predicted second label sequence; Backpropagation is performed using gradient descent to update the learnable parameters of the decoder NN and the encoder NN to reduce the loss function.

3. The method according to claim 2, characterized in that, The method also includes, during the inference phase Based on the predicted first label sequence and the predicted second label sequence, the label sequence of the third speech sequence is calculated.

4. The method according to any one of claims 1 to 3, characterized in that, Obtaining the first speech sequence includes: Obtain an input speech sequence including an input set of speech frame feature vectors, wherein each speech frame feature vector represents a corresponding speech frame at a corresponding time step; The input speech sequence is processed into the first speech sequence using the subsampling operation of the encoder NN, wherein the first speech sequence includes fewer speech frame feature vectors than the input speech sequence.

5. The method according to any one of claims 1 to 3, characterized in that, Obtaining the first speech sequence includes: Obtain an initial speech sequence including an initial set of speech frame feature vectors, wherein each speech frame feature vector represents a corresponding speech frame at a corresponding time step. The initial speech sequence is processed into the first speech sequence using another self-attention operation of the encoder NN prior to the time reduction operation.

6. The method according to claim 5, characterized in that, Obtaining the initial speech sequence includes: Obtain an input speech sequence including an input set of speech frame feature vectors, wherein each speech frame feature vector represents a corresponding speech frame at a corresponding time step; The input speech sequence is processed into the initial speech sequence using the subsampling operation of the encoder NN, wherein the initial speech sequence includes fewer speech frame feature vectors than the input speech sequence.

7. The method according to claim 5, characterized in that, The self-attention operation and the other self-attention operation are each executed by the corresponding sub-network of the self-attention layer.

8. The method according to claim 7, characterized in that, The method includes using a corresponding number of self-attention layers for each of the self-attention operation and the other self-attention operation, based on the obtained hyperparameters.

9. An automatic speech recognition computing system, characterized in that, include: Memory that stores executable instructions; A processing device communicating with the memory, the processing device being configured to execute the instructions to enable the computing system to implement an encoder NN and a decoder NN, wherein a time reduction operation, a self-attention operation, and a probabilistic operation are each executed using a corresponding sub-network of the encoder NN, and another self-attention operation and another probabilistic operation are each executed using a corresponding sub-network of the decoder NN: Obtain a first speech sequence including a first set of speech frame feature vectors, wherein each speech frame feature vector represents a corresponding speech frame at a corresponding time step. Using a time reduction operation, the first speech sequence is processed into a second speech sequence comprising a second set of speech frame feature vectors, each of which is concatenated with information from a corresponding plurality of speech frame feature vectors included in the first set, wherein the second speech sequence comprises fewer speech frame feature vectors than the first speech sequence; the time reduction operation is performed using one or more linear neural network layers of the encoder NN; Using a self-attention operation, the second speech sequence is transformed into a third speech sequence that includes a third set of speech frame feature vectors; The third speech sequence is processed using probabilistic operations to predict a first label sequence corresponding to a third set of feature vectors of the speech frames; The third speech sequence is processed using another self-attention operation and another probabilistic operation to predict a second label sequence corresponding to a third set of feature vectors of the speech frames.

10. The system according to claim 9, characterized in that, The processing device is configured to execute the instructions so that the computing system during the training phase of the encoder NN and the decoder NN: Calculate the loss function based on the predicted first label sequence and the predicted second label sequence; Backpropagation is performed using gradient descent to update the learnable parameters of the decoder NN and the encoder NN to reduce the loss function.

11. The system according to claim 9, characterized in that, The processing device is used to execute the instructions so that the computing system, during the inference phase: Based on the predicted first label sequence and the predicted second label sequence, the label sequence of the third speech sequence is calculated.

12. The system according to any one of claims 9 to 11, characterized in that, The processing device is configured to execute the instructions to cause the computing system to obtain the first speech sequence in the following manner: Obtain an input speech sequence including an input set of speech frame feature vectors, wherein each speech frame feature vector represents a corresponding speech frame at a corresponding time step; The input speech sequence is processed into the first speech sequence using a subsampling operation, wherein the first speech sequence includes fewer speech frame feature vectors than the input speech sequence.

13. The system according to any one of claims 9 to 11, characterized in that, The processing device is configured to execute the instructions so that the automatic speech recognition computing system obtains the first speech sequence in the following manner: Obtain an initial speech sequence including an initial set of speech frame feature vectors, wherein each speech frame feature vector represents a corresponding speech frame at a corresponding time step. The initial speech sequence is processed into the first speech sequence using another self-attention operation of the encoder NN prior to the time reduction operation.

14. The system according to claim 13, characterized in that, The processing device is configured to execute the instructions so that the automatic speech recognition computing system obtains the initial speech sequence in the following manner: Obtain an input speech sequence including an input set of speech frame feature vectors, wherein each speech frame feature vector represents a corresponding speech frame at a corresponding time step; The input speech sequence is processed into the initial speech sequence using the subsampling operation of the encoder NN, wherein the initial speech sequence includes fewer speech frame feature vectors than the input speech sequence.

15. The system according to claim 13, characterized in that, The processing device is used to execute the instructions to cause the automatic speech recognition computing system to perform the self-attention operation and the other self-attention operation using the corresponding sub-network of the self-attention layer.

16. The system according to claim 15, characterized in that, The processing device is used to execute the instructions so that the automatic speech recognition computing system uses a corresponding number of self-attention layers for each of the self-attention operation and the other self-attention operation, based on the obtained hyperparameters.

17. A computer-readable medium, characterized in that, The instruction includes instructions that, when executed by a processing device of a computer system, cause the computer system to perform the method according to any one of claims 1 to 8.

18. A computer program product, characterized in that, The instruction includes instructions that, when executed by a processing device of a computer system, cause the computer system to perform the method according to any one of claims 1 to 8.

19. An automatic speech recognition system, characterized in that, include: An encoder NN is used to process a first speech sequence comprising a first set of speech frame feature vectors, each of which represents a corresponding speech frame at a corresponding time step. The encoder NN implements: The time reduction operation transforms the first speech sequence into a second speech sequence comprising a second set of speech frame feature vectors, each of which is concatenated with information from a corresponding plurality of speech frame feature vectors included in the first set. The second speech sequence comprises fewer speech frame feature vectors than the first speech sequence. The time reduction operation is performed using one or more linear neural network layers of the encoder NN. Self-attention operation: using a self-attention mechanism to transform the second speech sequence into a third speech sequence that includes a third set of speech frame feature vectors; Probabilistic operation to predict the first label sequence corresponding to the third set of feature vectors of the speech frames; The decoder NN processes the third speech sequence to predict a second label sequence corresponding to a third set of feature vectors of the speech frames.

Citation Information

Patent Citations

  • CNN (Convolutional Neural Network) code testing method and apparatus

    CN105117330A

  • Very deep convolutional neural networks for end-to-end speech recognition

    US10510004B2