Method and Apparatus for Automatic Speech Recognition

Abstract

Automatic speech recognition (ASR) using a machine learning model comprising: Receiving speech; Extracting features of the speech using a trained ML model by: Inputting to the encoder module a plurali

Description

Field

[001] The present application generally relates to a method and apparatus for automatic speech recognition. In particular, the present application provides a method for using a machine learning, ML, model for automatic speech recognition. Background

[002] Automatic speech recognition (ASR) has greatly benefited from deep learning, reaching unprecedented levels of accuracy and enabling a number of successful products to support real-life use cases. ASR systems have followed the general trend in deep learning and increased steadily both in model size and complexity to push recognition accuracies. Modern industry-scale ASR models often contain hundreds of millions or even billions of neural parameters. Training these models, however, often requires high amount of GPU hours and results in a large carbon footprint. Resource-efficient deployment is also an important factor, especially for on-device scenarios.

[003] At the core of current state-of-the-art ASR systems are Multi-Head Self-Attention (MHSA) cells or blocks. MHSA learns interactions between pairs of frames originating from a speech signal and the interaction is also referred to as token mixing. MHSA has helped existing speech models to reach their current performance. However, MHSA exhibits a quadratic complexity, which slows down inference and training and increases memory consumption for long input speech signals. MHSA is therefore costly, particularly for long input sequences.

[004] The applicant has therefore identified the need for an improved method for automatic speech recognition. Summary

[005] The present Applicant formally generalizes MLP Mixer to a new class of methods, named Summary Mixing, adapted to deal with variable sequences and capturing the global context usually captured by MHSA, but with a linear-complexity. Introducing Summary Mixing in state-of-the-art ASR models makes it feasible to preserve or exceed previous speech recognition performance under various scenarios, while lowering the training and inference times by up to 27% and reducing the memory budget by a factor of two.

[006] In a first approach of the present techniques, there is provided a computer-implemented method for performing automatic speech recognition using a trained machine learning, ML, model, the method comprising: receiving an input audio data item, the input audio data item comprising speech of at least one speaker; extracting, using an encoder module of a trained ML model, features from the input audio data item, by: inputting, into the encoder module, a plurality of initial vectors corresponding to the input audio data item; generating a first set of vectors from each initial vector, the first set of vectors representing local information in the plurality of initial vectors; generating a second set of vectors from each initial vector, the second set of vectors representing global information in the plurality of initial vectors; generating a summary vector summarising information contained within the second set of vectors; concatenating the summary vector with each vector in the first set of vectors, and thereby generating a plurality of feature vectors; and generating a text output for the input audio data item by processing the generated plurality of feature vectors using a decoder module of the trained ML model.

[007] Advantageously, the present techniques provide a linear-time alternative to selfattention that does not rely on pair-wise interactions. Instead, the present techniques summarise a whole utterance (i.e. the speech within the input audio data item) as a mean or average over separate contributions for all time steps. The obtained summary is then fed back to each individual time step. This method is also referred to herein as “SummaryMixing”.

[008] Inputting a plurality of initial vectors corresponding to the input audio data item may comprise inputting a plurality of initial vectors where each initial vector represents features of the speech in a segment of the input audio data item. That is, the input audio data item may be divided into a plurality of segments or frames, and each initial vector is obtained by extracting features from one of those segments / frames. Each segment / frame may be of an appropriate size (in terms of time, e.g. 25ms) to enable automatic speech recognition to work well.

[009] Alternatively, inputting a plurality of initial vectors corresponding to the input audio data item may comprise inputting a plurality of vectors output by another module of the trained ML model. That is, the plurality of vectors output by another module each represent features of the speech in a segment of the input audio data item after each vector has already been processed by another module of the ML model. Thus, in this case, the plurality of vectors may not directly represent features of the speech, as the vectors have been operated on by other modules already. For example, the plurality of initial vectors here may arrive after passing through another (or the same) encoder module. In a typical automatic speech recognition ML model, the extracting step described above may happen multiple times, e.g. 12 or 24 times before the text output can finally be generated.

[010] The method may further comprise repeating the extracting using the encoder module after the concatenating, using the generated plurality of feature vectors. That is, as mentioned above, the whole process to extract features may be repeated multiple times before the final text output can be generated.

[011] Generating each vector in the first set of vectors may comprise multiplying one initial vector of the plurality of initial vectors with a first neural network, and generating each vector in the second set of vectors may comprise multiplying one initial vector of the plurality of initial vectors with a second neural network. Multiplying by the first / second neural network means multiplying each initial vector with a matrix representing weights of the first / second neural network. This multiplying occurs by passing the vectors through the first / second neural networks.

[012] Generating a summary vector may comprise: averaging values within the second set of vectors. Thus, a single summary vector is generated by calculating the average (e.g. the arithmetic mean) of the values of the vectors in the second set of vectors.

[013] Generating a first set of vectors and a second set of vectors from the plurality of initial vectors may comprise generating the first and second set of vectors in parallel. Thus, as noted above, the generating may comprise passing each initial vector through a first and second neural network. Therefore, each initial vector may be passed through the first and second neural networks in parallel / simultaneously, to increase the speed of the processing by the ML model.

[014] The plurality of initial vectors may be of a predefined size, and the concatenating may comprise generating a plurality of feature vectors of the same predefined size. Here, the term “concatenating” means that the dimension of each vector of the first set of vectors is extended by the summary vector. For example, the plurality initial vectors may have a predefined size or dimension X (e.g. 512). Similarly, the dimension of each vector of the first set of vectors may be N (e.g. 512) and the dimension of the summary vector may be M (e.g. 512), such that concatenating the summary vector with each vector in the first set of vectors generates a plurality of feature vectors each having a dimension of N+M (e.g. 1024). The concatenating may comprise downscaling the size / dimension of the generated feature vectors back to the original size / dimension of the vectors of the first set of vectors (e.g. 512). The downscaling may comprise using any suitable neural network to downscale the dimension N+M of the generated feature vectors to match the dimension of the plurality of initial vectors X.

[015] In a second approach of the present techniques, there is provided an apparatus for performing automatic speech recognition using a trained machine learning, ML, model, the apparatus comprising: a microphone for capturing an input audio data item comprising speech of at least one speaker; and at least one processor coupled to memory, arranged for: extracting, using an encoder module of a trained ML model, features from the input audio data item, by: inputting, into the encoder module, a plurality of initial vectors corresponding to the input audio data item; generating a first set of vectors from each initial vector, the first set of vectors representing local information in the plurality of initial vectors; generating a second set of vectors from each initial vector, the second set of vectors representing global information in the plurality of initial vectors; generating a summary vector summarising information contained within the second set of vectors; concatenating the summary vector with each vector in the first set of vectors, and thereby generating a plurality of feature vectors; and generating a text output for the input audio data item by processing the generated plurality of feature vectors using a decoder module of the trained ML model.

[016] The features described above with respect to the first approach apply equally to the second approach and therefore, for the sake of conciseness, are not repeated.

[017] The apparatus may be a constrained-resource device, but which has the minimum hardware capabilities to use a trained neural network / ML model. The apparatus may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge or smart vacuum cleaner). It will be understood that this is a non-exhaustive and nonlimiting list of example apparatuses.

[018] In a related approach of the present techniques, there is provided a computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out any of the methods described herein.

[019] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[020] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[021] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. Code components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs.

[022] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein.

[023] The techniques further provide processor control code to implement the abovedescribed methods, for example on a general purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD-ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system.

[024] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

[025] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method.

[026] The method described above may be wholly or partly performed on an apparatus, i.e. an electronic device, using a machine learning or artificial intelligence model. The model may be processed by an artificial intelligence-dedicated processor designed in a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[027] As mentioned above, the present techniques may be implemented using an Al model. A function associated with Al may be performed through the non-volatile memory, the volatile memory, and the processor. The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Al-dedicated processor such as a neural processing unit (NPU). The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (Al) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning. Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or Al model of a desired characteristic is made. The learning may be performed in a device itself in which Al according to an embodiment is performed, and / o may be implemented through a separate server / system.

[028] The Al model may consist of a plurality of neural network layers. Each layer has a plurality of weight values, and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

[029] The learning algorithm is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Brief description of the drawings

[030] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which:

[031] Figures 1A to 1C are schematic diagrams showing example uses of automatic speech recognition, ASR;

[032] Figure 2 is a schematic diagram of a process performed by a conventional encoder in ASR;

[033] Figure 3 is a schematic diagram of a process performed by an ASR encoder of the present techniques;

[034] Figure 4A is a diagram of a self-attention cell, Figure 4B is a diagram of the present “SummaryMixing” cell, and Figure 4C shows how the cells of Figures 4A or 4B can be used within a Branchformer machine learning model architecture for performing speech processing;

[035] Figure 5 is a flowchart of example steps to perform ASR using the present ASR encoder;

[036] Figure 6 shows results of experiments to test efficiency and VRAM consumption;

[037] Figure 7 is a table showing results of speech recognition experiments;

[038] Figure 8 is a table showing results of speech recognition experiments; and

[039] Figure 9 is a block diagram of an apparatus for performing speech recognition. Detailed description of the drawings

[040] Broadly speaking, embodiments of the present techniques provide a method and apparatus for automatic speech recognition. In particular, the present application provides a method for using a machine learning, ML, model for automatic speech recognition. Advantageously, the present techniques provide an ML model which preserves or exceeds previous speech recognition performance while lowers the training and inference times and reduces the memory requirement.

[041] Recent works have pointed out that under some conditions, expensive pair-wise selfattention operations may behave like linear operations for SOTA speech recognition models. For example, it has been shown that the upper encoder layers in trained Transformer-based ASR models behave like feed-forward layers, which has also been verified for Conformer models. Furthermore, it has been demonstrated that the attention matrices of trained Branchformer models tend to have diagonality scores near 0.5, which indicates that the attention weights are distributed almost uniformly and a simple average could achieve a similar token mixing.

[042] Therefore, the present techniques introduce a linear-time alternative to self-attention not relying on pair-wise interactions. The present techniques summarise a whole utterance as a mean over separate contributions for all time steps. The obtained summary is then fed back to each individual time step. This method is named “SummaryMixing”. The proposed SummaryMixing achieves a training time reduction of up to 28%, exhibits a stable real-time factor compared to a quadratic increase in latency for MHSA with utterance length, halves the memory consumption at training and decoding times in comparison with MHSA, and reaches the performance of SOTA ASR systems on five datasets of different languages and acoustic conditions. The findings have been extended to other speech understanding tasks including spoken language understanding (SLU) and keyword spotting (KWS). The present techniques provide a linear-time method which matches or surpasses the performance of MHSA for speech-processing tasks across various scenarios.

[043] SummaryMixing can be seen as a generalization of a recently proposed method called the HyperMixer, which derives from the MLP Mixer. The MLP Mixer was the first to show that token mixing can also be achieved outside the framework of self-attention. MLP Mixer learns a fixed-size MLP to perform token mixing throughout time and achieves competitive performance across a wide range of domains with linear time complexity. Unfortunately, speech tasks commonly have variable-length sequences and existing extensions of the MLP Mixer to this modality only propose to rely on chunked and fixed-length input vectors and achieve sub-optimal performance. HyperMixer, on the other hand, is an attempt at extending MLP Mixer to varying time dimensions in the input space for natural language processing. The HyperMixer has the same time complexity as the present SummaryMixing method, and a similar general architecture.

[044] Numerous existing efficient attention mechanisms attempt to re-create the original behaviour of self-attention but at a lower cost. For instance, active research directions include low-rank approximation, linearization, or sparsification of self-attention. In the context of ASR, some existing models obtain lower training times and memory consumption than models equipped with the standard self-attention by reducing the length of the sequence attended without decreasing the attention span to avoid ASR performance degradation. Such approaches do not overcome the quadratic time complexity of MHSA as they only change the input length to limit or control the growth in complexity.

[045] An important focus of the existing works is on the capacity to model pair-wise interactions, while the present assumption is that such a property can be discarded for ASR and SLU. Thus, SummaryMixing aims at summarizing a sentence to a global context vector. Also, approximations or sparsification of self-attention usually result in slight degradations in performance. On the contrary, it is observed that SummaryMixing matches or surpasses the performance of self-attention.

[046] Previous work has shown that speech recognition does not necessarily require longdistance fine-grained modelling at the acoustic level to reach state-of-the-art performance. SummaryMixing will now be introduced. Though it can be interpreted as a generalization of the HyperMixer, the relationship is most succinctly expressed after the introduction of SummaryMixing. After this, an explanation of how SummaryMixing can be integrated into a Branchformer for ASR and SLU is provided.

[047] Figures 1A to 1C are schematic diagrams showing example uses of automatic speech recognition, ASR. The present techniques may be used for any of the uses of ASR, such as the automatic transcription of voice messages in messaging applications or voice-controlled interactions with personal assistants (e.g. “Hi Bixby”).

[048] One advantage of the present techniques is that it can be used within any ASR model or system. This is particularly useful as devices with different resources may have different sized models (i.e. of different complexity). The present techniques can be added to any of these models as long as the model has a Transformer architecture.

[049] The present techniques enable faster training and faster inference (i.e. lower latency) for automatic speech recognition systems for Human-Machine voice-based interactions. Typically, any “Hi Bixby do X” type of interaction involves input audio containing speech being processed by a trained Transformer-based model that turns the speech into a textual representation. Those Transformer models exhibit quadratic time complexity. This means that the necessary computational resources, as well as the time required, to compute the textual output grows quadratically with the length of the user speech. This is due to the “self-attention” (SA) block in the Transformer architecture. The present techniques replace the SA block with a new one, called “SummaryMixing”, that reduces the complexity to the linear regime, resulting in much faster training times and lower latencies (hence feedback) for the end user. This also results in lower energy, i.e. battery, consumptions.

[050] Figure 2 is a schematic diagram of a process performed by a conventional encoder in ASR. Step 1 of the process is the input stage. During step 1, the encoder receives an input tensor made of T vectors. The input tensor may contain some speech acoustic features or be the output of a previous neural network block in the ASR model. Step 2 involves performing concurrent and fast operations. Each vector from to to T (hence a total of T vectors) from the input tensor is multiplied by three matrices (the neural parameters of the self-attention layer). This is very quick as it can be computed concurrently and the weight matrices are the same (same values) for all t vectors. Hence, for each t, 3 new vectors are obtained. They often have a dimension different from the original vectors. Step 3 involves performing a self-attention quadratic operation. Here, the T input vectors and the T output vectors have the same size and length. Each vector t composing the output of the self-attention is a product involving all 3 x T new vectors from the previous step. In short, for each t of the output (from 0 to T), the T computations are required. Hence, if T increases, the two quantities also increase proportionally, leading to large amount of compute being spent.

[051] Figure 3 is a schematic diagram of a process performed by an ASR encoder of the present techniques. Step 1 of the process is the input stage. During step 1, the encoder receives an input tensor made of T vectors. The input tensor may contain some speech acoustic features or be the output of a previous neural network block in the ASR model. This is the same as in the conventional encoder (Figure 2).

[052] Step 2 involves performing concurrent and fast operations. Each vector from to to T (hence a total of T vectors) from the input is multiplied by two neural networks (which can be any type(s) of neural network). The weights of the neural networks are different. Hence, for each t, two new vectors are obtained. They often have a dimension different from the original vectors. Step 2 is different to step 2 of the conventional encoder.

[053] It can be seen from Figure 3 that both self-attention and SummaryMixing take a sequence of T vectors as inputs. They are supposed to be the same sequences (in terms of what information they contain). SummaryMixing passes these vectors into two different neural networks (e.g. DNN) while self-attention utilises three linear transformations. Hence, SummaryMixing produces 2 new sequences of T vectors while self-attention produces 3.

[054] Step 3 differs from step 3 of the conventional encoder. Here, step 3 involves a summarization process. One of the two T new vectors is summarized over T. For example, the summarising may involve a simple average, but it may be any operation producing a single summary vector out of T vectors. This operation is linear in complexity. The output is a summary of the whole sequence of vectors taking the form of a single vector (no T anymore). Self-attention models pair-wise correlations between all T of a sequence while the present techniques advantageously model a summary of the interactions of all T at once.

[055] Self-attention performs a computation for each t composing the input T vectors and each t composing the 3 new sequences of T vectors. This leads to a quadratic time complexity. The effect of this is that each t of the output is a mix of all the interactions between all pairs of t composing the original T vectors and the T new vectors.

[056] SummaryMixing does not model these pair-wise interactions as this is what leads to the quadratic complexity and is useless for speech recognition. Instead, it summarizes the global interactions with a single vector. In turn, the T new vectors into a single vector with any summary operator, like an average. This is linear in complexity.

[057] The present techniques comprise an additional step, step 4, which involves concatenation. The summary vector is concatenated with every single t of the T new vectors coming from the Step 2. This means that every t composing the T new vectors now has some local information coming from its initial transformation in Step 2, but also global information coming from the Summarization in Step 3. The summary vector from Step 2 is the same for all t. This new sequence of T vectors is then transformer by a final neural network. The latter can be any type of neural network. This operation is also linear in time and executed in parallel.

[058] Thus, SummaryMixing has an extra step. It combines the summary vector with the transformation of the input vectors. This merging is done via any neural network and is necessary to make sure that global and local information blend properly in the output T vectors before going to the next block.

[059] In other words, the present techniques replace the pair-wise scores computed by selfattention with an average over time of a non-linear transformations of the input vector. The latter static vector resuming the whole sequence is then concatenated with another non-linear transformation of the input vector for merging with a final non-linear transformation. These computations are computed in parallel over time and once for each time step, hence leading to a linear time complexity. This leads to faster and cheaper (memory) training times, faster and cheaper (memory) inferences compared to state-of-the-art tradition self-attention equipped transformer for offline speech recognition while preserving the accuracy. Less compute and less memory requirements also help with reducing the energy consumption, hence extend the battery life.

[060] The present techniques differ from previous methods as the vast majority of ASR systems rely on the quadratic self-attention. All existing other linear time complexity alternatives fail at preserving the speech recognition accuracy.

[061] To reduce the complexity of both branches even further, the present techniques introduce a trick reducing by N times (N being the number of heads) the number of neural parameters within Summary Mixing. As shown in Figure 3, a multi-head Summary Mixing branch will see its input as well as non-linear transformation parameters separated in N different vectors for parallel computations. The output of these N computations is then concatenated to get the original vector size back.

[062] SummaryMixing. Figure 4A shows a self-attention cell. The cell takes an input sequence X e K.TxD = {x0, ...,xT} of T feature vectors xt of length D, and transforms them into hidden representations H e RTx'D' = {h0,..., hT}, that can be inputs for the next layer. The output of the self-attention cell is a weighted average, where the weights are computed for each time step, of “values” (computed with v) for each time step. This description, and the multitude of connections across time steps in Figure 4A highlight the quadratic time in the length of the input that self-attention takes.

[063] To reduce the quadratic time complexity, SummaryMixing is introduced (as illustrated in Figure 4B), which also transforms input vectors xt into hidden representations ht. The key to inducing linear time complexity is to summarise the whole utterance in a single vector s. The input xt is transformed by two functions. One is the local transformation function f : RD -> R°". The other is summary function s: RD -> R°"'. The resulting vectors s(xt) are averaged across all time steps (| S) to form the mean vector s. This single vector is passed back to each time step. This already indicates that compared to self-attention the time complexity is reduced to linear, and this will crystallize in the mathematics. The concatenation of it and the local information / (xt) is then transformed by the combiner function c: RD" + RD'" -> RDf.

[064] Each output vector is the function of one vector capturing the whole sequence and one capturing local information. Computing s takes O(T) time, after which each ht can be computed in constant time w.r.t. T. This compares to O(T2) in standard self-attention.

[065] Let X e RTxD = {x0,..., xT} represent an input sequence of T feature vectors xt of length D, which is to be transformed into hidden representations H e RTxD' = {h0, ...,hT}. Mathematically, the SummaryMixing process, parametrized through learnable functions f, s, c, can be described as: s = | s(xt); ht = c^f^, s), (1) where f: RD Rd" the transformation function; (1a) s: Rd Rd"' the summary function; (1b) c:Rd"+ R0'"-> Rd' the combiner function. (1c) Here, the function f is responsible for capturing the local information, which is a non-linear transformation of each individual input vector. The summary function s, on the other hand, computes a contribution for a global summary, that will later be averaged over time. Lastly, the combiner function c merges the information coming from the local and global transformations together before feeding it into the next block.

[066] The vector s is the global summary containing the averaged information across the entire sequence. The key is that the only interaction between feature vectors happens through s. Computing s takes O(T • D'") time, as opposed to O(T2 • D) as in standard self-attention. In short, each output vector is the function of one vector capturing the whole sequence and one capturing local information.

[067] Relationship to the HyperMixer. The HyperMixer was described in an outwardly different form, relating it to the MLP Mixer, but not showing why it gives linear complexity. It can be considered as a specific choice of functions in SummaryMixing in (1a): s(xt) = f\Xt) x xt, and D' = D, (2) where f: RD -> RD'" is a learnable transformation and x denotes the outer product between two vectors. The combiner function c does not exist in HyperMixer and is reduced to a mere dot product between s and the output of the transformation function f to obtain ht. However, the authors do not motivate such choices and, in fact, there is no particular reason for this other than a historical one where this form is derived from the original MLP Mixer. In the present definition of Summary Mixing, f, s and c can be any learnable function. 5

[068] Branchformer (1) and Conformer (2) reach state-of-the-art speech recognition and understanding accuracy. As shown in Figure 4C, the present techniques replace a selfattention cell in the Branchformer and Conformer techniques with a SummaryMixing cell, and thereby advantgeously reduce the complexity of the computation performed by 10 Branchformer / Conformer from quadratic to linear.

[069] In particular, the transformation ( / ), summary (s), and combiner (c) functions are all implemented as a dense linear layer followed by a GeLLI activation function. The input of the combiner is a concatenation of s and / (xt). The CNN branch of the Branchformer is an MLP 15 with convolutional gating inspired by cgMLP (3). The convolutional gating helps in capturing strong local relations as the kernel size typically is limited to a small number of frames. The CNN modules of the Conformer play a similar role but are computed sequentially with the MHSA blocks instead of in parallel as in the Branchformer. In the case of the Branchformer, the outputs of both branches, CNN and SummaryMixing, are then concatenated and fed to a 20 two-layered MLP followed by GeLU activations before feeding into the next block. For the Conformer, the output of the SummaryMixing block is simply fed to the next convolutional module. No layer normalization is applied within the SummaryMixing cell and a skip connection connects the input to the output after fusion from the last MLP. 25

[070] Branchformer with SummaryMixing-lite. The present techniques also include an additional variation of the Branchformer combined with SummaryMixing. This variation is specific to this architecture and can not be extended to the Conformer. Instead of implementing the self-attention block of Branchformer as a SummaryMixing cell, SummaryMixing can be fully integrated. This uses the “cgMLP” block for the local 3D transformation / , and the “MLP” block for the combiner function c. The summary function s and the sum take the place of the self-attention block. SummaryMixing-lite speeds up training even further and reduces the memory footprint. However, a slight degradation in performance may be encountered as SummaryMixing-lite has lost its per-time-step non-linear transformation, while the cgMLP offers a contextualized representation filtering over multiple 35 frames that may loose some local information.

[071] Figure 5 is a flowchart of example steps to perform ASR using the present ASR encoder. The method for performing automatic speech recognition uses a trained machine learning, ML, model, and comprises: receiving an input audio data item, the input audio data item comprising speech of at least one speaker (step S100); extracting, using an encoder module of a trained ML model, features from the input audio data item, by: inputting, into the encoder module, a plurality of initial vectors corresponding to the input audio data item (step S102); generating a first set of vectors from each initial vector, the first set of vectors representing local information in the plurality of initial vectors (step S104); generating a second set of vectors from each initial vector, the second set of vectors representing global information in the plurality of initial vectors (step S106); generating a summary vector summarising information contained within one of the first set of vectors and the second set of vectors (step S108); concatenating the summary vector with the other of the first set of vectors and second set of vectors, and thereby generating a plurality of feature vectors (step S110); and generating a text output for the input audio data item by processing the generated plurality of feature vectors using a decoder module of the trained ML model (step S112).

[072] Experiments: SummaryMixing is evaluated in a two-step process based on different experimental setups. First, empirical efficiency gains in terms of training speed, memory consumption as well as real-time decoding factor (RTF) are highlighted in controlled tasks. Then, the compared models are scaled to standard automatic speech recognition (ASR) and spoken language understanding (SLU) evaluations with common datasets and architectures.

[073] Baseline architectures. All ASR and SLU models, except those stated otherwise, are based on the encoder-decoder architecture (4). For ASR, decoders are either following the joint CTC / Attention (5) training scheme or CTC (6). The attentional decoder always is a standard Transformer. The CTC decoder is a simple linear layer of the size of the output vocabulary which varies depending on the dataset and the task. For classification tasks, such as keyword spotting, no decoder is used. The output representations of the encoder are averaged over time and then classified by a dense neural network whose output dimension is equal to the number of classes. Multi-head self-attention is implemented in SpeechBrain and based on either relative positional encoding with a custom PyTorch implementation or regular MHSA based on the official PyTorch implementation (i.e. CuDNN). The choice of the method is based on empirical results based on the task. Models are compared by varying the encoder architecture based on the literature.

[074] More precisely, a standard Transformer architecture is considered as it remains simple to comprehend but also achieves competitive performance. Then, the Conformer from Gulati et al. is considered as it currently is employed in most deployed ASR systems from real-world products. The Branchformer is included as the main baseline as it is an improvement over the Conformer. A Branchformer equipped with the FastFormer is considered (7) attention mechanism as a baseline for a SOTA linear complexity self-attention. Indeed, Fastformer has been shown to outperform the best linear alternatives to self-attention in various tasks. A Branchformer equipped with HyperMixer attention is also introduced (8) to highlight the performance improvements compared to this token mixing technique. Two CNN-only models are integrated with ContextNet (9) being considered for the competitive baseline. ContextNet remains, to this day, the best-performing CNN-only architecture for ASR according to the original results. It is worth emphasizing that no open-source implementation of ContextNet is able to reproduce the originally reported results. Finally, and to serve as a low bar baseline, a single branch Branchformer is proposed by removing the MHSA part, hence leading to a CNN-only Branchformer. This will be particularly useful to quantify the impact of the global context obtained from MHSA or other alternatives.

[075] Implementation details. ASR and SLU systems have been implemented within the widely used and open-source SpeechBrain toolkit, version 0.5.14. Experiments are conducted following officially available and open-source recipes with only the architecture of the models being changed. The latter allows for a fair comparison as the training procedure and environment are strictly equivalent, well adopted within the speech community, and therefore directly comparable and replicable.

[076] Efficiency task details. Models are comprised of an encoder, described thereafter for each candidate, and a simple linear layer for a decoder trained with CTC. No autoregressive decoder is used to avoid any side effects induced by the use of random input vectors and labels. Systems are benchmarked both in terms of measured training time, i.e., a combination of the time necessary to execute the forward and backward pass as well as the neural parameters update, and peak VRAM consumption across the entire task. In practice, five thousand sequences of random tensors corresponding to a signal of length L with 1 <L <100 in seconds and sampled at 16 kHz are generated and used as inputs, while one hundred random tokens corresponding to indices in a vocabulary of size 1,000 are used as targets. The final training time is an average of the 5,000 sentences and is expressed in seconds. Measurements are extracted on an isolated compute node with four Tesla A100 80GB. Bfloat16 mixed precision is enabled to replicate the real training conditions.

[077] Real-time factor task details. Real Time Factor (RTF) measurements are obtained by dividing the time taken to decode an utterance by its duration. Hence, all models pre-trained on Librispeech (10) are compared on the observed RTF by varying the length of the utterances 5 to decode. In particular, six sets of 2,500 sentences of duration 10,20,30,40,50,60 seconds were constructed. All sentences in a set have the exact same duration from the test-clean of the Librispeech dataset. Hence, they are either cropped to the corresponding duration or concatenated to reach longer lengths (typically 30,40,60 seconds). Models are compared on the exact same sets of sentences to avoid any variation. Measurements are obtained from a 10 batched greedy CTC decoding. The batch size is set to 16 to avoid exceeding the available amount of VRAM for the Branchformer model. Measurements are from an isolated node with a Tesla A100 80GB.

[078] Model architectures. The selected models for these experiments are the 15 Branchformer equipped with MHSA, FastFormer linear attention, SummaryMixing and SummaryMixing-lite as well as ContextNet. For the efficiency analysis, the number of neural parameters is set to roughly 80M for the SummaryMixing Branchformer and vanilla Branchformer, and 65M for the SummaryMixing-lite Branchformer and ContextNet. This corresponds to architectures reaching state-of-the-art WER on Librispeech without the 20 Transformer decoder. The SummaryMixing-lite Branchformer and ContextNet networks are slightly smaller since they replace attention with a simple linear projection. The Branchformers share the same internal dimension between blocks, 512, as well as the number of encoder blocks, 18. In practice, all parameters between both sets of models are identical, except for the attention branch. ContextNet definition follows the architecture description described in 25 the original work described in Reference (9) corresponding to a = 1.5 and leading to 65M parameters. For the RTF analysis, models are pre-trained from the Librispeech experiments.

[079] Figure 6 shows results of experiments to test efficiency and VRAM consumption. The left- and right-most curves represent the average time as well as the peak VRAM consumption 3D to process a sequence of various lengths during training. The curve in the middle shows the real-time factor for trained ASR systems. The standard Branchformer equipped with MHSA exhibits a quadratic increase in all scenarios while the rest of the models are linear. SummaryMixing-lite is the fastest and cheapest alternative. 35

[080] It is clear from the training time, RTF as well as peak VRAM consumption that the MHSA-equipped Branchformer leads to a quadratic increase in required resources. For instance, with speech utterances of duration 100 seconds, a 2.5 times longer training time is seen compared to the SummaryMixing Branchformer and an explosion of the VRAM consumption from 11.6 GB for the SummaryMixing Branchformer to 52 GB for the MHSA Branchformer. The latter phenomenon is particularly critical as it drastically impacts the price and availability of the required hardware. This is also true for the RTF as the MHSA Branchformer takes twice the amount of time necessary for the SummaryMixing Branchformer to transcribe 60 seconds of speech. This is particularly critical in applications with latency constraints. In short, ASR decoding with a SummaryMixing Branchformer will only be memory-bounded compared to the memory and latency boundaries for an MHSA Branchformer. Indeed, the RTF latency of the SummaryMixing Branchformer does not increase with the length of the audio signal. Also, the memory bound will be rapidly reached by the MHSA as it is quadratic compared to SummaryMixing which is linear, as shown by Figure 6.

[081] However, typical speech utterances from the training sets of academic benchmarks for speech recognition barely reach 30 seconds. Below this mark, the MHSA-equipped Branchformer seems to only underperform at the VRAM level as the difference in training time is reduced compared to the other systems. Yet, it remains slightly slower than all the alternatives except for the one-second-long sentences where only the SummaryMixing-lite Branchformer is faster. The backward pass offers the largest gains as, with a 100-second long utterance, the MHSA Branchformer is 3.1 times slower than its SummaryMixing-lite equivalent while, for the same sequence size, the delta for the inference time is reduced to a 1.4 times faster inference for the SummaryMixing-lite Branchformer. Overall, the introduced SummaryMixing appears to be much cheaper and faster than the MHSA-equipped Branchformer as it eliminates the quadratic complexity. The latter efficiency gains, however, must be validated with real speech recognition and understanding accuracies. Indeed, most efficient alternatives to MHSA for ASR fail to reach the same level of performance.

[082] Speech recognition tasks details. Speech recognition with an encoder-decoder architecture based on CTC (6) combined with Transformer decoding is performed (4). Additional experiments with CTC-only training and decoding are also performed to remove entirely self-attention from the ASR architecture. ASR is conducted on five datasets of different languages and complexities in terms of acoustic conditions and available training data: LibriSpeech (960 hours of clean and read English speech) (10), CommonVoice (version 13.0, clean and noisy read speech) (11) Italian (300 hours), Dutch (40 hours), and French (730 hours) as well as AISHELL-1 (170 hours of clean and read Mandarin speech) (12) and Ted-Lium 2 (207 hours of recorded Ted talks in English) (13). Evaluations are conducted on the official sets of each dataset. On Librispeech, models are evaluated without a language model on the dev-clean set, and compared with a transformer language model shallow fusion on the test-clean and test-other. No language models are used for the other datasets. All baselines have been retrained following the official recipes of SpeechBrain (version 0.5.14) to enable a controlled comparison.

[083] Speech understanding tasks details. Models are compared across two tasks of speech understanding with the SLURP dataset from (14) for scenarios, actions, and entity classification and the Google speech commands dataset for keyword spotting. SLURP contains around 58 hours of speech material made of 100 participants simulating interactions with a house robot assistant. Participants were asked to voluntarily move in the room while talking, or not to face the microphone to mimic real-world interactions. Google speech commands offers short sentences and labels corresponding to a vocabulary of 35 commands. All experiments strictly follow the SpeechBrain recipes (version 0.5.14). All models are trained with a single Nvidia RTX 3090.

[084] Model architectures. All baselines are selected for ASR with Librispeech as an initial performance and training cost analysis. Then, a reduced subset of the baselines including the Branchformer with MHSA, SummaryMixing, SummaryMixing-lite, and FastFormer is used across the seven other datasets for further comparison. This is done to lower the number of unnecessary training runs. The original recipe of SpeechBrain is used to start from a state-of-the-art Branchformer and Conformer, leading to model sizes of roughly 110M parameters for all considered methods except the SummaryMixing Conformer (103M), the FastFormer Branchformer (101M), the ContextNet (100M), the SummaryMixing-lite Branchformer (96M) and the CNN-only Branchformer (77M). Encoders are set up comparably to the efficiency analysis with an internal dimension of 512 with 18 blocks. All models share the same decoder architecture with a dense linear for the CTC predictions as well as a standard Transformer decoder with MHSA and 6 blocks. The vocabulary contains one thousand tokens obtained with BPE. The Transformer language model is pre-trained and obtained from the SpeechBrain toolkit. For the remaining ASR and SLU tasks, and following the reduced set of baselines selected from Librispeech results, another set of experiments is conducted with smaller architectures for a sensitivity analysis. A specific parametrization of the models is conducted leading to 21M parameters for all of them except for SummaryMixing-lite Branchformer (18.5M).

[085] Figure 7 is a table showing results of speech recognition experiments. Speech recognition results are shown for encoder-decoder models with CTC plus Transformer decoding on the Librispeech dataset. “Summary Decoder” adds a SummaryMixing decoder to the SummaryMixing encoder. “Dev-clean” word error rates (WER) are obtained without a language model while “Test-clean” and “Test-other” use shallow fusion and a Transformer language model. “GPU Hours” represent the total training time obtained by multiplying the real-time by the number of GPUs. “VRAM” reports the peak amount of VRAM over the four GPUs during training. Lower is better.

[086] All models, including the CNN-only alternatives, achieve competitive recognition rates. For instance, the CNN-only Branchformer achieved a WER of 3.1% on the dev-clean set, even beating the standard Transformer with 3.3% WER. This finding supports the evidence that MHSA may not be necessary for the encoder of speech recognizer systems to achieve good accuracies. It is interesting to notice, however, that using MHSA to incorporate the global context slightly improves the overall word error rates while strongly impacting the needed resources. In fact, the 0.2%, 0.2%, and 0.6% improvements on the dev-clean, test-clean and test-other sets respectively of the MHSA Branchformer compared to the CNN-only Branchformer is done at the expense of 49 hours of compute, representing an increase of 58% in training time. The necessary VRAM also goes from 22 GB to 45 GB for the CNN-only and MHSA versions respectively (i.e. increase of 105%).

[087] SummaryMixing Branchformers and Conformer reduce this disproportionate resource impact while preserving or improving the performance. Indeed, the SummaryMixing-lite Branchformer reduces the absolute performance gap to 0.1%, 0.0% and 0.1% on the dev-clean, test-clean and test-other compared to M HSA while only adding 15 hours of training time compared to the Branchformer CNN-only. This is also true for the VRAM consumption which is halved compared to MHSA. Moreover, the SummaryMixing Branchformer closes the gap with MHSA by achieving strictly the same performance with a reduction of the peak VRAM from 45 GB to 26 GB. ContextNet, despite achieving respectable performance, is not at the level initially reported in Ref. (9). However, there exists no replication of such results. Finally, the SummaryMixing Conformer also beats the standard Conformer with MHSA and reaches the best test-clean and test-other\NER among all the models while halving the required VRAM from 46 GB for the MHSA variant to 21 GB and exhibiting a 28% faster training time.

[088] Removing self-attention from the ASR attentional decoder. In practice, it is empirically found that replacing cross-attention in the decoder leads to unacceptable performance degradation. This is explained by the fact that cross-attention is critical in capturing the alignment between the text and the audio. However, keeping cross-attention and replacing the remaining self-attention with SummaryMixing leads to competitive performance as shown by the last row of Table 1. Efficiency-wise, the gains are not as impressive as with the encoder part since the sequence length of the decoded text is much shorter than the input audio. The performance of the SummaryMixing Branchformer encoder is validated without any attention using CTC-only decoding after CTC plus attention training, which may be common deployment or evaluation cases to reduce the decoding latency. Without any language model, the Branchformer obtained 2.6% and 6.2% of WER on the test-clean and test-other sets compared to 2.5% and 6.4% for the SummaryMixing Branchformer. Such numbers are comparable to the latest SOTA systems and even beat well-established dual-decoding models.

[089] Removing the attentional decoder. Finally, CTC-only training was performed on the two most promising architectures from Librispeech: Conformer with MHSA and SummaryMixing. In this scenario, the decoder is not able to recover from the lack of selfattention in the encoder part. With CTC greedy decoding, on the dev-clean, test-clean, and test-other, the Conformer with MHSA (28.8M) achieves 3.5%, 3.7%, 9.2% WERs respectively while the SummaryMixing-Conformer (26.5M) reaches 3.5%, 3.7%, and 9.4% WERs. Even without MHSA in the architecture, SummaryMixing performs on par with MHSA.

[090] Extended speech recognition and understanding experiments. Figure 8 is a table showing results of extended speech recognition and understanding experiments. The table shows a summary of the speech recognition, keyword spotting and speech understanding results. ASR accuracy is expressed in word error rate (WER, lower is better) for the test sets of CommonVoice Dutch (“Nl.”, 40 hours), Italian (“It.”, 300 hours), French (“Fr.”, 730 hours), AISHELL-1 denoted as “Al.” (170 hours) and Ted-Lium 2 denoted as “Ted.” (207 hours). No language model is applied. SLU results are expressed in SLU-F1 for SLURP and accuracy for Google Speech Command (GSC). This set of experiments focusses on the Branchformer with MHSA or FastFormer versus SummaryMixing Branchformer comparison by extending the number of languages and acoustic conditions for ASR. Two more tasks including keyword spotting and SLU are introduced. The Branchformer was selected instead of the conformer as it obtained better performance on average on the Librispeech experiment and in related work. For ASR, all datasets are much harder than LibriSpeech, due to important variations in the acoustic conditions but also in the amount of available data.

[091] Focusing on ASR, it is worth noting that the SummaryMixing Branchformers outperform the MHSA Branchformer in terms of WER with all datasets, and hence in all data regimes. Indeed, on average over all the model sizes and datasets, the SummaryMixing and SummaryMixing-lite Branchformers reduced the WER by 0.5% and 0.2% absolute compared to the MHSA Branchformer. Such results validate the hypothesis that MHSA is not strictly necessary to achieve state-of-the-art speech recognition performance even across a wide range of acoustic conditions and languages. The previous conclusion also extends to other tasks as the SummaryMixing Branchformers are able to reach their MHSA counterpart on spoken language understanding with an SLU-F1 score of 0.773 and 0.771 for the SummaryMixing and SummaryMixing-lite compared to 0.771 for MHSA. For keyword spotting, SummaryMixing-lite reaches strictly the same accuracy as MHSA, while SummaryMixing improves it by 0.1% absolute. These results indicate that MHSA may also not be strictly necessary for other speech-related tasks and that it can be replaced with SummaryMixing to save training time, energy, and memory without sacrificing on performance.

[092] Figure 9 is a block diagram of an apparatus 100 for performing speech recognition using a machine learning, ML, model 106. The apparatus comprises: at least one processor 102 coupled to memory 104. The at least one processor 102 may comprise one or more of: a microprocessor, a microcontroller, and an integrated circuit. The memory 104 may comprise volatile memory, such as random access memory (RAM), for use as temporary memory, and / or non-volatile memory such as Flash, read only memory (ROM), or electrically erasable programmable ROM (EEPROM), for storing data, programs, or instructions, for example.

[093] The at least one processor 102 may be arranged for: extracting, using an encoder module of a trained ML model, features from the input audio data item, by: inputting, into the encoder module, a plurality of initial vectors corresponding to the input audio data item; generating a first set of vectors from each initial vector, the first set of vectors representing local information in the plurality of initial vectors; generating a second set of vectors from each initial vector, the second set of vectors representing global information in the plurality of initial vectors; generating a summary vector summarising information contained within one of the first set of vectors and the second set of vectors; concatenating the summary vector with the other of the first set of vectors and second set of vectors, and thereby generating a plurality of feature vectors; and generating a text output for the input audio data item by processing the generated plurality of feature vectors using a decoder module of the trained ML model.

[094] The apparatus 100 may comprise one or more interfaces 108, for receiving data items for processing by the ML model 106. For example, the interfaces 108 may comprise a microphone or audio capture device for obtaining speech data items. The interfaces 108 may comprise a display for displaying text that results when the ML model 106 processes the speech data items and outputs corresponding text.

[095] References: (1) Yifan Peng et al. Branchformer: Parallel MLP-attention architectures to capture local and global context for speech recognition and understanding. In International Conference on Machine Learning, pp. 17627-17643. PMLR, 2022. (2) Anmol Gulati et al. Conformer: Convolution-augmented transformer for speech recognition. Proc. Interspeech 2020, pp. 5036-5040, 2020. (3) Sakuma et al.: M LP-based architecture with variable length input for automatic speech recognition. OpenReview, 2021. (4) Karita et al. A comparative study on transformer vs RNN in speech applications. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 449-456. IEEE, 2019. (5) Kim et al. Joint ctc-attention based end-to-end speech recognition using multi-task learning. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4835-4839. IEEE, 2017. (6) Alex Graves. Connectionist temporal classification. Supervised sequence labelling with recurrent neural networks, pp. 61-93, 2012. (7) Wu et al. (2021) Fastformer: Additive attention can be all you need. arXiv preprint arXiv:2108.09084, 2021. (8) Mai et al. HyperMixer: An MLP-based low cost alternative to transformers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15632-15654, Toronto, Canada, July 2023. (9) Han et al. ContextNet: Improving convolutional neural networks for automatic speech recognition with global context. Proc. Interspeech 2020, pp. 3610-3614, 2020. (10) Vassil Panayotov et al. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210. IEEE, 2015. (11) Rosana Ardila et al. Common voice: A massively-multilingual speech corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 4218^4222, 2020. (12) Hui Buet al. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In 2017 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech I / O systems and assessment (O-COCOSDA), pp. 1-5. IEEE, 2017. 5 (13) Anthony Rousseau, Paul Deleglise, Yannick Esteve, et al. Enhancing the ted-lium corpus with selected data for language modeling and more ted talks. In LREC, pp. 3935-3939, 2014. (14) Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. Slurp: A spoken language understanding resource package. In Proceedings of the 2020 Conference 10 on Empirical Methods in Natural Language Processing (EMNLP), pp. 7252-7262, 2020

[096] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and 15 methods disclosed in this description of the preferred embodiment. Those skilled in the art will recognise that present techniques have a broad range of applications, and that the embodiments may take a wide range of modifications without departing from any inventive concept as defined in the appended claims.

Claims

1. A computer-implemented method for performing automatic speech recognition using a trained machine learning, ML, model, the method comprising:receiving an input audio data item, the input audio data item comprising speech of at least one speaker;extracting, using an encoder module of a trained ML model, features from the input audio data item, by:inputting, into the encoder module, a plurality of initial vectors corresponding to the input audio data item;generating a first set of vectors from each initial vector, the first set of vectors representing local information in the plurality of initial vectors;generating a second set of vectors from each initial vector, the second set of vectors representing global information in the plurality of initial vectors;generating a summary vector summarising information contained within the second set of vectors;concatenating the summary vector with each vector in the first set of vectors, and thereby generating a plurality of feature vectors; andgenerating a text output for the input audio data item by processing the generated plurality of feature vectors using a decoder module of the trained ML model.

2. The method as claimed in claim 1 wherein inputting a plurality of initial vectors corresponding to the input audio data item comprises inputting a plurality of initial vectors where each initial vector represents features of the speech in a segment of the input audio data item.

3. The method as claimed in claim 1 wherein inputting a plurality of initial vectors corresponding to the input audio data item comprises inputting a plurality of vectors representing features of the speech in a segment of the input audio data item after processing by another module of the trained ML model.

4. The method as claimed in claim 1, 2 or 3 further comprising repeating the extracting using the encoder module after the concatenating, using the generated plurality of feature vectors.

5. The method as claimed in any preceding claim wherein:generating each vector in the first set of vectors comprises multiplying one initial vector of the plurality of initial vectors with a first neural network; andgenerating each vector in the second set of vectors comprises multiplying one initial vector of the plurality of initial vectors with a second neural network.

6. The method as claimed in any preceding claim wherein generating a summary vector comprises:averaging values within the second set of vectors.

7. The method as claimed in any preceding claim wherein generating a first set of vectors and a second set of vectors from the plurality of initial vectors comprises generating the first and second set of vectors in parallel.

8. The method as claimed in any preceding claim wherein the plurality of initial vectors are of a predefined size, and wherein the concatenating comprises generating a plurality of feature vectors of the predefined size.

9. A computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out the method of claims 1 to 8.

10. An apparatus for performing automatic speech recognition using a trained machine learning, ML, model, the apparatus comprising:a microphone for capturing an input audio data item comprising speech of at least one speaker; andat least one processor coupled to memory, arranged for:extracting, using an encoder module of a trained ML model, features from the input audio data item, by:inputting, into the encoder module, a plurality of initial vectors corresponding to the input audio data item;generating a first set of vectors from each initial vector, the first set of vectors representing local information in the plurality of initial vectors;generating a second set of vectors from each initial vector, the second set of vectors representing global information in the plurality of initial vectors;generating a summary vector summarising information contained within the second set of vectors;concatenating the summary vector with each vector in the first set ofvectors, and thereby generating a plurality of feature vectors; andgenerating a text output for the input audio data item by processing the generated plurality of feature vectors using a decoder module of the trained ML model.

Images