AUTOMATIC END-TO-END LANGUAGE RECOGNITION SYSTEM FOR CONVERSATIONAL, COMMAND AND CONTROL LANGUAGE

Description

BACKGROUND OF THE DISCLOSURE1. Field of the Disclosure

[0001] The present disclosure relates to systems and methods for automatic speech recognition and relates more particularly to end-to-end automatic speech recognition system for both conversational and Command-and-Control speech.2. Description of the Related Art

[0002] Traditional automatic speech recognizers (ASRs) handle an explicit Language Model (LM) that can be manipulated to incorporate arbitrary words or multi-words. In the case both conversational speech and command & control (C&C) speech are sought to be handled, one approach to minimize degradation of overall recognition performance involves utilizing two different ASRs which are run in parallel, one for the conversational speech and one for the C&C speech. However, this parallel approach increases complexity and requires additional logic for switching between the two ASR systems.

[0003] End-to-end (E2E) automatic speech recognizers (ASRs), in which the acoustic, pronunciation and language models are all contained in a single neural network model, have been proven to clearly outperform traditional or hybrid ASRs, especially when trained with large amounts of training data. However, since E2E ASRs are purely based on Deep Neural Networks (DNNs), it remains unclear how to efficiently adapt the E2E ASRs to properly handle newer sets of words or sequences of words, e.g., commands defined by the user in a command & control (C&C) scenario. Furthermore, in the case the ASR is tasked with handling both conversational speech and C&C speech, it is crucial that the ASR's ability to properly handle one type of speech is not reduced in favor of the other.

[0004] One implementation of E2E ASR involves the use of external LMs, thereby achieving customization advantages. However, external LMs have several disadvantages, including: i) requiring additional overhead in the recognition process; ii) being prone to influence the overall recognition (including conversational speech), which may result in degradation of recognition performance for the conversational speech; and iii) requirement to train the external LMs, which is not trivial.

[0005] Another implementation of the E2E ASR involves using an additional module for biasing (using a bias list) the recognition of a set of words or multi-words, but this type of implementation of the E2E ASR is not proven to be effective or designed for handling the combined conversational and C&C speech scenario. In order to handle a considerable amount of entries in the bias list, a word prefix mask is applied to restrict the number of prefixes that the ASR actually attends to at each moment of the transcription, i.e., the masking implements a predefined rule-based approach. Therefore, there is a need for improved E2E ASR system and method to achieve robust ASR accuracy in the case of handling combined conversational and C&C speech scenario, without incurring the drawbacks of the conventional approaches.

[0006] US20200357387A1 describes a method including receiving audio data encoding an utterance and obtaining a set of bias phrases corresponding to a context of the utterance. Each bias phrase includes one or more words. The method also includes processing, using a speech recognition model, acoustic features derived from the audio to generate an output from the speech recognition model. The speech recognition model includes a first encoder configured to receive the acoustic features, a first attention module, a bias encoder configured to receive data indicating the obtained set of bias phrases, a bias encoder, and a decoder configured to determine likelihoods of sequences of speech elements based on output of the first attention module and output of the bias attention module. The method also includes determining a transcript for the utterance based on the likelihoods of sequences of speech elements.

[0007] Chang et al, "Context-Aware Transformer Transducer for Speech Recognition", 2021 IEEE AUTOMATIC SPEECH RECOGNITION AND UNDERSTANDING WORKSHOP (ASRU), 13 December 2021, DOI: 10.1109 / ASRU51503.2021.9687895 describes end-to-end (E2E) automatic speech recognition (ASR) systems often have difficulty recognizing uncommon words, that appear infrequently in the training data. One method, to improve the recognition accuracy on such rare words, is to latch onto personalized / contextual information at inference. There is provided a context-aware transformer transducer (CATT) network that improves the state-of-the-art transformer-based ASR system by taking advantage of such contextual signals. Specifically, a multi-head attention-based context-biasing network is provided, which is jointly trained with the rest of the ASR sub-networks. Techniques to encode contextual data and to create the final attention context vectors are provided. BLSTM and pretrained BERT based models are leveraged to encode contextual data and guide the network training.

[0008] US20200357388A1 describes a method including receiving audio data encoding an utterance, processing, using a speech recognition model, the audio data to generate speech recognition scores for speech elements, and determining context scores for the speech elements based on context data indicating a context for the utterance. The method also includes executing, using the speech recognition scores and the context scores, a beam search decoding process to determine one or more candidate transcriptions for the utterance. The method also includes selecting a transcription for the utterance from the one or more candidate transcriptions.SUMMARY OF THE DISCLOSURE

[0009] The invention is set out in the appended set of claims.

[0010] According to an example embodiment of the present disclosure, an E2E ASR system based on DNN architecture incorporates at least one additional module that is seamlessly integrated in the DNN architecture to enable robust recognition of both conventional speech and C&C speech, which additional module can handle an external list of word commands or multi-word commands that serves to boost recognition of the C&C speech, without disturbing the recognition of conversational speech.

[0011] According to an example embodiment of the present disclosure, the additional module(s) incorporated in the E2E ASR system is configured to handle arbitrarily-defined entries in the external list, without requiring the E2E ASR to be trained beforehand with the arbitrarily-defined entries, thus allowing broad flexibility and immediate usability. This advantageous flexibility is enabled i) in part because "token" units that the E2E ASR handles are at the sub-word level, so any word can be composed with such inventory, and ii) in part because the ASR is capable of producing a meaningful representation of any such sequence for its own usage in the transcription process.

[0012] As used in the present disclosure, a "token" (also known as a terminal symbol) is the part of a grammar that defines words or other entities that may be spoken. For speech recognition, a token is typically an orthographic entity of the language being recognized, but a token may be any string that the speech recognizer can convert to a phonetic representation.

[0013] According to an example embodiment of the present disclosure, the additional module(s) incorporated in the E2E ASR system enables accurate speech recognition in situations involving i) conversational speech (e.g., a dialogue or a monologue) generated by one or more users, and ii) verbal commands delivered by the one or more users to a virtual assistant amidst the ongoing conversational speech.

[0014] According to an example embodiment of the present disclosure, the additional module incorporated in the E2E ASR system is configured to handle arbitrarily-defined entries in the external list, which list can include names and C&C words.

[0015] According to an example embodiment of the present disclosure, the E2E ASR system is configured using the Neural Transducer (NT) architecture.

[0016] According to an example embodiment of the present disclosure, the E2E ASR system is configured using the Listen, Attend, and Spell (LAS)-type architecture.

[0017] According to an example embodiment of the present disclosure, the additional module(s) incorporated in the E2E ASR system can include a bias encoder, a label encoder, a first attention mechanism, a second attention mechanism, and an additional attention mechanism.

[0018] According to an example embodiment of the present disclosure, prefix penalties are provided for restricting the first attention mechanism to only those entries that fit the current transcription context.

[0019] According to an example embodiment of the present disclosure, the prefix penalties are provided by the label encoder and the second attention mechanism.BRIEF DESCRIPTION OF THE FIGURES

[0020] FIG. 1 illustrates a high-level architecture of an example embodiment of an E2E ASR system. FIG. 2 illustrates various modules of an example embodiment of an end-to-end ASR system utilizing Neural Transducer (NT) architecture. FIG. 3 illustrates various modules of another example embodiment of an E2E ASR system utilizing Neural Transducer (NT) architecture. FIG. 4 illustrates various modules of an example embodiment of an E2E ASR system utilizing Listen, Attend, and Spell (LAS)-type architecture. DETAILED DESCRIPTION

[0021] FIG. 1 illustrates a high-level architecture of an example embodiment of an E2E ASR system according to the present disclosure. As shown in FIG. 1, the E2E ASR system 10 has an ASR module 11 which is enabled to transcribe any speech occurring in the example scenario, e.g., a medical consultation between a doctor 101 and a patient 102. As shown in FIG. 1, a Virtual Assistant (VA) 103 is configured to assist a doctor 101, e.g., allowing the doctor 101 to perform verbal commands to the VA 103 and generate and / or manage relevant medical report(s) based on the doctor-patient encounter by automatically transcribing any speech occurring during the doctor-patient encounter. Speech occurring during the doctor-patient encounter can include the doctor-patient conversation 1001, a dictation part 1002 from the doctor (e.g., intended to provide more technical information for the subsequent medical report), and any command 1003 that the doctor 101 wants to execute (e.g., making corrections or specifying some specific part of a doctor-patient medical encounter report the doctor wants to fill in). As noted earlier, these commands 1003 can be defined by the doctor at will, which commands become immediately usable as they are incorporated in the command list to be handled by the ASR module 11.

[0022] The output of the example ASR module 11 will contain the transcription (output 1004 shown in FIG. 1) of the conversational speech occurring during the doctor-patient encounter (denoted as "..." next to "D" for doctor and "P" for patient in FIG. 1) as well as the commands uttered during the doctor-patient encounter. Since the overall set of commands is explicitly indicated as an additional input for the ASR module 11 (along with any other words that would be considered likely to boost recognition performance, e.g., the patient's name), any command found in the output transcription would be already marked automatically by the ASR module 11, as it is aware of the command(s). This marking could occur, for example, by the use of additional tokens that indicate the start and end of a command, thereby addressing any potential ambiguity that might arise involving words that also belong to normal conversational speech. In this manner, the ASR output 1004 can be generated and curated, e.g., in real-time, and the curated output portions can be fed to the subsequent system components, e.g., in an online fashion.

[0023] As shown in FIG. 1, the identified commands in the output 1004 are fed into the C&C module 104, and the conversational speech components of the doctor-patient encounter in the output 1004 are fed into the report generator module 105. In an example embodiment, the identified commands fed into the C&C module 104 can trigger an immediate responsive action in the VA's platform 106. In an example embodiment, the report generator module 105 can start the generation of the medical encounter report based on the conversational speech components of the doctor-patient encounter as the encounter ends (or, alternatively, start the encounter report as soon as the conversational speech components of the doctor-patient encounter in the output 1004 start to arrive the report generator module 105), and then transmit the generated medical encounter report to the VA's platform 106. After the encounter, the generated medical encounter report would be available in the VA's platform 106, thus fulfilling the doctor's verbal command to generate the medical encounter report.

[0024] FIG. 2 illustrates various component modules of an example embodiment of an E2E ASR system. The type of E2E ASR system illustrated in FIG. 2 belongs to the Neural Transducer (NT) architecture, in which there are typically three networks (modules): the audio encoder 2001, which is in charge of processing the speech signal; the text predictor 2002, which in charge of predicting the next token to be transcribed, regardless of the audio, as some sort of LM; and the joiner 2003, which is in charge of processing the outputs from the other two networks (audio encoder 2001 and the text predictor 2002), which processed outputs are passed through a Softmax function layer (module) 2004 to produce token probabilities (the joiner 2003 and the Softmax function layer 2004 can be grouped as belonging to a single network, which can be generally denoted as a transcription token probability prediction network). The Softmax function is a function that turns a vector of K real values (positive, negative, zero, or greater than one) into a vector of K real values that sum to 1, i.e., Softmax transforms the input values into output values between 0 and 1, so that they can be interpreted as probabilities. In addition to the above-noted three networks, the proposed system incorporates two additional networks (modules), bias encoder 2005 and label encoder 2006, plus two attention mechanisms (first attention mechanism 2007 and second attention mechanism 2008).

[0025] It should be noted that in NT models there are two different sequence indexes, t and u. The first index, t, refers to the audio speech signal, i.e., t could be referring to each of the speech frames. The second index, u, refers to the output token sequence, which includes the sub-words that constitute the transcription. Therefore, the joiner 2003 and the Softmax function layer 2004 are operating at both t and u dimensions when determining the ASR solution. The other variables shown in FIG. 2 are defined as follows: x refers to the audio speech frames (whose sequence index is t) that serve as input to the E2E system; y refers to the transcription tokens (whose sequence index is u) that the E2E system has to predict; b refers to the bias entries (indexed by z); and Z refers to the total number of bias entries in the bias list. h refers to the hidden output of a module (the superscript associated with h refers to the type of information it is carrying (x for audio, y for transcription and b for bias entries); the subscript associated with h refers to sequence indexing it operates on (t for speech frames, u for transcription tokens and z for bias entries identifiers); and a prime (') would be used to distinguish between two h's with the same superscript and subscript). p refers to the prefix penalties, which are generated at each u step for each of the Z bias entries. P(X) refers to the probability of an event X.

[0026] The first additional module, the bias encoder 2005, is in charge of providing embeddings, i.e., distributed vector representations, for each entry in the bias list, which can include the C&C commands and other desired words to bias for recognition. The bias encoder 2005 can be implemented with, e.g., a recurrent network such as Long-Short-Term Memory Recurrent Neural Network (LSTM RNN). In order to determine which, if any, of the bias entries is suitable to be transcribed at each moment, the first attention mechanism 2007 is employed. In this first attention mechanism 2007, operated based on queries (Q), keys (K), values (V) and prefix penalties, the Q would be taken from the last output of the text predictor 2002, while both K and V would be the embeddings generated by the bias encoder 2005. The prefix penalties are a way of preventing attention to certain K and / or V, as will be explained below. The resulting output of the first attention mechanism 2007 is concatenated with the output of the text predictor 2002, so the joiner 2003 and the Softmax function layer 2004 can function as in normal NTs.

[0027] Before proceeding further, a brief explanation of Q, K and V is provided here. In an attention mechanism, we have a sequence of values (conforming V), e.g., [v1, v2, v3, v4, v5], from which we would like to take a weighted sum of its elements dependent on a certain query Q. Such a query would be evaluated over a mirrored version of the sequence V, named sequence K (in this example [k1, k2, k3, k4, k5]), in order to obtain the weights for each element in sequence V for the above-mentioned weighted sum. An example case is presented below: A) V can be the acoustic realizations of words sought to be transcribed with an ASR system, e.g.: [hello_audio, my_audio, name_audio, is_audio, Alex_audio]. B) Although Q will be changing at every decoding step, for the sake of simplicity let's focus on the step in which the ASR system hypothesizes that the next word to transcribe might be "name", so the focus should be on that part of the audio (i.e., the attention mechanism would indicate which part of the audio to focus on), and then subsequent modules would be in charge of verification of the next word to transcribe (which is beyond the scope of the present explanation of Q, K and V)). Therefore, let's assume for this example that Q is name_hypothesis. C) K can be the same as V (which is usually the case), but for illustrative purposes, let's assume K is a more orthographic representation of the words, not just acoustic as V, e.g.: [hello_ortho, my_ortho, name _ortho, is_ortho, Alex_ortho]. D) Then, we would perform some operation of Q against K, obtaining the similarity weights, e.g.: [0, 0, 1, 0, 0]. E) Finally the weighted sum of the elements in V would be: 0*hello_audio + 0*my_audio + 1 *name_audio + 0*is_audio + 0*Alex_audio = name_audio. F) In this manner, the attention mechanism would have provided the ASR system with the portion of the audio that actually corresponds to the "name" word part, which was the one the ASR system sought to focus on (or pay attention to). G) The above-described operation represents the general scheme for an attention mechanism. In the ASR system according to the present disclosure, the attention mechanism is configured such that Q, K and V can be defined to enable focusing on specific parts of V depending on Q / K.

[0028] As an alternative to utilizing the last output of the text predictor 2002 as Q, the output from the audio encoder 2001 can be used as Q, in which case the output of the first attention mechanism 2007 is concatenated with the output of the audio encoder 2001, running in t domain, rather than in u. An additional alternative, which is shown in FIG. 3, is to provide an additional attention mechanism 2009, so that the output of the additional attention mechanism is concatenated with the output of the audio encoder 2001, and the output of the first attention mechanism 2007 is concatenated with the output of the text predictor 2002, thereby enabling both modes of operation, enriching both the audio encoder output and the text predictor output.

[0029] The previously mentioned prefix penalties are meant for restricting the first attention mechanism 2007 to only those entries that fit the current transcription context. For example, it may be desired to mask any non-fitting entry in the following scenario: the complete command is of the structure "go to field A", and the current transcription so far is "... go to field". In this scenario, it makes sense to only allow attention to commands like "go to field A" or "go to field B", the only ones that would fit the context. This masking is beneficial for ensuring robust recognition performance on lists with a large number of entries. Furthermore, it is also possible to extend the prefix penalties operation from only working intra-command (as previously exemplified) by adding two additional tokens to the lexicon, e.g., "<start_of_command>" and "<end_of_command>" to bookend any occurrence of a command in the training data. In this manner, the E2E ASR system can learn to predict when a command is going to be uttered and / or has been uttered. Until the token "<start_of_command>" is predicted, the E2E ASR system would produce prefix penalties to mask all the command entries, thereby preventing any attention to them in the first attention mechanism 2007, but subsequently enabling such attention once the token <start of command> occurs, which attention is enabled until the "<end_of_command>" token appears.

[0030] The label encoder 2006 and the second attention mechanism 2008 are configured to automatically produce the above-mentioned prefix penalties. The label encoder 2006 processes the output transcription tokens as a way of encoding the current state of the transcription, as exemplified before. The label encoder 2006 can be implemented with a LSTM RNN, for example. This encoded current state would then constitute the Q for the second attention mechanism 2008, while the K / V would be the embeddings of the entries in the command / bias list also used as K / V in the first attention mechanism 2007.

[0031] The resulting output of the second attention mechanism 2008 would then be gated by a sigmoid function module 208, to gain a clear binary notion of whether an entry is suitable or not at the specific moment of the ongoing transcription. During training of the example embodiment of the system according to the present disclosure, the ground-truth prefix penalties have to be provided, modifying the optimization criterion to also predict the ground-truth prefix penalties. During training, it is desired to have the ASR model produce prefix penalties that are exactly the same as the ground-truth prefix penalties (which are computed separately by reading the training data). If this training is successful, this means that, in inference time, the E2E ASR system can be relied upon to automatically produce proper prefix penalties for whichever transcription is being output, and there is no need to compute them separately.

[0032] The above-mentioned modification of the optimization criterion to predict the ground-truth prefix penalties can be achieved by adding an additional loss to the optimization criterion, accompanied by a weight to specify the relevance of such loss. This loss would consist of a cross-entropy between the ground-truth penalties and the predicted prefix penalties (after the sigmoid function module 208). The ground-truth penalties for the training data can be trivially computed by analyzing the input transcription and checking which bias entry fits at each sequence step (each step being a sub-word token of the transcription sequence, and the notion of fitting as exemplified above). This means that for each training utterance of length U (measured in sub-word tokens), and for a bias list of size Z, the ground-truth penalties would be a matrix of UxZ, with values of zeros / ones depending on whether a bias entry z fits / does not fit at the sequence step u.

[0033] The above-described, fully automated operation of the additional modules (bias encoder 2005, label encoder 2006, first attention mechanism 2007, second attention mechanism 2008, and additional attention mechanism 2009) enables the example embodiments of the E2E ASR system according to the present disclosure to achieve the dual objectives of i) normal functioning of the ASR system during inference, in the same manner a conventional E2E ASR system would operate, and ii) the extra capability of being able to input an arbitrary list of commands (C&C) and bias words to boost the ASR performance, with no additional modification to the system being required.

[0034] As an alternative to the NT architecture example embodiments shown in FIGs. 2 and 3, it should be noted that another type of architecture can be utilized for the E2E ASR system according to the present disclosure: the Listen, Attend, and Spell (LAS)-type architecture, an example embodiment of which is shown in FIG. 4. In the LAS-type architecture E2E ASR system shown in FIG. 4, the additional modules (i.e., the bias encoder 2005, the label encoder 2006, the first attention mechanism 2007, and the second attention mechanism 2008) interact with the "Spell" part (denoted in accordance with the industry convention as "decoder" 4003 in FIG. 4) of the LAS-type architecture.

[0035] In connection with FIG. 4, the LAS operation is initially described here without mentioning any of the bias modules. First, the audio is processed by the audio encoder 2001. Then, for each decoding step (which yields the next sub-word token, y t , of the transcription), the decoder 4003 produces a context vector (denoted by variable d in the diagram, superscript x and subscript t) that will be used as the Q in the main attention mechanism 4009, and the output of the audio encoder 2001 ( h k x in FIG. 4) being used as the K / V. The result of the main attention mechanism 4009 ( h t x in FIG. 4) is then fed back to the decoder 4003, together with the previous sub-word token, y t-1 , to produce the next context vector as well as the output that predicts the next sub-word token, y t , through the Softmax layer 2004 (in the embodiment shown in FIG. 4, the decoder 4003 and the Softmax layer 2004 constitute the transcription token probability prediction network).

[0036] Next, we describe the LAS operation with the bias modules shown in FIG. 4. The functioning of bias encoder 2005, label encoder 2006 and second attention mechanism 2008 (for prefix penalties generation) is the same as in the NT architecture described, e.g., in connection with FIG. 2. In FIG. 4, a different Q is used for the first attention with prefix penalties module 2007 in comparison to the embodiment of the NT architecture shown in FIG. 2. In the embodiment shown in FIG. 4, the Q used for the first attention with prefix penalties module 2007 is a context vector produced at each decoding step by the decoder 4003 (denoted by variable d in the diagram, superscript b and subscript t). As the notation indicates, this context vector is different than the one used in the main attention mechanism 4009.

[0037] Because the first attention with prefix penalties module 2007 (which acts as the LAS bias attention mechanism) in FIG. 4 uses a Q that was generated by the decoder 4003, which in turn receives input from the previous labels and the acoustic information, we can expect this Q to be richer than in the case of the NT architecture, e.g., as shown in FIG. 2, in which architecture the first attention with prefix penalties module 2007 made use of either the flow of information coming from the previous labels or from the acoustics (at most, both flows can be used in a split manner, each at a different bias attention mechanism, as shown in the embodiment of FIG. 3). In any case, for the NT architecture both flows (with or without biasing) are eventually joined in the joiner 2003, which (with the assistance of Softmax function layer 2004) is in charge of making the final decision about the next token to be transcribed.

[0038] In summary, an E2E ASR system according to the present disclosure incorporates at least one additional module that is seamlessly integrated in the DNN architecture to enable robust recognition of both conventional speech and C&C speech, which additional module provides the flexibility of being able to handle an external list of word commands or multi-word commands that serves to boost ASR performance with respect to the C&C speech, without disturbing the recognition of conversational speech.

Claims

1. A contextual end-to-end automatic speech recognition, ASR, system, comprising: an audio encoder (2001) configured to process input audio signal to produce as output encoded audio signal; a bias encoder (2005) configured to produce as output at least one bias entry corresponding to a word to bias for recognition by the ASR system; a text predictor (2002) configured to produce as output a prediction of a next transcription token, based on a previous transcription token; a first attention mechanism (2007) configured to receive the at least one bias entry and the prediction of the next transcription token, and determine whether the at least one bias entry is suitable to be transcribed at a specific moment of an ongoing transcription; a label encoder (2006) configured to encode a current state of transcription; a second attention mechanism (2008) configured to receive the at least one bias entry and the encoded current state of transcription, and produce prefix penalties for restricting the first attention mechanism to only entries fitting a current transcription context based on the encoded current state of transcription; and a transcription token probability prediction network (2003, 2004) comprising a joiner module (2003) and a Softmax function layer (2004), wherein the transcription token probability prediction network is configured to produce as output a probability of a selected transcription token, based at least in part on the output of the first attention mechanism, the prediction of the next transcription token and the output of the audio encoder, which are input to the joiner module.

2. The system according to claim 1, further comprising an additional attention mechanism (2009) configured to receive the at least one bias entry and the encoded audio signal, wherein the transcription token probability prediction network is configured to produce the probability of the selected transcription token further based on the output of the additional attention mechanism being input to the joiner module.

3. A contextual end-to-end automatic speech recognition, ASR, system, comprising: an audio encoder (2001) configured to process input audio signal to produce as output encoded audio signal; a bias encoder (2005) configured to produce as output at least one bias entry corresponding to a word to bias for recognition by the ASR system; a first attention mechanism (2007) configured to receive the at least one bias entry and determine whether the at least one bias entry is suitable to be transcribed at a specific moment of an ongoing transcription; a label encoder (2006) configured to encode a current state of transcription; a second attention mechanism (2008) configured to receive the at least one bias entry and the encoded current state of transcription, and produce prefix penalties for restricting the first attention mechanism to only entries fitting a current transcription context based on the encoded current state of transcription; a main attention mechanism (4009) configured to receive the encoded audio signal; and a transcription token probability prediction network (2004, 4003) comprising a decoder module (4003) and a Softmax function layer (2004), wherein the transcription token probability prediction network is configured to produce as output a probability of a selected transcription token, based at least in part on the output of the first attention mechanism and the output of the main attention mechanism, which are input to the decoder module, wherein the decoder module produces an audio speech context vector and a bias entry context vector, wherein the main attention mechanism is further configured to receive the audio speech context vector and the first attention mechanism is further configured to receive the bias entry context vector.

4. The system according to any preceding claim, wherein the output of the second attention mechanism is gated by a sigmoid function module (208).

5. The system according to any preceding claim, wherein the at least one bias entry corresponds to commands.

6. The system of according to claim 5, wherein each command is associated with a first token indicating a start of the command and a second token indicating an end of the command.

7. The system according to any preceding claim, wherein the second attention mechanism produces prefix penalties when the encoded current state of transcription does not correspond to the at least one bias entry.

8. The system according to any preceding claim, further comprising a virtual assistant (103) configured to receive the input audio signal.

9. The system according to claim 8, wherein the virtual assistant is further configured to generate a report based on the output of the transcription token probability prediction network.

10. The system according to any preceding claim, wherein the bias encoder is implemented with a Long-Short-Term Memory Recurrent Neural Network.

11. The system according to any preceding claim, wherein the bias encoder is configured to output embeddings or vector representations for each of the at least one bias entry.

12. The system according to any preceding claim, wherein the input audio signal is from a consultation between a doctor and a patient.

13. The system according to any preceding claim, wherein the ASR system is a Deep Neural Network.

14. A method of operating a contextual end-to-end automatic speech recognition, ASR, system, comprising: processing, by an audio encoder (2001), an input audio signal to produce as output encoded audio signal; producing, by a bias encoder (2005), as output at least one bias entry corresponding to a word to bias for recognition by the ASR system; producing, by a text predictor (200), as output a prediction of a next transcription token, based on a previous transcription token; determining, by a first attention mechanism (2007), whether the at least one bias entry output by the bias encoder is suitable to be transcribed at a specific moment of an ongoing transcription based on the at least one bias entry and the prediction of the next transcription token; encoding, by a label encoder (2006), a current state of transcription; producing, by a second attention mechanism (2008), prefix penalties for restricting the first attention mechanism to only entries fitting a current transcription context based on the encoded current state of transcription and the at least one bias entry; and producing, by a transcription token probability prediction network comprising a joiner module (2003) and a Softmax function layer (2004), as output a probability of a selected transcription token, based at least in part on the output of the first attention mechanism, the prediction of the next transcription token and the output of the audio encoder, which are input to the joiner module.

15. A method of operating a contextual end-to-end automatic speech recognition, ASR, system, comprising: processing, by an audio encoder (2001), an input audio signal to produce as output encoded audio signal; producing, by a bias encoder (2005), as output at least one bias entry corresponding to a word to bias for recognition by the ASR system; determining, by a first attention mechanism (2007), whether the at least one bias entry output by the bias encoder is suitable to be transcribed at a specific moment of an ongoing transcription based at least on the at least one bias entry; encoding, by a label encoder (2006), a current state of transcription; producing, by a second attention mechanism (2008), prefix penalties for restricting the first attention mechanism to only entries fitting a current transcription context based on the encoded current state of transcription and the at least one bias entry; receiving, by a main attention mechanism (4009), the encoded audio signal; and producing, by a transcription token probability prediction network comprising a decoder module (4003) and a Softmax function layer (2004), as output a probability of a selected transcription token, based at least in part on the output of the first attention mechanism and the output of the main attention mechanism, which are input to the decoder module, producing, by the decoder module, an audio speech context vector and a bias entry context vector, further comprising receiving, by the main attention mechanism, the audio speech context vector and receiving, by the first attention mechanism, the bias entry context vector.