System and method for spoken language understanding

Abstract

A computer-implemented method 100 for performing spoken language understanding, comprising: receiving data representing an audio that comprises speech 102; processing the data using a model to determine text corresponding to content of the speech 104; receiving an input for performing a language understanding task 106, the input comprising text-based representation of one or more semantic labels; processing the input using at least part of the model to extract semantic information from the text corresponding to the content of the speech 108; and obtaining the extracted semantic information for the language understanding task 110.

Description

TECHNICAL FIELD Embodiments described herein relate to systems and methods for spoken language understanding. BACKGROUND Spoken language understanding is a computer-implemented technology closely related to speech processing and natural language processing. For example, spoken language understanding in conversational artificial intelligence aims to convert spoken utterances into semantic elements such as user intents, entities, or emotions. One type of spoken language understanding is zero-shot spoken language understanding, which is arranged to comprehend user utterances in new domains without prior training using training data in those new domains. Existing systems for performing zero-shot spoken language understanding are typically modular systems or end-to-end systems. A modular system typically includes an automatic speech recognition model arranged to transcribe speech into text and a separate natural language understanding model operable to process the text to produce semantic output. In a modular system, the two models are trained separately. An end-to-end system typically includes an automatic speech recognition model arranged to transcribe speech into text and a natural language understanding model operable to process the text to produce semantic output. In an end-to-end system, the two models are connected and trained together, and the speech input is directly mapped to semantic output. In these existing modular systems and end-to-end systems, the use of both the automatic speech recognition model and the natural language understanding model in performing zero-shot spoken language understanding may require a relatively large amount of computational resources (e.g., processing resources, memory consumption, etc.). BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings in which: Fig. 1 is a flowchart illustrating an operation for performing spoken language understanding in one embodiment of the invention; Fig. 2 is a schematic diagram illustrating an operation for performing spoken language understanding in one embodiment of the invention; Fig. 3 is a flowchart illustrating an operation for performing spoken language understanding in one embodiment of the invention; Fig. 4 is a flowchart illustrating an operation for performing spoken language understanding in one embodiment of the invention; Fig. 5 is a schematic diagram illustrating an operation for performing speech recognition in one embodiment of the invention; Fig. 6 is a schematic diagram illustrating an operation for performing language understanding in one embodiment of the invention; Fig. 7 is a schematic diagram illustrating an operation for performing automatic speech recognition in one embodiment of the invention; Fig. 8 is a schematic diagram illustrating an operation for performing intent classification in one embodiment of the invention; Fig. 9 is a schematic diagram illustrating an operation for performing slot filling in one embodiment of the invention; Fig. 10A is a schematic diagram illustrating an operation for performing automatic speech recognition in one embodiment of the invention; Fig. 10B is a schematic diagram illustrating an operation for performing intent classification and slot filling in one embodiment of the invention; Fig. 11 is a schematic diagram illustrating an example of large language model based question generation process for intent and slot labels in one embodiment of the invention; and Fig. 12 is a block diagram of an example information handling system operable to perform the operations in some embodiments of the invention. DETAILED DESCRIPTION Embodiments of the invention relate to spoken language understanding such as zeroshot end-to-end spoken language understanding. In an embodiment, there is provided a computer-implemented method for performing spoken language understanding. The computer-implemented method comprises: receiving data representing an audio that comprises speech, processing the data using a model to determine text corresponding to content of the speech, receiving an input, which comprises text-based representation of one or more semantic labels, for performing a language understanding task, processing the input using at least part of the model to extract semantic information from the text corresponding to the content of the speech, and obtaining the extracted semantic information for the language understanding task. Here, the model is used to process both the data and the input, i.e., at least part of the model is arranged to perform both speech recognition and language understanding (instead of using one model to perform speech recognition and another, different model to perform language understanding). The processing of the data (for speech recognition) using the model may facilitate or guide the processing of the input (for language understanding) using at least part of the model. In some cases, the processing of the input (for language understanding) may not require the use of any external language model(s). Advantageously, in some embodiments, the model may be more parameter-efficient, and can be trained and / or operated more effectively (e.g., consumes less memory resources). Advantageously, in some embodiments, the language understanding task, or more generally spoken language understanding, can be performed more effectively. In some embodiments, the model is operable to perform zero-shot end-to-end spoken language understanding. In some embodiments, the model comprises a transformer-based model. In some embodiments, the model comprises an encoder and a decoder operably coupled with the encoder. In some embodiments, the processing of the data is performed using the encoder and the decoder. In some embodiments, the processing of the input is performed using at least part of the decoder (e.g., only part of the decoder). The decoder, after being used to process the data (for speech recognition), may provide an audio-conditioned language model, which can be useful to process the input (for language understanding). In some embodiments, the encoder comprises a self-attention mechanism and a feedforward neural network. The encoder can include one or more encoder layers. Each encoder layer may respectively include a self-attention mechanism and a feedforward neural network. The feedforward neural network of the encoder and / or the feedforward neural network of the decoder may include a multilayer perceptron. In some embodiments, the decoder comprises a self-attention mechanism, a crossattention mechanism, and a feedforward neural network. The decoder can include one or more decoder layers. Each decoder layer may respectively include a self-attention mechanism, a cross-attention mechanism, and a feedforward neural network. The output of the encoder may be operably coupled with the or each cross-attention mechanism of the decoder. In some embodiments, the processing of the data is performed using the self-attention mechanism and the feedforward neural network of the encoder and the self-attention mechanism, the cross-attention mechanism, and the feedforward neural network of the decoder. In some embodiments, the processing of the input is performed using the self-attention mechanism and the feedforward neural network of the decoder, without using any crossattention mechanism of the decoder. By not using the encoder and cross-attention mechanism(s) of the decoder in the processing of the input, undesirable influence of speech embedding associated with the speech on the language understanding task can be reduced or minimised. In some embodiments, the computer-implemented method further comprises recording or storing states of the decoder obtained as a result of the processing of the data, and applying the recorded or stored states of the decoder to facilitate the processing of the input. The states of the decoder include parameters useful for processing the input. In some embodiments, the states of the decoder represent acoustic features obtained by the encoder as a result of processing the data and textual dependencies obtained by the decoder as a result of processing the data. In some embodiments, the text-based representation of the one or more semantic labels comprises one or more questions each associated with (e.g., formed based on) a respective one of the semantic labels. The input may be considered as a prompt. In some embodiments, the input further comprises the text corresponding to the content of the speech. For example, the input may include multiple entries each respectively comprising the text corresponding to the content of the speech and text-based representation of a respective semantic label. By including the text corresponding to the content of the speech in the input, the extraction of the semantic information can be performed more effectively and / or more accurately. In some embodiments, the language understanding task comprises an intent classification task, the one or more semantic labels comprises one or more intent labels, and the extracted semantic information comprises extracted intent information. In some embodiments, the extracted intent information indicates respective positive or negative association with each respective intent label. A positive association with an intent label may mean that the text matches or best matches the intent label. A negative association with an intent label may mean that the text does not match or does not best match the intent label. As an example, a text considered to have intent A may match with intent label “A” (hence is positively associated with intent label “A”) and not match with intent labels “B” and “C” (hence is negatively associated with intent labels “B” and “C”). A positive association with an intent label may be represented by the answer “Yes” and a negative association with an intent label may be represented by the answer “No”. In some embodiments, the language understanding task comprises a slot filling task, the one or more semantic labels comprises one or more slot labels, and the extracted semantic information comprises one or more words, which are extracted from the text and associated with at least one of the slot labels. In some embodiments, the computer-implemented method further comprises (e.g., after obtaining the extracted semantic information): receiving a further input, which comprises text-based representation of one or more further semantic labels, for performing a further language understanding task, processing the further input using at least part of the model to extract further semantic information from the text corresponding to the content of the speech, and obtaining the further extracted semantic information. The processing of the input (for language understanding) using at least part of the model may facilitate or guide the processing of the further input (for further language understanding) using at least part of the model. In some embodiments in which the extracted semantic information comprises extracted intent information, the one or more further semantic labels are based on the extracted intent information. For example, the one or more further semantic labels available may depend on the extracted intent information. In some embodiments, the text-based representation of the one or more further semantic labels comprises one or more questions each associated with (e.g., formed based on) a respective one of the further semantic labels. In some embodiments, the further input further comprises the text corresponding to the content of the speech. For example, the further input may include multiple entries each respectively comprising the text corresponding to the content of the speech and textbased representation of a respective further semantic label. By including the text corresponding to the content of the speech in the further input, the extraction of the further semantic information can be performed more effectively and / or more accurately. In some embodiments, the further language understanding task comprises a slot filling task, the one or more further semantic labels comprises one or more slot labels, and the further extracted semantic information comprises one or more words, which are extracted from the text and associated with at least one of the slot labels. In some embodiments, the processing of the further input is performed using the at least part of the model (i.e. the same part of the model) used to process the input. In some embodiments, the processing of the further input is performed using at least part of the decoder (e.g., only part of the decoder) of the model. For example, the processing of the further input may be performed using the self-attention mechanism and the feedforward neural network of the decoder, without using any cross-attention mechanism of the decoder. By not using the encoder and cross-attention mechanism(s) of the decoder in the processing of the further input, undesirable influence of speech embedding associated with the speech on the further language understanding task can be reduced or minimised. In some embodiments, the data representing the audio comprises a visual representation of the audio. For example, the visual representation may include an oscillogram or a spectrogram. In some embodiments, the data representing the audio comprises an audio file. In some embodiments, the model comprises a visual representation generator operable to process the audio file to produce a visual representation of the audio (e.g., oscillogram or spectrogram). In some embodiments, the model has been trained based on prefix-tuning. Prefix-tuning allows the model to be adapted or trained in a parameter-efficient way. In some embodiments, the computer-implemented method further comprises outputting or presenting the obtained extracted semantic information. For example, the obtained extracted semantic information may be displayed on a display, played by a speaker, etc. In some embodiments, the computer-implemented method further comprises outputting or presenting the obtained further extracted semantic information. For example, the obtained further extracted semantic information may be displayed on a display, played by a speaker, etc. In some embodiments, the obtained extracted semantic information and / or the obtained further extracted semantic information may be used to facilitate or guide the performing of a task, trigger or provide a response, facilitate or guide the execution of a command, etc. In some embodiments, the computer-implemented method may be applied to perform more than two language understanding tasks (i.e., not only the language understanding task and the further language understanding task). In an embodiment, there is provided a system comprising one or more processors configured to perform the above described computer-implemented method. The system may be a zero-shot end-to-end spoken language understanding system. In a further embodiment, there is provided a carrier medium carrying computer readable instructions adapted to cause one or more processors to perform the computer-implemented method of the first aspect. In some embodiments, the carrier medium is a transitory computer readable medium (e.g., signal, wave, etc.). In some embodiments, the carrier medium is a non-transitory computer readable medium such as non-transitory computer readable storage medium. In a further embodiment, there is provided a computer program or a computer program product comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the computer-implemented method of the first aspect. Other features and aspects will become apparent by consideration of the detailed description and accompanying drawings. Any feature(s) described herein in relation to one aspect or embodiment may be applied to any other aspect or embodiment (may or may not be described herein) as appropriate and applicable. For example, any feature(s) described herein in relation to one aspect or embodiment may be combined with any other feature(s) described herein in relation to any other aspect or embodiment as appropriate and applicable. Terms of degree such as “generally”, “about”, “substantially”, or the like used herein to account for manufacture tolerance, degradation, trend, tendency, imperfect practical condition(s), etc., depending on context. Unless otherwise specified, the terms “connected”, “coupled”, “mounted” or the like used herein are intended to encompass both direct and indirect connection, coupling, mounting, etc. Fig. 1 illustrates an operation 100 for performing spoken language understanding in one embodiment of the invention. The operation 100 is a computer-implemented operation. In one example, the operation 100 is for performing zero-shot spoken language understanding. In one example, the operation 100 is for performing zero-shot end-to-end spoken language understanding. The operation 100 includes, in 102, receiving data representing an audio that includes speech. In one example, the data representing the audio includes a visual representation of the audio, such as oscillogram or spectrogram. In another example, the data representing the audio includes an audio file (digital file). In one example, the audio consists only of speech. In another example, the audio include speech as well as one or more other sounds (e.g., background noise, other noise, music, etc.). The operation 100 further includes, in 104, processing the data received in 102 using a model to determine text corresponding to the content of the speech. This processing to determine the text may be considered as speech recognition or automatic speech recognition. The model is a machine learning model, preferably a deep learning model. The model may be operable to perform zero-shot end-to-end spoken language understanding. In one example, the model may include a transformer model. In one example, the model may include an encoder and a decoder operably coupled with the decoder. In one example in which the model includes an encoder and a decoder, the processing of the data in 104 is performed using both the encoder and the decoder. The model may be or may have been trained based on prefix-tuning. The operation 100 further includes, in 106, receiving an input for performing a language understanding task. The input includes text-based representation of one or more semantic labels. In one example, the text-based representation of the one or more semantic labels may include one or more questions each associated with a respective semantic label. The input may further include the text corresponding to the content of the speech (determined as a result of 104). For example, the input may include one or more entries each respectively including the text corresponding to the content of the speech and text-based representation of a respective semantic label. In some cases, the input may be considered as a prompt. The language understanding task may include an intent classification task, a slot filling task, an emotion recognition task, a sentiment analysis task, a named entity recognition task, etc. In the example in which the language understanding task is an intent classification task, the one or more semantic labels include one or more intent labels. In the example in which the language understanding task is a slot filling task, the one or more semantic labels include one or more slot labels. The operation 100 further includes, in 108, processing the input received in 106 using at least part of the model (used in the processing in 104) to extract semantic information from the text corresponding to the content of the speech (determined as a result of 104). This processing to extract the semantic information may be considered as language understanding (natural language understanding). The processing of the data using the model in 104 may facilitate or guide the processing of the input in 108 using the at least part of the model. Thus, in some cases, the processing of the input in 108 may not require the use of any external language model(s). Also, as features related to the text corresponding to content of the speech may be embedded in the parameters of the model (as a result of the processing in 104), the input may not require or may not include the text corresponding to content of the speech. In one example in which the model includes an encoder and a decoder, the processing of the input in 108 is performed using the decoder (without using the encoder). The operation 100 further includes, in 110, obtaining the extracted semantic information for the language understanding task. In the example in which the language understanding task is an intent classification task, the extracted semantic information includes extracted intent information. For example, the extracted intent information may indicate respective positive or negative association with each respective intent label. A positive association with an intent label may mean that the text matches or best matches (e.g., probability based) the intent label. A negative association with an intent label may mean that the text does not match or does not best match (e.g., probability based) the intent label. In the example in which the language understanding task is a slot filling task, the extracted semantic information includes slot information, which includes one or more words extracted from the text and associated with at least one of the slot labels. In one embodiment, the obtained extracted semantic information is outputted or presented (e.g., via a display, a speaker, etc.). In some cases, the obtained extracted semantic information may be used to facilitate or guide the performing of a task, trigger or provide a response, facilitate or guide the execution of a command, etc., by a computing device. Fig. 2 illustrates an operation 200 for performing spoken language understanding in one embodiment of the invention. The operation 200 is a computer-implemented operation. In one example, the operation 200 is for performing zero-shot spoken language understanding. In one example, the operation 200 is for performing zero-shot end-to-end spoken language understanding. The operation 200 may be considered as an example implementation of the operation 100 so for brevity, the operation 200 will not be described in detail (as the details can be found in the disclosure related to operation 100). Briefly, in this embodiment, operation 200 includes receiving data representing an audio with speech (similar to or same as 102 in operation 100), processing the data using the model M to determine text corresponding to the content of the speech (similar to or same as 104 in operation 100), receiving and processing an input (which at least includes textbased representation of one or more semantic labels) for performing a language understanding task using part of the model M to extract semantic information from the text corresponding to the content of the speech (similar to or same as 106 and 108 in operation 100), and obtaining the semantic information for the language understanding task (similar to or same as 110 in operation 100). In this embodiment, the model M includes an encoder and a decoder. Both the encoder and the decoder of the model M are used in the processing of the data (for speech recognition). However, only the decoder of the model M is used in the processing of the input (for language learning). Fig. 3 illustrates an operation 300 for performing spoken language understanding in one embodiment of the invention. The operation 300 is a computer-implemented operation. In one example, the operation 300 is for performing zero-shot spoken language understanding. In one example, the operation 300 is for performing zero-shot end-to-end spoken language understanding. In this embodiment, the operation 300 includes the operation 100, with 302 in operation 300 generally corresponds to (substantially the same as or identical to) 102 in operation 100, 304 in operation 300 generally corresponds to (substantially the same as or identical to) 104 in operation 100, 306 in operation 300 generally corresponds to (substantially the same as or identical to) 106 in operation 100, 308 in operation 300 generally corresponds to (substantially the same as or identical to) 108 in operation 100, and 310 in operation 300 generally corresponds to (substantially the same as or identical to) 110 in operation 100. For brevity, 302-310 in operation 300 will not be described in detail (as the details can be found in the disclosure related to 102-110 in operation 100). The operation 300 further includes, in 312, receiving a further input for performing a further language understanding task (which may be the same as or different from the language understanding task). The further input includes text-based representation of one or more further semantic labels. In one example, the text-based representation of the one or more further semantic labels may include one or more questions each associated with a respective further semantic label. In one example, the one or more further semantic labels available depend on the extracted semantic information (obtained in 310). The further input may further include the text corresponding to the content of the speech (determined as a result of 304). For example, the further input may include one or more entries each respectively including the text corresponding to the content of the speech and text-based representation of a respective further semantic label. In some cases, the further input may be considered as a prompt. The further language understanding task may include an intent classification task, a slot filling task, an emotion recognition task, a sentiment analysis task, a named entity recognition task, etc. In the example in which the further language understanding task is an intent classification task, the one or more further semantic labels include one or more intent labels. In the example in which the further language understanding task is a slot filling task, the one or more further semantic labels include one or more slot labels. The operation 300 further includes, in 314, processing the further input received in 312 using at least part of the model (used in 304) to extract further semantic information from the text corresponding to the content of the speech (determined as a result of 304). This processing to extract the further semantic information may be considered as language understanding (natural language understanding). The processing of the data using the model in 304 and the processing of the input using at least part of the model in 308 may facilitate or guide the processing of the further input in 314 using at least part of the model. Thus, in some cases, the processing of the further input in 314 may not require the use of any external language model(s). Also, as features related to the text corresponding to content of the speech may be embedded in the parameters of the model (as a result of the processing of the data in 304 and the processing of the input using at least part of the model in 308), the further input may not require or include the text corresponding to content of the speech. The at least part of the model used the processing in 314 may be the same as the at least part of the model used in the processing in 308. In one example in which the model includes an encoder and a decoder, the processing of the input in 314 is performed using the decoder (without using the encoder). The operation 300 further includes, in 316, obtaining the further extracted semantic information for the further language understanding task. In the example in which the further language understanding task is an intent classification task, the further extracted semantic information includes extracted intent information. For example, the further extracted intent information may indicate respective positive or negative association with each respective intent label. A positive association with an intent label may mean that the text matches or best matches (e.g., probability based) the intent label. A negative association with an intent label may mean that the text does not match or does not best match (e.g., probability based) the intent label. In the example in which the further language understanding task is a slot filling task, the further extracted semantic information includes slot information, which includes one or more words extracted from the text and associated with at least one of the slot labels. In one embodiment, the obtained further extracted semantic information is outputted or presented (e.g., via a display, a speaker, etc.). In some cases, the obtained further extracted semantic information may be used to facilitate or guide the performing of a task, trigger or provide a response, facilitate or guide the execution of a command, etc., by a computing device. Fig. 4 illustrates an operation 400 for performing spoken language understanding in one embodiment of the invention. The operation 400 is a computer-implemented operation. In one example, the operation 400 is for performing zero-shot spoken language understanding. In one example, the operation 400 is for performing zero-shot end-to-end spoken language understanding. The operation 400 may be considered as an example implementation of the operation 300, with 402-416 of operation 400 generally corresponding to (substantially the same as or identical to) 302-316 of operation 300 and in which the language understanding task in 406 is an intent classification task and the further language understanding task in 412 is a slot filling task. Fig. 5 illustrates an operation 500 for performing speech recognition in one embodiment of the invention. The operation 500 is a computer-implemented operation. The operation 500 may be considered as an example implementation of 102-104 in operation 100, 302-304 in operation 300, or 402-404 in operation 400. As shown in Fig. 5, the operation 500 is performed using a model including an encoder E and a decoder D. The encoder E includes multiple encoder layers Ei to En (n is an integer larger than 1). Each encoder layer Ei to En respectively includes a self-attention module and a feedforward neural network. The decoder D includes multiple decoder layers Di to Dm (m is an integer larger than 1 and it can be the same as or different from n). Each decoder layer Di to Dm respectively includes a self-attention module, a cross-attention module, and a feedforward neural network. The output of the encoder E, in particular encoder layer En, may be operably coupled with the cross-attention modules in each of the decoder layers Di to Dm- In operation 500, data representing an audio with speech, such as a spectrogram, is received and processed by the encoder E and the decoder D to transcribe the speech (i.e., produce text corresponding to the speech). The processing is performed using the self-attention module and the feedforward neural network of each encoder layer Ei to En as well as the self-attention module, the cross-attention module, and the feedforward neural network of each decoder layer Di to Dm. As part of the processing, the decoder D receives tokens as input to guide or facilitate the transcription of the speech. The tokens may include the transcribed words and special tokens (e.g., a special token as a command to generate transcript, a special token as an indicator of end of sequence or transcription, etc.). A word that has been transcribed from the speech may be input to the decoder D to guide the transcription of the next word from the speech. The states of the decoder D obtained as a result of performing the transcription can be recorded or stored (e.g., in a cache) to facilitate subsequent processing. These states of the decoder D may represent acoustic features obtained by the encoder E and textual dependencies obtained by the decoder D, as a result of performing the transcription. Fig. 6 illustrates an operation 600 for performing language understanding in one embodiment of the invention. The operation 600 is a computer-implemented operation. The operation 600 may be considered as an example implementation of 106-110 in operation 100, 306-310 in operation 300, or 406-410 in operation 400. In this embodiment, the operation 600 is implemented using the decoder D used in the operation 500. In operation 600, an input for performing a language understanding task is received by the decoder D. In this embodiment, the input includes the transcribed text of the speech and text-based representation of one or more semantic labels. The decoder D processes the input without using any of the cross-attention modules (i.e., only the self-attention module and the feedforward neural network in each decoder layer Di to Dm are used) to extract semantic information from the transcribed text of the speech. In this embodiment, the recorded or stored states of the decoder D (obtained as a result of performing the transcription) are applied to facilitate the processing of the input. Extracted semantic information for the language understanding task can thus be obtained. The following description in relation to Figs. 7 to 9 relate to an operation for performing spoken language understanding in one embodiment. Figs. 7 to 9 respectively illustrate an operation 700 for performing automatic speech recognition in one embodiment (e.g., as an example of 102-104 in operation 100, an example of 302-304 in operation 300, an example of 402-404 in operation 400, or an example of operation 500), an operation 800 for performing intent classification in one embodiment (e.g., as an example of 106-110 in operation 100, an example of 306-310 or 312-316 in operation 300, an example of 406-410 in operation 400, or an example of operation 600), and an operation 900 for performing slot filling in one embodiment (e.g., as an example of 106-110 in operation 100, an example of 306-310 or 312-316 in operation 300, an example of 412-416 in operation 400, or an example of operation 600). The operations 700 to 900 in these embodiments are performed based on the same model. In the following, for ease of presentation, operation 700 and at least one of operations 800 and 900 are described as belonging to the same operation for performing spoken language understanding. Inventors of the present invention have devised that a model with an encoder-decoder framework (with an encoder and a decoder) can be used for automatic speech recognition and such a model, when suitably trained, can become a robust audioconditioned language model capable of performing complex language understanding tasks. In this embodiment, such a model operable for both automatic speech recognition and natural language understanding is used to perform operation 700 and at least one of operations 800 and 900. The model in this embodiment does not include a separate natural language understanding model that uses large language model(s). In other words, the model in this embodiment does not use any external language model. In this example, the natural language understanding tasks are intent classification and slot filling (correspond to operations 800 and 900). In this embodiment, to enable zero-shot operation of the model, the natural language understanding tasks (intent classification and slot filling) are formulated as questionanswering problems. The semantic labels, such as intents and slot types, are transformed into questions that represent the semantic meaning of the labels. In this way, when new intents or slot types appear, questions can be formed accordingly to represent their semantics, without having to further train the model with labelled data. In this embodiment, during inference for natural language understanding, the questions are input to the decoder of the model to generate relevant answers for the natural language understanding tasks. The questions prepared for intents and the questions prepared for slot types may be in different formats. For example, for intent classification, an intent label "play_music" can be transformed into an intent question “Does the user want to play music?”, and the answer to the intent question can be binary (e.g., “Yes” or “No”). For example, for slot filling, a slot type "date" can be transformed into a slot question “What is the date?”, and the answer to the slot question can be the slot value(s) that appear in the query (e.g., the transcribed text) or a special token as the end-of-sequence [EOS] (if there is no slot value associated with the question, e.g., if none of the transcribed text corresponds to an answer to the question). The general operation for performing spoken language understanding in this embodiment is as follows. First, the model receives a speech audio and transcribes the speech of the speech audio to obtain text corresponding to the speech. The states of the decoder of the model obtained during or after the transcription are stored or recorded. Then, prompts are organised or formed for input into the decoder of the model, by appending or concatenating the transcribed text with each semantic question (each associated with a respective semantic label). The decoder of the model receives the prompts and processes them to generate task-specific answers, based on the stored or recorded states of the decoder. A more detailed description will now be presented with reference to Figs. 7 to 9. In Fig. 7, the operation 700 for performing speech recognition (as part of an operation for performing spoken language understanding) is performed using a transformer-based model with a transformer encoder and a transformer decoder. As shown in Fig. 7, the transformer encoder includes N transformer encoder layers (N is an integer), each respectively including a self-attention mechanism and a feedforward neural network FFN. Each of the transformer encoder layers respectively include a layer input (only one is shown in Fig. 7 for simplicity). The transformer decoder includes N transformer decoder layers (N is an integer), each respectively including a self-attention mechanism, a cross-attention mechanism, and a feedforward neural network FFN. Each of the transformer decoder layers respectively include a layer input (only one is shown in Fig. 7 for simplicity). The output of the transformer encoder is coupled with the cross-attention mechanisms of the transformer decoder layers. In operation 700, the transformer encoder of the model receives data representing a speech audio (acoustic features) and processes the data to produce speech embedding. The transformer decoder of the model, which is operably coupled with the transformer encoder of the model, initially beginning with a [transcript] token, generates transcribed text (e.g., a word) of the speech based on the speech embedding. The generated transcribed text (e.g., a word) is feedback into the transformer decoder to guide or facilitate the generation of further transcribed text (e.g., the next word) of the speech. This process may repeat until the transcription of the speech is completed. In this way, the transformer decoder takes previously generated token(s) (e.g., token such as a word of the transcribed text obtained in a previous decoding step) as input to serve as context to guide the next decoding step, i.e., to guide the decoding hence the transcription. In this embodiment, for example, when the first word “buy” is transcribed from the speech, the word “buy” is input back to the transformer decoder to serve as context for the next decoding step (i.e., to facilitate transcription of the next word “a”). At the end of the decoding (or the transcription), the transfer decoder may generate an end-of-sequence [EOS] token to signify the completion of the transcription. In this example, as shown in Fig. 7, the transcribed text of the speech is “buy a concert ticket on friday”. Referring now to Fig. 8, which shows the operation 800 for performing intent classification (as part of the operation for performing spoken language understanding). In this embodiment, the operation 800 is performed after operation 700, using the transformer-based model used in operation 700. In particular, the operation 800 is performed using the transformer decoder of the model (without using the transformer encoder and the cross-attention mechanisms in the decoder, to avoid using the speech embedding). In operation 800, intent classification prompts (multiple input entries each respectively including the transcript “buy a concert ticket on friday” and an intent question corresponding to an intent label “Does the user want to [intent label]?”) are input into the transformer decoder. The [intent label] used in this example are “book a ticket”, “set an alarm”, and “play music”. In this embodiment, the transformer decoder receives the intent classification prompts associated with the different intent labels together as a batch. The transformer decoder processes these prompts to extract intent information (intent answers) from the transcribed text. In this embodiment, the intent answers are binary (e.g., either “Yes” or “No”). In this embodiment, for the last token in the intent classification prompts, the probabilities of token “Yes” is selected from its vocabulary distribution. More specifically, the intent classification prompts received by the transformer decoder are padded to be the same length. For each token (e.g., words and symbols) in the intent classification prompts, a probability distribution exists at the output layer (final transformer decoder layer), which has the dimension of vocabulary size. The distributions of the last token (e.g., “?”) in each intent classification prompt is considered, and the probability of the "Yes" token is selected from the distributions. The "Yes" probabilities of the intent classification prompts are compared with each other. The intent classification prompt with the highest “Yes” probability is chosen, and the corresponding intent is determined to be the predicted intent. Operation 800 conditions on the stored states of the model (in particular the transformer decoder) obtained as a result of the transcription in operation 700. Referring now to Fig. 9, which shows the operation 900 for performing slot filling (as part of the operation for performing spoken language understanding). In this embodiment, the operation 900 is performed after operation 700 (and operation 800 may not be performed), using the transformer-based model used in the operation 700. In particular, the operation 900 is performed using the transformer decoder of the transformer-based model (without using the transformer encoder and the cross-attention mechanisms in the decoder, to avoid using the speech embedding). In operation 900, slot prompts (multiple input entries each respectively including the transcript “buy a concert ticket on friday” and a slot filling question corresponding to a slot label “What is the [slot label]?”) are input into the transformer decoder. The [slot label] used in this example are “type of the ticket”, “time of the alarm”, and “name of the song”. In this embodiment, the transformer decoder receives the slot prompts associated with the different slot labels together as a batch. The transformer decoder processes these prompts to extract slot information (slot answers) from the transcribed text. In this embodiment, the slot answers are open-ended (not simply “Yes” or “No”), and include one or more words present in the transcribed text. In this embodiment, the slot answers or values are generated in a similar way as the transcription in operation 700 (the generation is terminated by a [EOS] token). Operation 900 conditions on the stored states of the model (the transformer decoder) obtained as a result of the transcription in operation 700. In this embodiment, prefix-tuning is applied to train the model to enable zero-shot abilities in operations 700 to 900. In this embodiment, a series of randomly initialised vectors, i.e., prefix, are prepended to the decoder states (“ASR states”) of each of the transformer decoder layers to guide the model in generating desired outputs. In this embodiment, the prefix vectors in the transformer decoder are shared for the automatic speech recognition task (operation 700) and the natural language understanding task (operation 800, 900). Similar prefix vectors are used to facilitate automatic speech recognition adaptation at the transformer encoder. In this embodiment, the model is kept frozen (model parameters such as weights kept unchanged) during training. Experiments are conducted under in-corpus and cross-corpus settings to verify the performance of the operation for performing spoken language understanding in this embodiment (based on operations 700 to 900). In the experiments, the Whisper model (also referred to as “Whisper SLU”, with a Whisper-large-v2 architecture) disclosed in Radford et al.’s “Robust speech recognition via large-scale weak supervision” (2023) is used as the model to perform spoken language understanding based on operations 700 to 900, and a modular system (also referred to as “Modular ASR-NLU”) including an automatic speech recognition (ASR) model (Whisper-large-v2) and a natural language understanding (NLU) model (GPT2-large) is used as a baseline system (for comparing with the Whisper model) to perform the spoken language understanding operations. In this example, the GPT2-large model is used to implement the natural language understanding model in the baseline system because the GPT2-large model has a similar size as, hence is comparable with, the decoder in Whisper-large-v2. For the in-corpus experiment, the Whisper model is prefix-tuned on the training set SLURP (containing 64 intents and 50 slot types) and tested on the test set containing 5 unseen slots. For the cross-corpus experiment, the Whisper model prefix-tuned on SLURP is then tested on FSC (containing 15 intents). On the other hand, both models of the baseline system are trained using the prefix-tuning strategy. Example details of the prefix-tuning strategy can be found in, e.g., Li et al.’s “Prefix-Tuning: Optimizing Continuous Prompts for Generation” (2021). The results of the experiments in this example are as follows: Table 1: In-corpus experiment results in one example: zero-shot slot filling SLU-F1 on the test set of SLURP Model / System Architecture #Total parameters Word error rate (WER) Slot filling SLU-F1 Modular ASR-NLU (benchmark) ASR: Whisper-large-v2 NLU: GPT2-large 2.3 billion 8.5 43.6 Whisper SLU (this embodiment) Whisper-large-v2 1.5 billion 8.3 50.0 Table 2: Cross-corpus experiment results in one example: zero-shot intent classification accuracy on FSC Model / System Word error rate (WER) Intent classification accuracy (%) Modular ASR-NLU (benchmark) 8.3 95.1 Whisper SLU (this embodiment) 8.2 95.0 As seen from Tables 1 and 2, for both in-corpus and cross-corpus experiments, the operation implemented using the Whisper model achieves similar or better zero-shot results compared to the operation implemented using the modular system. The natural language understanding model of the modular system, the GPT2-large model, is pretrained on massive amount of text data and is believed to have strong language understanding abilities. From the results, however, it is noted that the Whisper model demonstrates equally good (if not better) zero-shot proficiency in performing spoken language understanding, and with substantially fewer model parameters, which is advantageous in some practical applications. The operations 700 to 900 implemented using the model in this embodiment can perform zero-shot spoken language understanding by feeding prompts to the decoder of the model. This eliminates the need of any external language models. Also, the model in this embodiment is trained with prefix-tuning strategy hence can be trained efficiency. In this embodiment, a model with encoder-decoder structure is used in an operation to conduct zero-shot end-to-end spoken language understanding, including automatic speech recognition and natural language understanding (such as intent detection / classification and slot filling), without dependence on external language models. In this embodiment, zero-shot functionalities (generalisation to unseen intents and slots (not seen in training)) can be achieved through a prompt-based method applied to the decoder of the model. In this embodiment, the model is better adapted to spoken language understanding tasks by employing an efficient prefix-tuning strategy. In this embodiment, the model is parameter-efficient and may consume less computational resource during inference than other existing systems / models. The following description in relation to Figs. 10A and 10B relate to an operation for performing spoken language understanding in one embodiment. Figs. 10A and 10B respectively illustrate an operation 1000A for performing automatic speech recognition in one embodiment (e.g., as an example of 102-104 in operation 100, an example of 302-304 in operation 300, an example of 402-404 in operation 400, or an example of operation 500) and an operation 1000B for performing intent classification and slot filling in one embodiment (e.g., as an example of 306-316 in operation 300, or an example of 406-416 in operation 400). These operations 1000A and 1000B are performed based on the same model. In the following, for ease of presentation, operations 1000A and 1000B are described as belonging to the same operation for performing spoken language understanding. In this embodiment, the operations 1000A and 1000B are performed using the Whisper model integrated into a prompt-based question-answering framework (with a specifically designed semantic question generation method), and the Whisper model is prefix-tuned for multitasking automatic speech recognition and zero-shot end-to-end spoken language understanding. The Whisper model used in this embodiment is an open-sourced speech processing model developed by OpenAI, Inc. Example details of the model can be found in Radford et al.’s “Robust speech recognition via large-scale weak supervision” (2023). Briefly, the Whisper model has been trained on 68000 hours of multilingual and multitask supervised data sourced from the internet. This diverse dataset contributes to enhanced robustness, allowing the Whisper model to effectively handle variations in accents, background noise, and technical language. The Whisper model has an end-to-end architecture that builds upon an encoder-decoder transformer (with an encoder and a decoder). The encoder processes the log-Mel spectrogram of a 30-second audio chunk as input. Depending on the resulting speech embeddings and user-provided tokens (e.g., special tokens), the decoder may execute tasks such as speech transcription, language identification, or X-to-English translation. As shown in Fig. 10A, the encoder includes L encoder layers (L is an integer), each respectively including a self-attention mechanism and a feedforward neural network FFN. Each of the encoder layers respectively include a layer input (only one is shown in Fig. 10A for simplicity). The decoder includes L decoder layers (L is an integer), each respectively including a self-attention mechanism, a cross-attention mechanism, and a feedforward neural network FFN. Each of the decoder layers respectively include a layer input (only one is shown in Fig. 10A for simplicity). The output of the encoder is coupled with the cross-attention mechanisms of the decoder layers. Inventors of the present invention have found that information incorporated in the decoding prompt of the model may influence the model output. Inventors of the present invention have devised, through their research, experiments, and trials, that the Whisper decoder may be utilised as a robust audio-conditioned language model, hence may be suitable for zero-shot spoken language understanding. The prompt-based question-answering framework for spoken language understanding in this embodiment is now presented. In this embodiment, given a spoken utterance X, the goal is to identify its intent from a set of candidates I, and to find all the entities in X corresponding to a set of slot types S. Inventors of the present invention have noticed that in conventional supervised learning methods, these tasks are usually treated as utterance or word-level classification problems. Inventors of the present invention have found that this approach lacks flexibility in accommodating new elements introduced to I and S without further training the classifier (a scenario typically known as zero-shot learning). To address this problem, this embodiment provides a framework called “ZS-Whisper-SLU”, a generative question-answering framework based on Whisper, for conducting zero-shot intent classification and slot filling. The framework and workflow of “ZS-Whisper-SLU” is illustrated in Figs. 10A and 10B. In this embodiment, descriptive questions are created for each element in 1 and S to construct question sets Q, and Qs. The tasks are executed in three stages. Stage 1 relates to automatic speech recognition (ASR), the operation of which is illustrated in Fig. 10A. In this example, the transcription operation is based on the standard transcription procedure performed using the Whisper model (both the encoder and the decoder are used), which involves the use of tokens as described above. In this embodiment, the decoder states (the automatic speech recognition decoder states, i.e., “ASR states”) are reserved or stored in a cache. These states encompass both acoustic features from the encoder and textual dependencies captured at the decoder. In this example, the transcribed text of the speech is “play bbc radio”. Stage 2 relates to intent classification, the operation of which is illustrated in Fig. 10B. In this example, the 1-best transcript obtained from stage 1, i.e., “play bbc radio”, is duplicated and combined with all questions in Q, The resulting sequences (intent prompts) are organised into a minibatch and are directly input to the decoder of the Whisper model (the encoder and the cross-attention of the decoder of the Whisper model are not used) to generate binary answers “Yes” or “No", in parallel. In this example, both answers, in plain texts, are tokenised using Whisper model’s own vocabulary, so that no additional verbaliser (binary classifier) is needed. As the intent questions in these text prompts do not have paired speech embeddings, the cross-attention module at each decoder layer is skipped (not used in the processing for intent classification). In this example, in addition to using the transcribed text in the prompt, the intent answer generation also relies on the cached decoder states for the purpose of end-to-end modelling. Stage 3 relates to intent-restricted slot filling, the operation of which is illustrated in Fig. 10B. The decoding process in stage 3 is similar to the decoding process in stage 2 (the encoder and the cross-attention of the decoder are also not used in the processing). In this example, however, instead of feeding all questions in Qs to the decoder of the Whisper model, only those determined (e.g., by computer) to be relevant to the intent identified in stage 2 are selected (input to and processed by the decoder). This may help to reduce computation costs (e.g., reduce processing power required, reduce memory consumption, etc.) and to reduce or minimise distracting slot types that are irrelevant (e.g. the slot type “artist” may not be pertinent to the intent “takeaway_order”). In this example, the slot answers are open-ended and correspond to the entities mentioned in X. If an entity does not exist for a certain slot type, the system may produce an [EOT] (end of transcript) symbol. Given the potential size of Q, and Qs, in this embodiment, contrastive learning is employed on each utterance for efficient training. This involves creating a mini-batch by combining positive questions representing the ground-truth intent and slots, along with N randomly-sampled negative questions for both semantics. In this embodiment, the system is optimised using cross-entropy loss, with prepared answers serving as labels. During inference, the entire Qj set is prompted to the Whisper model, and the question with the highest likelihood of “Yes” answer is selected to be the predicted intent. Nonempty answers to the chosen slot questions are collected as entity outputs. In cases where multiple slot types yield the same answer, the one with the highest probability is selected. Within the framework in this embodiment, new intents and slot types can be readily introduced by adding their questions to Qj and Qs respectively. In this embodiment, prefix-tuning, as a parameter-efficient fine-tuning technique for adapting deep neural networks, is applied. In prefix-tuning, a relatively small (e.g., minimal) amount of additional parameters (typically less than 1%) to the existing model are optimised. Thus, compared to the traditional full fine-tuning strategies, a better adaptation effect can be achieved. As illustrated in Figs. 10A and 10B, to steer the Whisper model towards producing reliable automatic speech recognition (ASR) transcripts and desired spoken language understanding (SLU) answers, a sequence of trainable prefix vectors is concatenated with the speech / text embeddings at each encoder / decoder layer. Specifically, these vectors are injected to the key K and value V elements involved in the Whisper model’s self-attention modules: K := Concat(pK, K), V := Concat(pv,V) (1) where pK and pv denote the prefix vectors generated by a prefix encoder (implemented as an embedding layer), serving as the soft contextual information for the ongoing encoding / decoding process. In one example, the prefix vectors are shared across the automatic speech recognition task and the spoken language understanding task at each decoder layer. This facilitates the exchange of shared knowledge such as entity names. In the system of this embodiment, the prefix vectors can help to adapt the Whisper model to the acoustic characteristics of the spoken language understanding corpus, enhance language modelling for task-specific domains, address text normalisation issues like capitalisation and punctuation in automatic speech recognition transcripts, and guide the Whisper model’s decoder in producing intent and slot answers conditioned on the automatic speech recognition decoder states. In this embodiment, the employment of prefix-tuning preserves the capabilities of the original Whisper model, thus ensuring flexibility for other use cases. Inventors of the present invention have realised that high-quality questions (for input to the model) are particularly useful for more accurately extracting semantic arguments within question-answering based natural or spoken language understanding framework. Inventors of the present invention have further realised that some existing question design methods are prone to human biases and may suffer from poor generalisation. For instance, the SLURP corpus includes an intent class labelled convert_datetime, where users seek to convert date or time between different time zones. In this case, it is obvious that the true intent is not explicitly conveyed in the label. Thus, transforming such labels or names directly into semantic questions may produce misleading prompts, which may adversely affect the performing of natural or spoken language understanding. Inventors of the present invention have thus devised that a large language model may be leveraged to generate semantic questions, providing both the label name and a few user utterance examples to mitigate naming biases. In this embodiment, the large language model GPT-3.5 is used. Example details of the GPT-3.5 model can be found in Brown et al.’s “Language models are few-shot learners” (2020). As shown in Fig. 11, in this example, the GPT-3.5 model is first instructed to produce a concise description for each semantic label, using the prompt templates illustrated. Then the output description is transformed into a question of a fixed format. In this example, an enhanced stability can be observed by adopting a two-step question generation process (compared to prompting the GPT-3.5 model for a single-shot question production). Experiments are conducted to evaluate the performance of the ZS-Whisper-SLU framework in this embodiment. In the experiments in this example, the ZS-Whisper-SLU framework is evaluated on three datasets, under in-corpus and cross-corpus settings, and ablation study is also performed. The first dataset used is SLURP (see, e.g., Bastianelli et al.’s “SLURP: A spoken language understanding resource package” (2020)). The SLURP dataset includes 65 intents and 56 slots within the domain of in-home personal robot assistant. In this example, a new data split is created for a zero-shot evaluation on 5 randomly selected slot types {podcast_name, artist_name, audiobook_name, business_name, radio_name}. This adapted dataset is utilised for in-corpus evaluation. The second dataset used is FSC (see, e.g., Lugosch et al.’s “Speech model pre-training for end-to-end spoken language understanding” (2019)). The FSC dataset is a widely used voice assistant corpus, featuring spoken commands labelled with {action, object, location} arguments. The argument values are drawn from a predefined set and constitute a total of 31 intents. In this example, they are reorganised into 15 intents and 2 slots ({language, location}, in order to accommodate the [None] value), and the results are transformed back to the original format for evaluation. This FSC dataset is used for the cross-corpus evaluations. The third dataset used is SmartLight (close-field) (see, e.g., Saade et al.’s “Spoken language understanding on the edge” (2019)). The SmartLight dataset is a subset of the SNIPS SLU benchmark that contains instructions for controlling smart lights. The dataset comes with 6 intents and 3 slots to cover the functions and attributes of lights. This SmartLight dataset is also used for the cross-corpus evaluations. In this example, the ZS-Whisper-SLU framework uses Whisper-large-v2 (with 1.5 billion parameters) as the foundation automatic speech recognition model for zero-shot spoken language understanding adaptation. The ZS-Whisper-SLU framework that uses Whisper-large-v2 is compared against three benchmark systems: • Modular-1: the modular system disclosed in Sun et al.’s, “Knowledge-aware audio-grounded generative slot filling for limited annotated data” (2023), which includes a conformer-LSTM (long short-term memory) automatic speech recognition module and a GPT-2 natural language understanding module • KA2G: the knowledge-aware audio grounded (KA2G) system disclosed in Sun et al.’s, “Knowledge-aware audio-grounded generative slot filling for limited annotated data” (2023), which is an end-to-end counterpart of the Modular-1 system, with the automatic speech recognition and natural language understanding modules connected through a neural aligner at the decoder state level • Modular-2: a modular system developed by the inventors of the present invention for use in this experiment, which includes a Whisper automatic speech recognition module and a GPT-2 natural language understanding module. In the modular-2 system, for fair comparison, the automatic speech recognition module also utilises Whisper-large-v2, and the natural language understanding module employs GPT-2-large (with 0.8 billion parameters), which shares a similar architecture with the automatic speech recognition decoder. Both modules are prefixed-tuned using the same hyper-parameters as for the proposed system. For the in-corpus experiments performed using SLURP, the ZS-Whisper-SLU in this embodiment and the GPT-2 modules in Modular-1, KA2G, and Modular-2 systems are trained using N = 10 negative intent and slot examples for each utterance. The prefix length is set to 10 per task, resulting in 10 prefix vectors at each encoder layer (for automatic speech recognition) and 30 prefix vectors at each decoder layer (for joint automatic speech recognition, intent classification, and slot filling). All models (in the system of the embodiment and the benchmark systems) undergo 10 epochs of training using the AdamW optimiser with a mini-batch size of 12. A linear learning rate scheduler, starting from 0.002 without warmup, is employed for weight decay. For the cross-corpus experiments, the SLURP-developed models are directly applied to the cross-corpus evaluations on FSC and SmartLight. However, the encoder prefix vectors are removed to prevent transfer of acoustic knowledge. In this experiment, the official test set of FSC and all utterances in SmartLight (which lacks a data split) are utilised. For fast inference, all transcripts and semantic answers are generated with greedy-search decoding. The results of the experiments in this example are as follows. Table 3 shows the in-corpus automatic speech recognition and slot filling results on the zero-shot test set of SLURP. It can be seen that the ZS-Whisper-SLU system outperforms both the Modular-1 and KA2G systems in both metrics. The ZS-Whisper-SLU system, with the automatic speech recognition power of the Whisper model, achieves a substantial 9.7% reduction in word-error-rate (WER) when compared to KA2G. This further leads to an absolute 40.7% improvement in SLU-F1 score on the unseen slots. ZS-Whisper-SLU also demonstrates superiority over the Modular-2 system which incorporates a GPT-2-large large language model for natural language understanding, with a relative SLU-F1 gain of 14.9% ((50-43.5) / 43.5)). In addition to enhanced performance, ZS-Whisper-SLU utilises significantly fewer parameters (34.8% relative reduction ((2.3-1.5) / 2.3))) than the Modular-2 system. This indicates the robust language understanding capabilities of the Whisper decoder and the efficacy of end-to-end modelling in the ZS-Whisper-SLU system. From a training efficiency perspective, the zero-shot spoken language understanding functionalities in the ZS-Whisper-SLU system are achieved by optimising a prefix encoder with only 3.3 million parameters, which constitutes about 0.2% of the original Whisper model. Table 3: In-corpus evaluation experiment results in one example: Word error rate (WER) (%) and slot filling SLU-F1 (%) on the zero-shot test set of SLURP. The system architecture, trainable and total number of parameters of the systems are also provided, f indicates components in the end-to-end system. Model Architecture #Trainable Parameters #Total Parameters WER (%) SLU -F1 (%) Modular-1 ASR: Conformer-LSTM NLU: GPT-2-small =220M =220M - 10.1 KA2G ASRf: Conformer-LSTM NLUf: GPT-2-small =220M =220M 18.0 9.3 Modular-2 ASR: Whisper-large-v2 NLU: GPT-2-large 3.3M 2.3B 8.5 43.5 ZS-Whisper-SLU w / o ASR transcript w / o ASR state Whisper-large-v2 3.3M 1.5B 8.3 8.4 8.5 50.0 45.8 36.3 Tables 4 and 5 shows the cross-corpus zero-shot evaluation results for FSC and SmartLight respectively. In this experiment, to illustrate the robustness of ZS-Whisper-SLU, the ZS-Whisper-SLU system is compared with existing supervised training systems on both datasets. These existing systems include: • Whisper-SLU, disclosed in Meeus et al.’s “Whisper-SLU: Extending a pretrained speech-to-text transformer for low resource spoken language understanding” (2023) • Finstreder, disclosed in Bermuth et al.’s “Finstreder: Simple and fast spoken language understanding with finite state transducers using modern speech-to-text models” (2022) • Modular-2 (discussed above) The results shown in Tables 4 and 5 indicate that the ZS-Whisper-SLU system in this embodiment consistently achieves competitive performance when compared with the modular structure. The ZS-Whisper-SLU system performs comparably well relative to the supervised models, as it shows a slight difference of 4.7% on FSC and 5.5% (PP) on SmartLight, despite not being exposed to any in-corpus data during training. This indicates the ability of ZS-Whisper-SLU to effectively transfer knowledge from seen domains to unseen domains. Table 4: Cross-corpus evaluation experiment results in one example. WER (%) and intent classification accuracy (Acc.) (%) on the test set of FSC. System with * is supervised trained with labelled data. Model WER Acc. (%) Whisper-SLU* - 99.7 Modular-2 0.8 95.1 ZS-Whisper-SLU 0.8 95.0 Table 5: Cross-corpus evaluation experiment results in one example. WER (%), intent classification accuracy (Acc.) (%), slot filling SLU-F1 (%) and perfect-parsing (PP) (%) on the full set of SmartLight. Systems with * are supervised trained with labelled data. Model WER Acc. (%) SLU-F1 (%) PP (%) Finstreder* 6.1 - - 88.0 Whisper-SLU* - 95.4 - - Modular-2 2.8 91.9 90.8 82.0 ZS-Whisper-SLU 2.7 91.6 90.9 82.5 In this experiment, ablation study is also performed. The results are shown in the final row of Table 3. Specifically, in this study, to assess the key factor influencing the zeroshot performance of the ZS-Whisper-SLU in this embodiment, some components of the model are selectively pruned during the question-answering decoding process. In the first case (“w / o ASR transcript”), the automatic speech recognition transcript (i.e., the transcribed text) is not used in the question prompts. In this case, decoding relies solely on the intent / slot question and cached automatic speech recognition states, in particular decoder states, generated during transcription. In the second case (“w / o ASR state”), the automatic speech recognition states, in particular decoder states, are not used in the processing. This aligns with the modular approach, where the Whisper model’s decoder functions as an independent natural langue understanding module, processing textbased automatic speech recognition transcript (i.e., the transcribed text) and semantic questions in the prompts. As shown in Table 3, in the first case, a moderate decrease of 4.2% in SLU-F1 is observed. This suggests that including the transcribed text in the question prompts may help to stabilise semantics extraction by mitigating acoustic variance embedded in the automatic speech recognition states, in particular decoder states. On the other hand, in the second case, a significant performance drop of 13.7% in SLU-F1 is observed, as this case undermines the end-to-end decoding property of the Whisper model hence leads to inferior behaviour. These results suggest that in the ZS-Whisper-SLU system in this embodiment, the use of joint modelling on speech and text modalities can help to achieve optimal zero-shot performance. In the ZS-Whisper-SLU system in this embodiment, the Whisper model, a standalone speech processing model, is used for zero-shot end-to-end spoken language understanding. To handle unseen semantic labels, spoken language understanding tasks are integrated into a question-answering framework, which prompts the Whisper model’s decoder for semantics deduction. The system is efficiently trained with prefixtuning, which optimises a smaller or minimal set of parameters (rather than the parameters of the entire model). In this embodiment, to enable zero-shot functionalities, the spoken language understanding tasks are formatted or reformulated as questionanswering problems and the Whisper model is accordingly adapted through decoderprompting and prefix-tuning. Fig. 12 shows an example information handling system 1200 in one embodiment of the invention. The information handling system 1200 can be used to implement the computer-implemented operation, partly or entirely, in various embodiments of the invention. The information handling system 1200 generally comprises suitable components necessary to receive, store, and execute appropriate computer instructions, commands, and / or codes. The main components of the information handling system 1200 are a processor 1202 and a memory (storage) 1204. The processor 1202 may include one or more: CPU(s), MCU(s), GPU(s), logic circuit(s), Raspberry Pi chip(s), digital signal processor(s) (DSP), application-specific integrated circuit(s) (ASIC), field-programmable gate array(s) (FPGA), or any other digital or analog circuitry / circuitries configured to interpret and / or to execute program instructions and / or to process signals and / or information and / or data. In one example, the processor 1202 includes a CPU and a GPU. The memory 1204 may include one or more volatile memory (such as RAM, DRAM, SRAM, etc.), one or more non-volatile memory (such as ROM, PROM, EPROM, EEPROM, FRAM, MRAM, FLASH, SSD, NAND, NVDIMM, etc.), or any of their combinations. Appropriate computer instructions, commands, codes, information and / or data may be stored in the memory 1204. Computer instructions for executing or facilitating executing the operation in various embodiments of the invention may be stored in the memory 1204. Models, model parameters, etc., in various embodiments of the invention may be stored in the memory 1204. The processor 1202 and memory (storage) 1204 may be integrated or separated (and operably connected). Optionally, the information handling system 1200 further includes one or more input devices 1206. Example of such input device 1206 include: keyboard, mouse, stylus, image scanner, microphone, tactile / touch input device (e.g., touch sensitive screen), image / video input device (e.g., camera), etc. Optionally, the information handling system 1200 further includes one or more output devices 1208. Example of such output device 1208 include: display (e.g., monitor, screen, projector, etc.), speaker, headphone, earphone, printer, additive manufacturing machine (e.g., 3D printer), etc. The display may include a LCD display, a LED / OLED display, or other suitable display, which may or may not be touch sensitive. The information handling system 1200 may further include one or more disk drives 1212 which may include one or more of: solid state drive, hard disk drive, optical drive, flash drive, magnetic tape drive, etc. A suitable operating system may be installed in the information handling system 1200, e.g., on the disk drive 1212 or in the memory 1204. The memory 1204 and the disk drive 1212 may be operated by the processor 1202. Optionally, the information handling system 1200 also includes a communication device 1210 for establishing one or more communication links (not shown) with one or more other computing devices, such as servers, personal computers, terminals, tablets, phones, watches, loT devices, or other wireless computing devices. The communication device 1210 may include one or more of: a modem, a Network Interface Card (NIC), an integrated network interface, a NFC transceiver, a ZigBee transceiver, a Wi-Fi transceiver, a Bluetooth® transceiver, a radio frequency transceiver, a cellular (2G, 3G, 4G, LTE, 5G, 6G, or the like) transceiver, an optical port, an infrared port, a USB connection, or other wired or wireless communication interfaces. Transceiver may be implemented by one or more devices (integrated transmitter(s) and receiver(s), separate transmitter(s) and receiver(s), etc.). The communication link(s) may be wired or wireless for communicating commands, instructions, information and / or data. In one example, the processor 1202, the memory 1204 (optionally the input device(s) 1206, the output device(s) 1208, the communication device(s) 1210 and the disk drive(s) 1212, if present) are connected with each other, directly or indirectly, through a bus, a Peripheral Component Interconnect (PCI), such as PCI Express, a Universal Serial Bus (USB), an optical bus, or other like bus structure. In one embodiment, at least some of these components may be connected wirelessly, e.g., through a network, such as the Internet or a cloud computing network. A person skilled in the art would appreciate that the information handling system 1200 shown in Fig. 12 is merely an example and that the information handling system 1200 can have different configurations (e.g., include additional components, has fewer components, etc.) in other embodiments. Although not required, one or more embodiments of the invention can be implemented as an application programming interface (API) or as a series of libraries for use by a developer or can be included within another software application, such as a terminal or computer operating system or a portable computing device operating system. In one or more embodiments, as program modules include routines, programs, objects, components, and data files assisting in the performance of particular functions, the functionality of the software application may be distributed across a number of routines, objects and / or components to achieve the same functionality desired herein. It will be appreciated that where the methods and systems of the invention are either wholly implemented by computing system(s) or partly implemented by computing system(s) then any appropriate computing system architecture may be utilised. This will include stand-alone computers, network computers, dedicated or non-dedicated hardware devices. Where the terms “computing system” and “computing device” are used, these terms are intended to include any appropriate arrangement of computer or information processing hardware capable of implementing the function described. Embodiments of the invention may be practically applied in computer implemented tools such as chatbot or virtual agent for human-machine or human-human interaction (e.g., human-machine or human-human dialogue). Embodiments of the invention, as an area in computer science, may be inextricably linked to computing system(s). It will be appreciated by a person skilled in the art that variations and / or modifications may be made to the described and / or illustrated embodiments of the invention to provide other embodiments of the invention. The described and / or illustrated embodiments of the invention should therefore be considered in all respects as illustrative, not restrictive. For example, the model may include one or more artificial neural networks. For example, some embodiments of the invention may be applied to zero-shot end-to-end spoken language understanding. For example, some embodiments of the invention may be applied to perform more than one language understanding task or even more than two language understanding tasks. For example, the method in some embodiments of the invention may be applied to other language understanding task such as emotion recognition task, sentiment analysis task, named entity recognition task, etc., by suitably modifying the text-based representation of one or more labels of the input. For example, the method in some embodiments of the invention may be applied to perform spoken language understanding in a non-English language (such as German, French, Japanese, Chinese, etc.). Whilst certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel devices, and methods described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the devices, methods and products described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.

Claims

1. A computer-implemented method for performing spoken language understanding, comprising:receiving data representing an audio that comprises speech;processing the data using a model to determine text corresponding to content of the speech;receiving an input for performing a language understanding task, the input comprising text-based representation of one or more semantic labels;processing the input using at least part of the model to extract semantic information from the text corresponding to the content of the speech; andobtaining the extracted semantic information for the language understanding task.

2. The computer-implemented method of claim 1, wherein the model comprises a transformer-based model.

3. The computer-implemented method of claim 1 or 2, wherein:the model comprises an encoder and a decoder operably coupled with the encoder;the processing of the data is performed using the encoder and the decoder; and the processing of the input is performed using at least part of the decoder.

4. The computer-implemented method of claim 3, wherein:the encoder comprises a self-attention mechanism and a feedforward neural network;the decoder comprises a self-attention mechanism, a cross-attention mechanism, and a feedforward neural network;the processing of the data is performed using the self-attention mechanism and the feedforward neural network of the encoder and the self-attention mechanism, the cross-attention mechanism, and the feedforward neural network of the decoder; andthe processing of the input is performed using the self-attention mechanism and the feedforward neural network of the decoder, without using any cross-attention mechanism of the decoder.

5. The computer-implemented method of claim 3 or 4, further comprising:recording or storing states of the decoder obtained as a result of the processing of the data; andapplying the recorded or stored states of the decoder to facilitate the processing of the input.

6. The computer-implemented method of claim 5, wherein the states of the decoder represent acoustic features obtained by the encoder as a result of processing the data and textual dependencies obtained by the decoder as a result of processing the data.

7. The computer-implemented method of any one of claims 1 to 6, wherein the textbased representation of the one or more semantic labels comprises one or more questions each associated with a respective one of the semantic labels.

8. The computer-implemented method of any one of claims 1 to 7, wherein the input further comprises the text corresponding to the content of the speech.

9. The computer-implemented method of any one of claims 1 to 8, wherein: the language understanding task comprises an intent classification task; the one or more semantic labels comprises one or more intent labels; and the extracted semantic information comprises extracted intent information.

10. The computer-implemented method of claim 9, wherein the extracted intent information indicates respective positive or negative association with each respective intent label.

11. The computer-implemented method of any one of claims 1 to 8, wherein: the language understanding task comprises a slot filling task;the one or more semantic labels comprises one or more slot labels; and the extracted semantic information comprises one or more words extracted from the text and associated with at least one of the slot labels.

12. The computer-implemented method of any one of claims 1 to 10, further comprising:receiving a further input for performing a further language understanding task, the further input comprising text-based representation of one or more further semantic labels;processing the further input using at least part of the model to extract further semantic information from the text corresponding to the content of the speech; and obtaining the further extracted semantic information;wherein the further language understanding task is different from the language understanding task.

13. The computer-implemented method of claim 12, wherein the one or more further semantic labels are based on the extracted semantic information.

14. The computer-implemented method of claim 12 or 13, wherein the text-based representation of the one or more further semantic labels comprises one or more questions each associated with a respective one of the further semantic labels.

15. The computer-implemented method of any one of claims 12 to 14, wherein the further input further comprises the text corresponding to the content of the speech.

16. The computer-implemented method of any one of claims 12 to 15, wherein: the further language understanding task comprises a slot filling task;the one or more further semantic labels comprises one or more slot labels; andthe further extracted semantic information comprises one or more words extracted from the text and associated with at least one of the slot labels.

17. The computer-implemented method of any one of claims 12 to 16, wherein: the processing of the further input is performed using the at least part of the model used to process the input.

18. The computer-implemented method of any one of claims 1 to 17, wherein the data representing the audio comprises a visual representation of the audio.

19. The computer-implemented method of any one of claims 1 to 17, wherein the data representing the audio comprises an audio file.

20. The computer-implemented method of any one of claims 1 to 19, wherein the model has been trained based on prefix-tuning.

21. A system comprising one or more processors configured to perform the computer-implemented method of any one of claims 1 to 20.

22. A carrier medium carrying computer readable instructions adapted to cause one or more processors to perform the computer-implemented method of any one of claims 1 to 20.31

Images