Systems and methods for spoken language understanding

Abstract

To provide a system and method for performing utterance language understanding with fewer computing resources.SOLUTION: A computer implemented method for carrying out utterance language understanding executes: receiving data representing audio including a voice; processing data using a model for determining a text corresponding to a content of the voice; receiving input for carrying out a language understanding task including a presentation based on a text of one or more semantic labels; processing input using at least a portion of the model to extract semantic information from the text corresponding to the content of the voice; and acquiring the semantic information extracted in connection with the language understanding task.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION The embodiments described herein relate to systems and methods for spoken language understanding. [Background technology]

[0002] Spoken language understanding is a computer-implemented technology closely related to speech processing and natural language processing. For example, in traditional artificial intelligence, spoken language understanding aims to convert an utterance into semantic elements, such as a user's intent, entities, or emotions.

[0003] One type of spoken language understanding is zero-shot spoken language understanding, which is configured to understand user utterances in new domains without prior training using training data in those domains.

[0004] Existing systems for implementing zero-shot spoken language understanding are generally modular or end-to-end systems. Modular systems generally include a speech recognition model configured to transcribe speech into text and a separate natural language understanding model operable to process the text to create a semantic output. In modular systems, the two models are trained separately. End-to-end systems generally include a speech recognition model configured to transcribe speech into text and a natural language understanding model operable to process the text to create a semantic output. In end-to-end systems, the two models are connected and trained together, and speech input is directly mapped to a semantic output.

[0005] In these existing modular and end-to-end systems, the use of both speech recognition models and natural language understanding models in implementing zero-shot spoken language understanding can require relatively large amounts of computational resources (e.g., processing resources, memory consumption, etc.).

[0006] Embodiments of the present invention will now be described, by way of example, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a flowchart illustrating operations for performing spoken language understanding in one embodiment of the present invention. [Figure 2] FIG. 2 is a schematic diagram illustrating operations for performing spoken language understanding in one embodiment of the present invention. [Figure 3] FIG. 3 is a flowchart illustrating operations for performing spoken language understanding in one embodiment of the present invention. [Figure 4] FIG. 4 is a flowchart illustrating operations for performing spoken language understanding in one embodiment of the present invention. [Figure 5] FIG. 5 is a diagram illustrating operations for performing speech recognition in one embodiment of the present invention. [Figure 6] FIG. 6 is a diagram illustrating operations for performing language understanding in one embodiment of the present invention. [Figure 7] FIG. 7 is a diagram illustrating operations for performing speech recognition in one embodiment of the present invention. [Figure 8] FIG. 8 is a diagram illustrating operations for performing intent classification in one embodiment of the present invention. [Figure 9] FIG. 9 is a diagram illustrating operations for performing slot filling in one embodiment of the present invention. [Figure 10A] FIG. 10A is a diagram illustrating operations for performing speech recognition in one embodiment of the present invention. [Figure 10B] FIG. 10B is a diagram illustrating operations for performing intent classification and slot filling in one embodiment of the present invention. [Figure 11] FIG. 11 is a schematic diagram illustrating an example of a large-scale language model-based question generation process for intent and slot labels in one embodiment of the present invention. [Figure 12] FIG. 12 is a block diagram of an exemplary information processing system operable to perform operations in some embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0008] FIELD OF THE INVENTION Embodiments of the present invention relate to spoken language understanding, such as zero-shot end-to-end spoken language understanding.

[0009] In one embodiment, a computer-implemented method for performing spoken language understanding is provided. The computer-implemented method includes receiving data representing audio with speech, processing the data using a model to determine text corresponding to the content of the speech, receiving an input with a text-based representation of one or more semantic labels for performing a language understanding task, processing the input using at least a portion of the model to extract semantic information for the language understanding task from the text corresponding to the content of the speech, and obtaining the extracted semantic information for the language understanding task. Here, the model is used to process both the data and the input, i.e., at least a portion of the model is configured to perform both speech recognition and language understanding (instead of using one model to perform speech recognition and another, different model to perform language understanding). Processing the data (for speech recognition) using the model can facilitate or guide processing the input (for language understanding) using at least a portion of the model. In some cases, processing the input (for language understanding) may not require the use of any external language model. Advantageously, in some embodiments, the model may be more parameter-efficient and may be trained and / or operated more effectively (e.g., consuming fewer memory resources). Advantageously, in some embodiments, language understanding tasks, or spoken language understanding more generally, may be performed more effectively.

[0010] In some embodiments, the model is operable to perform zero-shot end-to-end spoken language understanding.

[0011] In some embodiments, the model comprises a Transformer-based model.

[0012] In some embodiments, the model comprises an encoder and a decoder operably coupled to the encoder.

[0013] In some embodiments, the processing of the data is performed using an encoder and a decoder.

[0014] In some embodiments, processing of the input is performed using at least a portion of (e.g., only a portion of) the decoder, which may provide an audio-conditioned language model that may be useful for processing the input (for language understanding) after being used to process the data (for speech recognition).

[0015] In some embodiments, the encoder comprises a self-attention mechanism and a feedforward neural network. The encoder may include one or more encoder layers. Each encoder layer may comprise a self-attention mechanism and a feedforward neural network. The encoder's feedforward neural network and / or the decoder's feedforward neural network may comprise a multi-layer perceptron.

[0016] In some embodiments, the decoder comprises a self-attention mechanism, a cross-attention mechanism, and a feed-forward neural network. The decoder may include one or more decoder layers. Each decoder layer may include a self-attention mechanism, a cross-attention mechanism, and a feed-forward neural network, respectively. The output of the encoder may be operatively coupled to the or each cross-attention mechanism of the decoder.

[0017] In some embodiments, processing of the data is performed using a self-attention mechanism and a feed-forward neural network in the encoder, and a self-attention mechanism, a cross-attention mechanism, and a feed-forward neural network in the decoder.

[0018] In some embodiments, the processing of the input is performed using a self-attention mechanism in the decoder and a feedforward neural network without using any cross-attention mechanism in the decoder. By not using cross-attention mechanisms in the encoder and decoder in processing the input, undesirable effects of speech-related speech embeddings on language understanding tasks may be reduced or minimized.

[0019] In some embodiments, the computer-implemented method further comprises recording or storing a plurality of states of the decoder obtained as a result of processing the data, and applying the recorded or stored plurality of states of the decoder to facilitate processing of the input.

[0020] The decoder states include parameters useful for processing the input, and in some embodiments, represent acoustic features obtained by the encoder as a result of processing the data and text dependencies obtained by the decoder as a result of processing the data.

[0021] In some embodiments, the text-based representation of the one or more semantic labels comprises one or more questions, each related to (e.g., formed based on) a respective one of the one or more semantic labels.

[0022] The input may be considered a prompt. In some embodiments, the input further comprises text corresponding to the audio content. For example, the input may include multiple entries, each comprising text corresponding to the audio content and a text-based representation of a respective semantic label. By including text corresponding to the audio content in the input, extraction of semantic information may be performed more effectively and / or more accurately.

[0023] In some embodiments, the language understanding task comprises an intent classification task, the one or more semantic labels comprise one or more intent labels, and the extracted semantic information comprises extracted intent information.

[0024] In some embodiments, the extracted intent information indicates a respective positive or negative association with each intent label. A positive association with an intent label may mean that the text matches or best matches the intent label. A negative association with an intent label may mean that the text does not match or best matches the intent label. As an example, text deemed to have intent A may match intent label “A” (and thus be positively associated with intent label “A”) and may not match intent labels “B” and “C” (and thus be negatively associated with intent labels “B” and “C”). A positive association with an intent label may be represented by an answer “yes,” and a negative association with an intent label may be represented by an answer “no.”

[0025] In some embodiments, the language understanding task comprises a slot-filling task, the one or more semantic labels comprise one or more slot labels, and the extracted semantic information comprises one or more words associated with at least one of the one or more slot labels extracted from the text.

[0026] In some embodiments, the computer-implemented method further comprises (e.g., after obtaining the extracted semantic information), receiving a further input comprising a text-based representation of one or more further semantic labels for performing a further language understanding task, processing the further input using at least a portion of the model to extract further semantic information from the text corresponding to the speech content, and obtaining the extracted further semantic information. The processing of the input (for language understanding) using at least a portion of the model can facilitate or guide the processing of the further input (for further language understanding) using at least a portion of the model.

[0027] In some embodiments in which the extracted semantic information comprises extracted intent information, the one or more further semantic labels are based on the extracted intent information. For example, the one or more further semantic labels available may depend on the extracted intent information.

[0028] In some embodiments, the text-based representation of the one or more further semantic labels comprises one or more questions, each related to (e.g., formed based on) a respective one of the one or more further semantic labels.

[0029] In some embodiments, the further input further comprises text corresponding to the audio content. For example, the further input may include multiple entries, each of which respectively comprises text corresponding to the audio content and a text-based representation of a respective further semantic label. By including text corresponding to the audio content in the further input, extraction of the further semantic information may be performed more effectively and / or more accurately.

[0030] In some embodiments, the further language understanding task comprises a slot-filling task, the one or more further semantic labels comprise one or more slot labels, and the extracted further semantic information comprises one or more words associated with at least one of the one or more slot labels extracted from the text.

[0031] In some embodiments, the processing of the further input is performed using at least part of the model used to process the input (i.e., the same part of the model).

[0032] In some embodiments, the processing of the further inputs is performed using at least a portion of the decoder (e.g., only the decoder portion) of the model. For example, the processing of the further inputs may be performed using a self-attention mechanism in the decoder and a feedforward neural network without using any cross-attention mechanism in the decoder. By not using cross-attention mechanisms in the encoder and decoder in processing the further inputs, undesirable effects of speech-related speech embeddings on further language understanding tasks may be reduced or minimized.

[0033] In some embodiments, the data representing the audio comprises a visual representation of the audio, for example, the visual representation may include an oscillogram or a spectrogram.

[0034] In some embodiments, the data representing the audio comprises an audio file, hi some embodiments, the model comprises a visual representation generator operable to process the audio file to create a visual representation (e.g., an oscillogram or spectrogram) of the audio.

[0035] In some embodiments, the model is trained based on prefix tuning, which allows the model to be adapted or trained in a parameter-efficient manner.

[0036] In some embodiments, the computer-implemented method further comprises outputting or presenting the obtained, extracted semantic information. For example, the obtained, extracted semantic information can be displayed on a display, played by a speaker, etc.

[0037] In some embodiments, the computer-implemented method further comprises outputting or presenting the obtained further extracted semantic information, for example, the obtained further extracted semantic information can be displayed on a display, played by a speaker, etc.

[0038] In some embodiments, the obtained extracted semantic information and / or the obtained further extracted semantic information may be used to facilitate or guide the performance of a task, to trigger or provide a response, to facilitate or guide the execution of a command, etc.

[0039] In some embodiments, the computer-implemented method may be applied to perform more than two language understanding tasks (ie, not just a language understanding task and a further language understanding task).

[0040] In one embodiment, a system is provided that includes one or more processors configured to perform the computer-implemented method described above. The system may be a zero-shot end-to-end spoken language understanding system.

[0041] In further embodiments, a carrier medium is provided that carries a plurality of computer-readable instructions adapted to cause one or more processors to perform the computer-implemented method of the first aspect. In some embodiments, the carrier medium is a transitory computer-readable medium (e.g., a signal, a wave, etc.). In some embodiments, the carrier medium is a non-transitory computer-readable medium, such as a non-transitory computer-readable storage medium.

[0042] In a further embodiment there is provided a computer program or computer program product comprising instructions which, when the computer program is executed by a computer, cause the computer to perform the computer-implemented method of the first aspect.

[0043] Other features and aspects will become apparent by consideration of the detailed description and accompanying drawings. A feature described herein with respect to one aspect or one embodiment may, where appropriate, be applied to any other aspect or embodiment (which may or may not be described herein). For example, a feature described herein with respect to one aspect or one embodiment may, where appropriate, be combined with any other feature described herein with respect to any other aspect or embodiment.

[0044] Terms of degree, such as "generally," "about," and "substantially," are used herein to account for manufacturing tolerances, degradation, trends, tendencies, imperfect actual conditions, and the like, depending on the context.

[0045] Unless otherwise specified, the terms "connected," "coupled," "attached," and the like, as used herein, are intended to encompass both direct and indirect connections, couplings, attachments, and the like.

[0046] 1 illustrates operations 100 for performing spoken language understanding in one embodiment of the present invention. Operations 100 are computer-implemented operations. In one example, operations 100 are for performing zero-shot spoken language understanding. In one example, operations 100 are for performing zero-shot end-to-end spoken language understanding.

[0047] The operation 100 includes, at 102, receiving data representing audio including speech. In one example, the data representing the audio includes a visual representation of the audio, such as an oscillogram or a spectrogram. In another example, the data representing the audio includes an audio file (digital file). In one example, the audio consists solely of speech. In another example, the audio includes speech as well as one or more other sounds (e.g., background noise, other noises, music, etc.).

[0048] The operations 100 further include, at 104, processing the data received at 102 using a model to determine text corresponding to the content of the audio. This process to determine the text may be considered speech recognition or automatic speech recognition. The model is a machine learning model, preferably a deep learning model. The model may be operable to perform zero-shot end-to-end spoken language understanding. In one example, the model may include a Transformer model. In one example, the model may include an encoder and a decoder operatively coupled to the encoder. In one example where the model includes an encoder and a decoder, processing the data at 104 is performed using both the encoder and the decoder. The model may be, or may have been, trained based on prefix tuning.

[0049] The operation 100 further includes, at 106, receiving input for performing a language understanding task. The input includes a text-based representation of one or more semantic labels. In one example, the text-based representation of the one or more semantic labels may include one or more questions, each associated with a respective semantic label. The input may further include text corresponding to the content of the speech (determined as a result of 104). For example, the input may include one or more entries, each including text corresponding to the content of the speech and a text-based representation of a respective semantic label. In some cases, the input may be considered a prompt. The language understanding task may include an intent classification task, a slot-filling task, an emotion recognition task, a sentiment analysis task, a named entity recognition task, etc. In an example where the language understanding task is an intent classification task, the one or more semantic labels include one or more intent labels. In an example where the language understanding task is a slot-filling task, the one or more semantic labels include one or more slot labels.

[0050] The operations 100 further include, at 108, processing the input received at 106 using at least a portion of the model (used in the processing at 104) to extract semantic information from text corresponding to the audio content (determined as a result of 104). This process of extracting semantic information may be considered language understanding (natural language understanding). Processing the data using the model at 104 may facilitate or guide processing of the input at 108 using at least a portion of the model. Thus, in some cases, processing the input at 108 may not require the use of any external language model. Also, the input may not require or include text corresponding to the audio content, because features related to the text corresponding to the audio content may be embedded within the parameters of the model (as a result of the processing at 104). In one example where the model includes an encoder and a decoder, processing of the input at 108 is performed using the decoder (without using the encoder).

[0051] The operation 100 further includes, at 110, obtaining extracted semantic information for the language understanding task. In an example where the language understanding task is an intent classification task, the extracted semantic information includes extracted intent information. For example, the extracted intent information may indicate a positive or negative association with each intent label. A positive association with an intent label may mean that the text matches or is a best match with the intent label (e.g., on a probability basis). A negative association with an intent label may mean that the text does not match or is not a best match with the intent label (e.g., on a probability basis). In an example where the language understanding task is a slot-filling task, the extracted semantic information includes slot information including one or more words associated with at least one of the multiple slot labels extracted from the text.

[0052] In one embodiment, the obtained, extracted semantic information is output or presented (e.g., via a display, speaker, etc.) In some cases, the obtained, extracted semantic information may be used by the computing device to facilitate or guide the performance of a task, to trigger or provide a response, to facilitate or guide the execution of a command, etc.

[0053] 2 illustrates operations 200 for performing spoken language understanding in one embodiment of the present invention. Operations 200 are computer-implemented operations. In one example, operations 200 are for performing zero-shot spoken language understanding. In one example, operations 200 are for performing zero-shot end-to-end spoken language understanding. Operations 200 may be considered an example implementation of operations 100, and therefore, for the sake of brevity, operations 200 will not be described in detail (as details may be found in this disclosure with respect to operations 100).

[0054] Briefly, in this embodiment, operation 200 includes receiving data representing audio with speech (similar to or the same as 102 in operation 100); processing the data using model M to determine text corresponding to the content of the speech (similar to or the same as 104 in operation 100); receiving and processing input (including at least a text-based representation of one or more semantic labels) for performing a language understanding task using portions of model M to extract semantic information from the text corresponding to the content of the speech (similar to or the same as 106 and 108 in operation 100); and obtaining semantic information related to the language understanding task (similar to or the same as 110 in operation 100). In this embodiment, model M includes an encoder and a decoder. Both the encoder and the decoder of model M are used in processing the data (for speech recognition). However, only the decoder of model M is used in processing the input (for language learning).

[0055] 3 illustrates operations 300 for performing spoken language understanding in one embodiment of the present invention. The operations 300 are computer-implemented operations. In one example, the operations 300 are for performing zero-shot spoken language understanding. In one example, the operations 300 are for performing zero-shot end-to-end spoken language understanding. In this embodiment, the operations 300 include operation 100, with 302 in operation 300 generally corresponding to (substantially the same or identical to) 102 in operation 100, 304 in operation 300 generally corresponding to (substantially the same or identical to) 104 in operation 100, 306 in operation 300 generally corresponding to (substantially the same or identical to) 106 in operation 100, 308 in operation 300 generally corresponding to (substantially the same or identical to) 108 in operation 100, and 310 in operation 300 generally corresponding to (substantially the same or identical to) 110 in operation 100. For the sake of brevity, operations 302-310 in operation 300 will not be described in detail (as details can be found in the present disclosure with respect to operations 102-110 in operation 100).

[0056] The operations 300 further include, at 312, receiving an additional input for performing an additional language understanding task (which may be the same as or different from the language understanding task). The additional input includes a text-based representation of one or more additional semantic labels. In one example, the text-based representation of the one or more additional semantic labels may include one or more questions, each associated with a respective additional semantic label. In one example, the available one or more additional semantic labels depend on the extracted semantic information (obtained in 310). The additional input may further include text corresponding to the content of the audio (determined as a result of 304). For example, the additional input may include one or more entries, each including text corresponding to the content of the audio and a text-based representation of a respective additional semantic label. In some cases, the additional input may be considered a prompt. The additional language understanding task may include an intent classification task, a slot-filling task, an emotion recognition task, a sentiment analysis task, a named entity recognition task, etc. In an embodiment in which the additional language understanding task is an intent classification task, the one or more additional semantic labels include one or more intent labels. In examples where the additional language understanding task is a slot-filling task, the one or more additional semantic labels include one or more slot labels.

[0057] The operations 300 further include, at 314, processing the additional input received at 312 using at least a portion of the model (used at 304) to extract additional semantic information from text corresponding to the audio content (determined as a result of 304). This processing to extract additional semantic information may be considered language understanding (natural language understanding). Processing the data using the model at 304 and processing the input using at least a portion of the model at 308 may facilitate or guide processing the additional input at 314 using at least a portion of the model. Thus, in some cases, processing the additional input at 314 may not require the use of any external language model. Also, because features related to the text corresponding to the audio content may be embedded within the parameters of the model (as a result of processing the data at 304 and processing the input using at least a portion of the model at 308), the additional input may not require or include text corresponding to the audio content. At least a portion of the model used in the processing at 314 may be the same as at least a portion of the model used in the processing at 308. In one example where the model includes an encoder and a decoder, the processing of the input at 314 is performed using the decoder (without using the encoder).

[0058] The operation 300 further includes, at 316, obtaining further extracted semantic information related to the further language understanding task. In an example where the further language understanding task is an intent classification task, the further extracted semantic information includes extracted intent information. For example, the further extracted intent information may indicate a positive or negative association with each intent label. A positive association with an intent label may mean that the text matches or is a best match with the intent label (e.g., on a probability basis). A negative association with an intent label may mean that the text does not match or is not a best match with the intent label (e.g., on a probability basis). In an example where the further language understanding task is a slot-filling task, the further extracted semantic information includes slot information including one or more words associated with at least one of the plurality of slot labels extracted from the text.

[0059] In one embodiment, the obtained further extracted semantic information is output or presented (e.g., via a display, speaker, etc.) In some cases, the obtained further extracted semantic information may be used by the computing device to facilitate or guide performance of a task, to trigger or provide a response, to facilitate or guide execution of a command, etc.

[0060] 4 illustrates operations 400 for performing spoken language understanding in one embodiment of the present invention. Operations 400 are computer-implemented operations. In one example, operations 400 are for performing zero-shot spoken language understanding. In one example, operations 400 are for performing zero-shot end-to-end spoken language understanding. Operations 400 may be considered an example implementation of operations 300, where operations 402-416 of operations 400 generally correspond to (are substantially the same as or identical to) operations 302-316 of operations 300, and the language understanding task at 406 is an intent classification task, and the further language understanding task at 412 is a slot-filling task.

[0061] 5 illustrates operations 500 for performing speech recognition in one embodiment of the present invention. Operations 500 are computer-implemented operations. Operations 500 may be considered an example implementation of operations 102-104 in operations 100, operations 302-304 in operations 300, or operations 402-404 in operations 400. As shown in FIG. 5, operations 500 are performed using a model including an encoder E and a decoder D. Encoder E includes multiple encoder layers E1 to E n (n is an integer greater than 1). n Each of the decoders D includes a self-attention module and a feedforward neural network. m (m is an integer greater than 1, which may be the same as or different from n). m Each of the encoders E includes a self-attention module, a cross-attention module, and a feedforward neural network. n The output of the decoder layers D1 to D m The cross-attention module may be operably coupled to the cross-attention module in each of the plurality of sub-sub ...units.

[0062] In operation 500, data representing speech-equipped audio, such as a spectrogram, is received and processed by an encoder E and a decoder D to transcribe the speech (i.e., to create text corresponding to the speech). The processing continues at each encoder layer E1 to E2. n The self-attention module and feedforward neural network of each decoder layer D1 to D mThe decoder D is implemented using a self-attention module, a cross-attention module, and a feed-forward neural network. As part of the processing, decoder D receives tokens as input to guide or facilitate the transcription of the speech. The tokens may include transcribed words and special tokens (e.g., special tokens as commands to generate a transcript, special tokens as indicators of the end of a sequence or transcription, etc.). A word being transcribed from the speech may be input to decoder D to guide the transcription of the next word from the speech. The states of decoder D obtained as a result of performing the transcription may be recorded or stored (e.g., in a cache) to facilitate subsequent processing. These states of decoder D may represent the acoustic features obtained by encoder E and the text dependencies obtained by decoder D as a result of performing the transcription.

[0063] 6 illustrates operations 600 for performing language understanding in one embodiment of the present invention. Operations 600 are computer-implemented operations. Operations 600 may be considered an example implementation of 106-110 in operations 100, 306-310 in operations 300, or 406-410 in operations 400. In this embodiment, operations 600 are implemented using decoder D used in operation 500.

[0064] In operation 600, input for performing a language understanding task is received by decoder D. In this embodiment, the input includes a speech transcribed text and a text-based representation of one or more semantic labels. Decoder D processes the input without using any of the cross-attention modules (i.e., each decoder layer D1 through D2) to extract semantic information from the speech transcribed text. m(Only the self-attention module and feedforward neural network in the decoder D are used.) In this embodiment, the recorded or stored states of the decoder D (obtained as a result of performing transcription) are applied to facilitate the processing of the input, thereby obtaining extracted semantic information relevant to the language understanding task.

[0065] The following description of Figures 7-9 relates to operations for implementing spoken language understanding in one embodiment.

[0066] 7 to 9 illustrate an example of an operation 700 for performing speech recognition in one embodiment (e.g., as an example of an operation 102-104 in operation 100, an example of an operation 302-304 in operation 300, an example of an operation 402-404 in operation 400, or an example of operation 500), an operation 800 for performing intent classification in one embodiment (e.g., as an example of an operation 106-110 in operation 100, an example of an operation 306-310 or an operation 312-316 in operation 300, an example of an operation 406-410 in operation 400, or an example of operation 600), and an operation 900 for performing slot filling in one embodiment (e.g., as an example of an operation 106-110 in operation 100, an example of an operation 306-310 or an operation 312-316 in operation 300, an example of an operation 412-416 in operation 400, or an example of operation 600), respectively. Operations 700 through 900 in these embodiments are performed based on the same model.

[0067] In the following, for ease of presentation, operation 700 and at least one of operations 800 and 900 are described as belonging to the same operation for performing spoken language understanding.

[0068] The inventors of the present invention have devised that a model with an encoder-decoder framework (with an encoder and a decoder) can be used for speech recognition, and that such a model, when properly trained, can be a robust, audio-tuned language model capable of performing complex language understanding tasks. In this embodiment, such a model operable for both speech recognition and natural language understanding is used to perform operation 700 and at least one of operations 800 and 900. The model in this embodiment does not include a separate natural language understanding model that uses a large-scale language model. In other words, the model in this embodiment does not use any external language model. In this example, the natural language understanding task is intent classification and slot filling (corresponding to operations 800 and 900).

[0069] In this embodiment, to enable zero-shot operation of the model, the natural language understanding task (intent classification and slot filling) is formulated as a question-answering problem. Semantic labels, such as intents and slot types, are converted into questions that express the semantic meaning of the labels. In this way, as new intents or slot types emerge, questions can be formulated accordingly to express their meaning without the need to further train the model on labeled data.

[0070] In this embodiment, during inference for natural language understanding, a question is input to the model's decoder to generate a relevant answer for the natural language understanding task.

[0071] The questions prepared for the intent and the questions prepared for the slot type may be in different formats. For example, for intent classification, the intent label "play_music" may be converted to the intent question "Does the user want to play music?", and the answer to the intent question may be binary (e.g., "yes" or "no"). For example, for slot filling, the slot type "date" may be converted to the slot question "What day is it?", and the answer to the slot question may be the slot value appearing in the query (e.g., the transcribed text) or a special token as end-of-sequence [EOS] (if there is no slot value associated with the question, e.g., none of the transcribed text corresponds to the answer to the question).

[0072] The general operations for implementing spoken language understanding in this embodiment are as follows: First, the model receives speech audio and transcribes the speech audio to obtain text corresponding to the speech. The state of the model's decoder obtained during or after transcription is stored or recorded. Prompts are then organized or formed for input to the model's decoder by attaching or concatenating the transcribed text to each semantic question (each associated with a respective semantic label). The model's decoder receives the prompts and processes them to generate task-specific answers based on the decoder's stored or recorded state. A more detailed description will now be presented with reference to Figures 7 through 9.

[0073] In FIG. 7 , operations 700 for performing speech recognition (as part of operations for performing spoken language understanding) are performed using a Transformer-based model including a Transformer encoder and a Transformer decoder. As shown in FIG. 7 , the Transformer encoder includes N Transformer encoder layers (N is an integer), each including a self-attention mechanism and a feed-forward neural network (FFN). Each of the Transformer encoder layers includes a layer input (only one is shown in FIG. 7 for simplicity). The Transformer decoder includes N Transformer decoder layers (N is an integer), each including a self-attention mechanism, a cross-attention mechanism, and a feed-forward neural network (FFN). Each of the Transformer decoder layers includes a layer input (only one is shown in FIG. 7 for simplicity). The output of the Transformer encoder is coupled to the cross-attention mechanism of the Transformer decoder layer.

[0074] In operation 700, the model's Transformer Encoder receives data representing speech audio (acoustic features) and processes the data to create speech embeddings. The model's Transformer Decoder, operably coupled to the model's Transformer Encoder, first generates transcribed text of the speech (e.g., words) based on the speech embeddings, starting with the [transcript] tokens. The generated transcribed text (e.g., words) is fed back to the Transformer Decoder to guide or facilitate the generation of further transcribed text of the speech (e.g., the next word). This process may be repeated until the speech transcription is complete. In this way, the Transformer Decoder utilizes previously generated tokens (e.g., tokens such as words of the transcribed text obtained in a previous decoding step) as input to serve as context for guiding the next decoding step, i.e., for guiding the decoding, and therefore the transcription. In this embodiment, for example, when the first word "buy" is transcribed from speech, the word "buy" is input back into the Transformer Decoder to serve as context for the next decoding step (i.e., to facilitate the transcription of the next word "a"). At the end of decoding (or transcription), the Transformer Decoder may generate an end-of-sequence [EOS] token to signal the completion of transcription. In this example, as shown in Figure 7, the transcribed text of the speech is "buy a concert ticket on Friday."

[0075] Reference is now made to Figure 8, which illustrates operations 800 for performing intent classification (as part of the operations for performing spoken language understanding). In this embodiment, operation 800 is performed after operation 700, using the Transformer-based model used in operation 700. Specifically, operation 800 is performed using a Transformer decoder of the model (without using a Transformer encoder and cross-attention mechanism in the decoder, to avoid using audio embeddings).

[0076] In operation 800, intent classification prompts (multiple input entries, each including the transcript "buy a concert ticket on Friday" and an intent question "Does the user want to [intent label]?" corresponding to an intent label) are input to a Transformer decoder. The [intent labels] used in this example are "book a ticket," "set an alarm," and "play music." In this embodiment, the Transformer decoder receives intent classification prompts associated with different intent labels together as a batch. The Transformer decoder processes these prompts to extract intent information (intent answers) from the transcribed text. In this embodiment, the intent answers are binary (e.g., either "yes" or "no"). In this embodiment, for the last token in the intent classification prompt, the probability of the token "yes" is selected from its vocabulary distribution. More specifically, the intent classification prompts received by the Transformer decoder are padded to have the same length. For each token (e.g., word and symbol) in the intent classification prompt, a probability distribution exists in the output layer (final Transformer Decoder layer) with a dimension of the vocabulary size. The distribution of the last token (e.g., "?") in each intent classification prompt is considered, and the probability of a "yes" token is selected from the distribution. The "yes" probabilities of the intent classification prompts are compared with each other. The intent classification prompt with the highest "yes" probability is selected, and the corresponding intent is determined as the predicted intent. Operation 800 is conditioned on the stored state of the model (specifically, the Transformer Decoder) obtained as a result of the transcription in operation 700.

[0077] Reference is now made to Figure 9, which illustrates operations 900 for performing slot filling (as part of the operations for performing spoken language understanding). In this embodiment, operation 900 is performed after operation 700 (operation 800 may not be performed) using the Transformer-based model used in operation 700. Specifically, operation 900 is performed using a Transformer decoder of the Transformer-based model (without using a Transformer encoder and cross-attention mechanism in the decoder to avoid using audio embeddings).

[0078] In operation 900, slot prompts (multiple input entries, each including the transcript "buy a concert ticket on Friday" and a slot-filling question "What is the [slot label]?" corresponding to a slot label) are input into a Transformer decoder. The [slot labels] used in this example are "ticket type," "alarm time," and "song name." In this embodiment, the Transformer decoder receives slot prompts associated with different slot labels together as a batch. The Transformer decoder processes these prompts to extract slot information (slot answers) from the transcribed text. In this embodiment, the slot answers are open-ended (not simply "yes" or "no") and include one or more words present in the transcribed text. In this embodiment, a slot answer or value is generated in a manner similar to the transcription in operation 700 (the generation is terminated by an [EOS] token). Operation 900 is conditioned on the stored state of the model (Transformer decoder) obtained as a result of the transcription in operation 700.

[0079] In this embodiment, prefix tuning is applied to train the model to enable zero-shot capability in operations 700 to 900.

[0080] In this embodiment, a series of randomly initialized vectors, or prefixes, are prepended to each decoder state ("ASR state") of the Transformer decoder layer to guide the model in producing the desired output.

[0081] In this embodiment, prefix vectors in the Transformer decoder are shared for speech recognition tasks (operation 700) and natural language understanding tasks (operations 800, 900). Similar prefix vectors are used to facilitate speech recognition adaptation in the Transformer encoder. In this embodiment, the model remains frozen (model parameters such as weights remain unchanged) during training.

[0082] To verify the performance of the operations for implementing spoken language understanding in this embodiment (based on operations 700 to 900), experiments were conducted under intra-corpus and cross-corpus settings.

[0083] In the experiments, the Whisper model (also called "Whisper SLU" with Whisper-large-v2 architecture) disclosed in Radford et al.'s "Robust speech recognition via large-scale weak supervision" (2023) is used as a model for performing spoken language understanding based on operations 700 to 900, and a modular system (also called "modular ASR-NLU") including an automatic speech recognition (ASR) model (Whisper-large-v2) and a natural language understanding (NLU) model (GPT2-large) is used as a baseline system (for comparison with the Whisper model) for performing spoken language understanding operations. In this example, the GPT2-large model is used to implement the natural language understanding model in the baseline system because it has a similar size to, and is therefore comparable to, the decoder in Whisper-large-v2.

[0084] For the intra-corpus experiments, the Whisper model is prefix-tuned to the training set SLURP (containing 64 intents and 50 slot types) and tested on a test set containing 5 unseen slots. For the cross-corpus experiments, the Whisper model prefix-tuned to SLURP is then tested on FSC (containing 15 intents). On the other hand, both models of the baseline system are trained using the prefix-tuning strategy. Exemplary details of the prefix-tuning strategy can be found, for example, in Li et al., "Prefix-Tuning: Optimizing Continuous Prompts for Generation" (2021).

[0085] The results of the experiment in this example are as follows:

[0086] Table 1: Intracorpus experiments yield an example: Zero-shot slot-filling SLU-F1 on the test set of SLURP [Table 1]

[0087] Table 2: Experiments across corpora yield an example: Zero-shot intent classification accuracy for FSC [Table 2]

[0088] As can be seen from Tables 1 and 2, for both the within-corpus and across-corpus experiments, the operations implemented using the Whisper model achieve similar or better zero-shot results compared to the operations implemented using the modular system. The modular system's natural language understanding model, the GPT2-large model, is pre-trained on large amounts of text data and is considered to have strong language understanding capabilities. However, these results show that the Whisper model demonstrates equally good (if not better) zero-shot skills in performing spoken language understanding using substantially fewer model parameters, which is advantageous in some practical applications.

[0089] Operations 700 through 900 implemented using the model in this embodiment can perform zero-shot spoken language understanding by feeding prompts to the model's decoder, eliminating the need for any external language model. Also, the model in this embodiment can be trained with a prefix tuning strategy and therefore can be trained efficiently.

[0090] In this embodiment, a model with an encoder-decoder structure is used in operation to perform zero-shot end-to-end spoken language understanding, including speech recognition (such as intent detection / classification and slot filling) and natural language understanding, without relying on an external language model. In this embodiment, zero-shot functionality (generalization to unseen intents and slots (not seen in training)) can be achieved through a prompt-based method applied to the model's decoder. In this embodiment, the model is better adapted to spoken language understanding tasks by employing an efficient prefix tuning strategy. In this embodiment, the model is parameter-efficient and may consume fewer computational resources during inference than other existing systems / models.

[0091] The following description of Figures 10A and 10B relates to operations for implementing spoken language understanding in one embodiment.

[0092] 10A and 10B illustrate, respectively, an operation 1000A for performing speech recognition in one embodiment (e.g., as an example of 102-104 in operation 100, an example of 302-304 in operation 300, an example of 402-404 in operation 400, or an example of operation 500) and an operation 1000B for performing intent classification and slot filling in one embodiment (e.g., as an example of 306-316 in operation 300, or an example of 406-416 in operation 400). These operations 1000A and 1000B are performed based on the same model.

[0093] In the following, for ease of presentation, operations 1000A and 1000B are described as belonging to the same operation for performing spoken language understanding.

[0094] In this embodiment, operations 1000A and 1000B are performed using a Whisper model embedded in a prompt-based question-answering framework (with a specifically designed semantic question generation method), where the Whisper model is prefix-tuned for multitasking speech recognition and zero-shot end-to-end spoken language understanding.

[0095] The Whisper model used in this embodiment is an open-source speech processing model developed by OpenAI, Inc. Exemplary details of the model can be found in Radford et al., "Robust speech recognition via large-scale weak supervision" (2023). Briefly, the Whisper model was trained on 68,000 hours of multilingual and multitask supervised data sourced from the internet. This diverse dataset contributes to enhanced robustness, enabling the Whisper model to effectively handle variations in accents, background noise, and technical language. The Whisper model has an end-to-end architecture built on an encoder-decoder transformer (comprising an encoder and a decoder). The encoder processes log-mel spectrograms of 30-second audio chunks as input. Depending on the resulting speech embeddings and user-provided tokens (e.g., special tokens), the decoder can perform tasks such as speech transcription, language identification, or X-English translation. As shown in Figure 10A, the encoder includes L encoder layers (L is an integer), each of which includes a self-attention mechanism and a feed-forward neural network (FFN). Each of the encoder layers includes a layer input (only one is shown in Figure 10A for simplicity). The decoder includes L decoder layers (L is an integer), each of which includes a self-attention mechanism, a cross-attention mechanism, and a feed-forward neural network (FFN). Each of the decoder layers includes a layer input (only one is shown in Figure 10A for simplicity). The encoder output is coupled to the cross-attention mechanism of the decoder layer.

[0096] The inventors of the present invention have found that information embedded within the decoding prompts of the model can affect the model output. Through their research, experiments, and trials, the inventors of the present invention have devised that the Whisper decoder can be utilized as a robust audio-trained language model and may therefore be suitable for zero-shot spoken language understanding.

[0097] A prompt-based question-answering framework for spoken language understanding in this embodiment is then presented.

[0098] In this embodiment, given an utterance X, the goal is to identify its intent from a set of candidates I and to find all entities in X that correspond to a set of slot types S. The inventors of the present invention have observed that in traditional supervised learning methods, these tasks are typically treated as utterance or word-level classification problems. The inventors have found that this approach lacks flexibility in accommodating the introduction of new elements into I and S without further training the classifier (a scenario commonly known as zero-shot learning).

[0099] To address this issue, this embodiment provides a framework called "ZS-Whisper-SLU," a Whisper-based generative question-answering framework for zero-shot intent classification and slot filling. This framework and the workflow of "ZS-Whisper-SLU" are illustrated in Figures 10A and 10B.

[0100] In this embodiment, the question set Q I and Q S To construct I and S, descriptive questions are generated for each element in I and S. The task is carried out in three stages.

[0101] Stage 1 relates to speech recognition (ASR), the operation of which is illustrated in Figure 10A. In this example, the transcription operation is based on a standard transcription procedure implemented using the Whisper model (where both an encoder and decoder are used), which requires the use of tokens as described above. In this embodiment, the decoder states (speech recognition decoder states, i.e., "ASR states") are cached or stored. These states encompass both acoustic features from the encoder and text dependencies captured in the decoder. In this example, the transcribed text of the speech is "play bbc radio."

[0102] Stage 2 concerns intent classification, the operation of which is illustrated in Figure 10B. In this example, the best transcript obtained from Stage 1, i.e., "play bbc radio", is replicated and I The text prompts are combined with all questions in the sequence. The resulting sequences (intent prompts) are organized into mini-batches and fed directly into the Whisper model's decoder to generate binary answers "yes" or "no" in parallel (cross-attention between the Whisper model's encoder and decoder is not used). In this example, both plaintext answers are tokenized using the Whisper model's own vocabulary so that no additional verbalizer (binary classifier) ​​is required. Because the intent questions in these text prompts do not have paired speech embeddings, the cross-attention modules in each decoder layer are omitted (they are not used in the intent classification process). In this example, in addition to using the transcribed text in the prompts, intent answer generation also relies on cached decoder states for end-to-end model formation.

[0103] Stage 3 relates to intent-constrained slot filling, the operation of which is illustrated in Figure 10B. The decoding process in stage 3 is similar to that of stage 2 (encoder and decoder cross-attention is also not used in the process). In this example, however, Q S Instead of feeding all questions in to the Whisper model's decoder, only questions that are determined (e.g., by a computer) to be relevant to the intent identified in stage 2 are selected (to be input to and processed by the decoder). This can help reduce computational costs (e.g., reduce required processing power, reduce memory consumption, etc.) and reduce or minimize irrelevant and confusing slot types (e.g., the slot type "artist" may not be related to the intent "takeaway_order"). In this example, the slot answers are open-ended and correspond to the entities mentioned in X. If an entity does not exist for a certain slot type, the system can create an [EOT] (end of transcript) symbol.

[0104] Q I and Q S Given a potential size of , this embodiment employs contrastive learning for each utterance for efficient training. This involves generating mini-batches by combining positive questions representing the ground truth intent and slots with N randomly sampled negative questions for both meanings. In this embodiment, the system is optimized using cross-entropy error, with prepared answers serving as labels. During inference, the overall Q IThe set is prompted to the Whisper model, and the question with the highest likelihood of a "yes" answer is selected to become the predicted intent. Non-empty answers to the selected slot questions are collected as entity outputs. If multiple slot types result in the same answer, the one with the highest probability is selected. Within the framework in this embodiment, new intents and slot types are generated by assigning those questions to Q I and Q S can be easily implemented by adding

[0105] In this embodiment, prefix tuning is applied as a parameter-efficient fine-tuning technique for adapting deep neural networks. In prefix tuning, a relatively small amount (e.g., a minimal amount) of additional parameters (typically less than 1%) is optimized for the existing model. Therefore, compared with the traditional full fine-tuning strategy, a better adaptation effect can be achieved.

[0106] As illustrated in Figures 10A and 10B, a sequence of trainable prefix vectors is concatenated with speech / text embeddings at each encoder / decoder layer to drive the Whisper model to produce reliable automatic speech recognition (ASR) transcripts and desired spoken language understanding (SLU) answers. Specifically, these vectors are injected into the key K and value V elements required in the Whisper model's self-attention module:

[0107]

number

[0108] In the formula, p K and p Vdenotes the prefix vectors (implemented as embedding layers) generated by the prefix encoder, which serve as soft context information for the ongoing encoding / decoding process. In one example, the prefix vectors are shared across the speech recognition and spoken language understanding tasks at each decoder layer. This facilitates the exchange of shared knowledge, such as entity names.

[0109] In this embodiment of the system, prefix vectors can help adapt the Whisper model to the acoustic characteristics of a spoken language understanding corpus, extend language modeling to task-specific domains, address text standardization issues such as capitalization and punctuation in speech recognition transcripts, and guide the Whisper model's decoder in creating intents and slot answers that are tailored to the speech recognition decoder state. In this embodiment, the adoption of prefix tuning preserves the capabilities of the original Whisper model, thus ensuring flexibility for other use cases.

[0110] The inventors of the present invention have realized that high-quality questions (for input to a model) are particularly useful for more accurately extracting semantic arguments within a question-and-answer-based natural language understanding framework or spoken language understanding framework. The inventors of the present invention have further realized that some existing question design methods are prone to human bias and may generalize poorly. For example, the SLURP corpus contains an intent classification labeled "convert_datetime," in which a user asks to convert a date or time between different time zones. In this case, it is clear that the true intent is not explicitly conveyed in the label. Therefore, directly converting such labels or names into semantic questions may create misleading prompts, which may adversely affect the implementation of natural language understanding or spoken language understanding.

[0111] The inventors of the present invention have therefore devised that a large-scale language model can be utilized to generate semantic questions and provide both label names and a small number of user utterance examples to mitigate naming bias. In this embodiment, the large-scale language model GPT-3.5 is used. Exemplary details of the GPT-3.5 model can be found in Brown et al., "Language models are few-shot leaners" (2020). As shown in FIG. 11, in this example, the GPT-3.5 model is first instructed to create a concise description for each semantic label using an example prompt template. The output description is then converted into a fixed-format question. In this example, enhanced stability can be observed by incorporating a two-stage question generation process (compared to prompting the GPT-3.5 model for single-shot question generation).

[0112] Experiments were conducted to evaluate the performance of the ZS-Whisper-SLU framework in this embodiment.

[0113] In this example, the experiments evaluated the ZS-Whisper-SLU framework on three datasets under intra-corpus and cross-corpus settings, and an ablation study was also conducted.

[0114] The first dataset used is SLURP (see, for example, Bastianelli et al., "SLURP: A spoken language understanding resource package," 2020). The SLURP dataset contains 65 intents and 56 slots within the domain of in-home personal robotic assistants. In this example, a new data split is created for zero-shot evaluation on five randomly selected slot types: {podcast_name, artist_name, audiobook_name, business_name, radio_name}. This adapted dataset is then utilized for intra-corpus evaluation.

[0115] The second dataset used is FSC (see, for example, Lugosch et al., "Speech model pre-training for end-to-end spoken language understanding," 2019). The FSC dataset is a widely used voice assistant corpus featuring spoken commands labeled with {action, object, location} arguments. These argument values ​​are drawn from a predefined set, constituting a total of 31 intents. In this example, these are reorganized into 15 intents and 2 slots ({language, location} to accommodate the [None] value), and the results are converted back to the original format for evaluation. This FSC dataset is used for evaluation across the corpus.

[0116] The third dataset used is SmartLight (close-field) (see, for example, Saade et al., "Spoken language understanding on the edge," 2019). The SmartLight dataset is a subset of the SNIPS SLU benchmark that contains instructions for controlling smart lights. This dataset has six intents and three slots to cover light functions and attributes. This SmartLight dataset is also used for corpus-wide evaluation.

[0117] In this example, the ZS-Whisper-SLU framework uses Whisper-large-v2 (with 1.5 billion parameters) as the underlying speech recognition model for zero-shot spoken language understanding adaptation.

[0118] The ZS-Whisper-SLU framework using Whisper-large-v2 is compared against three benchmark systems: Module 1: A modular system disclosed in Sun et al.'s "Knowledge-aware audio-grounded generative slot filling for limited annotated data" (2023), which includes a conformer LSTM (long short-term memory) speech recognition module and a GPT-2 natural language understanding module. KA2G: The knowledge-aware audio grounded (KA2G) system disclosed in Sun et al.'s "Knowledge-aware audio-grounded generative slot filling for limited annotated data" (2023), which is an end-to-end counterpart of the Modular 1 system, with a speech recognition module and a natural language understanding module connected through a neural aligner at the decoder state level. Modular 2: A modular system developed by the inventors for use in this experiment, including a Whisper speech recognition module and a GPT-2 natural language understanding module. In the Modular 2 system, for a fair comparison, the speech recognition module also utilizes Whisper-large-v2, and the natural language understanding module employs GPT-2-large (with 800 million parameters), which shares a similar architecture with the speech recognition decoder. Both modules are prefix-tuned using the same hyperparameters as the proposed system.

[0119] For intra-corpus experiments conducted using SLURP, the GPT-2 modules of ZS-Whisper-SLU in this embodiment, as well as the Modular 1, KA2G, and Modular 2 systems, are trained using N = 10 negative intents and slot examples for each utterance. The prefix length is set to 10 for each task, resulting in 10 prefix vectors at each encoder layer (for speech recognition) and 30 prefix vectors at each decoder layer (for joint speech recognition, intent classification, and slot filling). All modules (in this system and the benchmark system) undergo 10 training epochs using the AdamW optimizer with a mini-batch size of 12. A linear learning rate scheduler is employed for weight decay, starting from 0.002 without warm-up.

[0120] For corpus-wide experiments, the SLURP-developed model is directly applied to corpus-wide evaluations on FSC and SmartLight. However, the encoder prefix vector is removed to prevent the transfer of acoustic knowledge. For this experiment, the official test sets of all utterances in FSC and SmartLight (which lack data splitting) are utilized. All transcripts and semantic answers are generated using greedy-search decoding for fast inference.

[0121] The results of this example experiment are as follows: Table 3 shows the in-corpus speech recognition and slot-filling results for the SLURP zero-shot test set. We see that the ZS-Whisper-SLU system outperforms both the Modular 1 and KA2G systems in both metrics. Armed with the Whisper model's speech recognition capabilities, the ZS-Whisper-SLU system achieves a substantial 9.7% reduction in word error rate (WER) compared to KA2G. This further yields an absolute 40.7% improvement in the SLU-F1 score for first-sight slots. ZS-Whisper-SLU also demonstrates superiority over the Modular 2 system, which incorporates the GPT-2-large large-scale language model for natural language understanding, with a relative SLU-F1 gain of 14.9% ((50 - 43.5) / 43.5). In addition to enhanced performance, ZS-Whisper-SLU utilizes significantly fewer parameters than the Modular 2 system (a 34.8% relative reduction ((2.3 - 1.5) / 2.3)). This demonstrates the robust language understanding capabilities of the Whisper decoder and the effectiveness of end-to-end model formation in the ZS-Whisper-SLU system. In terms of training efficiency, the zero-shot spoken language understanding functionality of the ZS-Whisper-SLU system is achieved by optimizing the prefix encoder with only 3.3 million parameters, which constitute approximately 0.2% of the original Whisper model.

[0122] Table 3: Intra-corpus evaluation experiments provide an example: Word Error Rate (WER) (%) on the zero-shot test set of SLURP and slot-filling SLU-F1 (%). The system architecture, trainable parameters of the system, and total number of parameters are also provided. † denotes components within the end-to-end system. [Table 3]

[0123] Tables 4 and 5 show the zero-shot evaluation results across the corpora for FSC and SmartLight, respectively. In this experiment, to illustrate the robustness of ZS-Whisper-SLU, the ZS-Whisper-SLU system is compared with existing supervised training systems on both datasets. These existing systems include: Whisper-SLU, as disclosed in Meeus et al.'s "Whisper-SLU: Extending a pretrained speech-to-text transformer for low-resource spoken language understanding" (2023) Finstreder, as disclosed in Bermuth et al., "Finstreder: Simple and fast spoken language understanding with finite state transducers using modern speech-to-text models" (2022) Modular 2 (discussed above)

[0124] The results shown in Tables 4 and 5 indicate that the ZS-Whisper-SLU system in this embodiment consistently achieves competitive performance when compared with modular structures. Despite not being exposed to any in-corpus data during training, the ZS-Whisper-SLU system performs as well as supervised models, showing a slight difference of 4.7% for FSC and 5.5% for SmartLight (PP). This demonstrates the ability of ZS-Whisper-SLU to effectively transfer knowledge from known to unseen domains.

[0125] Table 4: Evaluation experiments across the corpus yield an example. WER (%) and intent classification accuracy (Acc.) (%) on the test set of FSC. Systems with * were trained in a supervised manner using labeled data. [Table 4]

[0126] Table 5: Evaluation experiments across the corpus yield an example. WER (%), intent classification accuracy (Acc.) (%), slot-filling SLU-F1 (%), and perfect parsing (PP) (%) for the complete SmartLight set. Systems with * are supervised trained using labeled data. [Table 5]

[0127] In this experiment, an ablation study was also conducted. The results are shown in the last row of Table 3. Specifically, in this study, several components of the model were selectively removed during the question-answer decoding process to determine the main factors affecting the zero-shot performance of ZS-Whisper-SLU in this embodiment. In the first case ("No ASR Transcript"), the speech recognition transcript (i.e., the transcribed text) was not used in the question prompt. In this case, decoding relies only on the intent / slot question and cached speech recognition state, specifically the decoder state, generated during transcription. In the second case ("No ASR State"), the speech recognition state, specifically the decoder state, was not used in the processing. This is consistent with a modular approach in which the decoder of the Whisper model functions as an independent natural language understanding module that processes the text-based speech recognition transcript (i.e., the transcribed text) and the semantic question in the prompt.

[0128] As shown in Table 3, in the first case, a moderate decrease of 4.2% is observed in SLU-F1. This suggests that including transcribed text in the question prompt can help stabilize semantic extraction by reducing the acoustic variance embedded in the speech recognition state, specifically the decoder state. On the other hand, in the second case, a significant performance decrease of 13.7% is observed in SLU-F1 because it degrades the end-to-end decoding properties of the Whisper model and thus leads to lower performance. These results suggest that in this embodiment of the ZS-Whisper-SLU system, the use of joint model building for the speech and text modalities can help achieve optimal zero-shot performance.

[0129] In this embodiment, the ZS-Whisper-SLU system uses the Whisper model, a standalone speech processing model, for zero-shot end-to-end spoken language understanding. To process first-sight semantic labels, the spoken language understanding task is embedded in a question-answering framework, prompting the Whisper model's decoder for semantic inference. The system is efficiently trained using prefix tuning, which optimizes a smaller or minimal set of parameters (rather than model-wide parameters). In this embodiment, to enable zero-shot functionality, the spoken language understanding task is formatted or reformulated as a question-answering problem, and the Whisper model is adapted accordingly through decoder prompts and prefix tuning.

[0130] 12 illustrates an exemplary information processing system 1200 in accordance with one embodiment of the present invention. The information processing system 1200 may be used to partially or entirely implement computer-implemented operations in various embodiments of the present invention.

[0131] The information processing system 1200 generally comprises suitable components necessary to receive, store, and execute appropriate computer instructions, commands, and / or code. The main components of the information processing system 1200 are a processor 1202 and memory (storage) 1204. The processor 1202 may include one or more of a CPU, an MCU, a GPU, a logic circuit, a Raspberry Pi chip, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or any other digital or analog circuit configured to interpret and / or execute program instructions and / or process signals and / or information and / or data. In one example, the processor 1202 includes a CPU and a GPU. The memory 1204 may include any one or more volatile memories (such as RAM, DRAM, SRAM), one or more non-volatile memories (such as ROM, PROM, EPROM, EEPROM, FRAM, MRAM, FLASH, SSD, NAND, NVDIMM, etc.), or a combination thereof. Suitable computer instructions, commands, code, information, and / or data may be stored in the memory 1204. Computer instructions for performing or facilitating the performance of operations in various embodiments of the present invention may be stored in the memory 1204. Models, model parameters, etc. in various embodiments of the present invention may be stored in the memory 1204. The processor 1202 and the memory (storage) 1204 may be integrated or separate (but operatively connected).

[0132] Optionally, the information processing system 1200 further includes one or more input devices 1206. Examples of such input devices 1206 include a keyboard, a mouse, a stylus, an image scanner, a microphone, a haptic / touch input device (e.g., a touch-sensitive screen), an image / video input device (e.g., a camera), etc. Optionally, the information processing system 1200 further includes one or more output devices 1208. Examples of such output devices 1208 include a display (e.g., a monitor, a screen, a projector, etc.), speakers, headphones, earphones, a printer, an additive manufacturing machine (e.g., a 3D printer), etc. The display may include an LCD display, an LED / OLED display, or other suitable display that may or may not be touch-sensitive. The information processing system 1200 may further include one or more disk drives 1212, which may include one or more of a solid-state drive, a hard disk drive, an optical drive, a flash drive, a magnetic tape drive, etc. A suitable operating system may be installed in information processing system 1200, for example, on disk drive 1212 or in memory 1204. Memory 1204 and disk drive 1212 may be operated by processor 1202. Optionally, information processing system 1200 also includes a communication device 1210 for establishing one or more communication links (not shown) with one or more other computing devices, such as a server, a personal computer, a terminal, a tablet, a phone, a watch, an IoT device, or other wireless computing device. Communication device 1210 may include one or more of a modem, a network interface card (NIC), an integrated network interface, an NFC transceiver, a ZigBee transceiver, a Wi-Fi transceiver, a Bluetooth transceiver, a radio frequency transceiver, a cellular (2G, 3G, 4G, LTE, 5G, 6G, etc.) transceiver, an optical port, an infrared port, a USB connection, or other wired or wireless communication interface.The transceiver may be implemented by one or more devices (e.g., an integrated transmitter and receiver, separate transmitters and receivers, etc.). The communication link may be wired or wireless for communicating commands, instructions, information, and / or data. In one example, the processor 1202, memory 1204 (and optionally, if present, the input device 1206, the output device 1208, the communication device 1210, and the disk drive 1212) are connected to each other directly or indirectly through a bus, a Peripheral Component Interconnect (PCI) such as PCI Express, a Universal Serial Bus (USB), an optical bus, or other similar bus structure. In one embodiment, at least some of these components may be connected wirelessly, for example, through a network such as the Internet or a cloud computing network. Those skilled in the art will appreciate that the information processing system 1200 shown in FIG. 12 is merely an example, and that in other embodiments, the information processing system 1200 may have a different configuration (e.g., may include additional components, may have fewer components, etc.).

[0133] Although not required, one or more embodiments of the present invention may be implemented as an application programming interface (API) or as a set of libraries for use by developers, or may be included within another software application, such as a terminal or computer operating system, or a portable computing device operating system. In one or more embodiments, program modules include routines, programs, objects, components, and data files that help implement particular functions, and thus the functionality of a software application may be distributed across several routines, objects, and / or components to achieve the same functionality as desired herein.

[0134] Whether the methods and systems of the present invention are implemented entirely or partially by a computing system, it will be appreciated that any suitable computing system architecture may be utilized. This will include stand-alone computers, network computers, dedicated or non-dedicated hardware devices. When the terms "computing system" and "computing device" are used, these terms are intended to include any suitable configuration of computer or information processing hardware capable of implementing the described functionality.

[0135] Embodiments of the present invention may have practical applications in computer-implemented tools such as chatbots or virtual agents for human-machine or human-human interactions (e.g., human-machine or human-human conversations). As a field in computer science, embodiments of the present invention may be closely linked to computing systems.

[0136] Those skilled in the art will appreciate that changes and / or modifications may be made to the described and / or illustrated embodiments of the present invention to provide other embodiments of the present invention. The described and / or illustrated embodiments of the present invention should therefore be considered in all respects as illustrative and not restrictive. For example, the model may include one or more artificial neural networks. For example, some embodiments of the present invention may be applied to zero-shot end-to-end spoken language understanding. For example, some embodiments of the present invention may be applied to perform two or more language understanding tasks or even three or more language understanding tasks. For example, the method in some embodiments of the present invention may be applied to other language understanding tasks, such as emotion recognition tasks, sentiment analysis tasks, named entity recognition tasks, etc., by suitably modifying the text-based representation of one or more labels of the input. For example, the method in some embodiments of the present invention may be applied to perform spoken language understanding of languages ​​other than English (e.g., German, French, Japanese, Chinese, etc.).

[0137] While several embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the invention. Indeed, the novel devices and methods described herein may be embodied in a variety of other forms, and various omissions, substitutions, and changes may be made in the forms of the devices, methods, and products described herein without departing from the spirit of the invention. The accompanying claims and their equivalents are intended to cover all such forms or modifications that fall within the scope and spirit of the invention.

Claims

1. 1. A computer-implemented method for performing spoken language understanding, comprising: receiving data representing audio with speech; processing the data using a model to determine text corresponding to the audio content; receiving an input for performing a language understanding task, the input being at least one of an intent classification task or a slot-filling task, the input comprising a text-based representation of one or more semantic labels; processing the input using at least a portion of the model to extract semantic information from the text that corresponds to the content of the speech for the language understanding task; obtaining the extracted semantic information for the language understanding task; Equipped with if the language understanding task is an intent classification task, the extracted semantic information comprises extracted intent information; and if the language understanding task is a slot-filling task, the extracted semantic information comprises one or more words associated with at least one of one or more slot labels extracted from the text as the semantic label.

2. The computer-implemented method of claim 1 , wherein the model comprises a Transformer-based model.

3. the model comprises an encoder and a decoder operatively coupled to the encoder; said processing of said data is performed using said encoder and said decoder; said processing of said input is performed using at least a portion of said decoder; The computer-implemented method of claim 1 .

4. the encoder comprises a self-attention mechanism and a feedforward neural network; the decoder comprises a self-attention mechanism, a cross-attention mechanism, and a feed-forward neural network; the processing of the data is performed using the self-attention mechanism and the feed-forward neural network in the encoder, and the self-attention mechanism, the cross-attention mechanism, and the feed-forward neural network in the decoder; the processing of the input is performed using the self-attention mechanism of the decoder and the feedforward neural network without using any cross-attention mechanism of the decoder; The computer-implemented method of claim 3 .

5. recording or storing a plurality of states of the decoder obtained as a result of said processing of said data; applying the recorded or stored states of the decoder to facilitate the processing of the input; and The computer-implemented method of claim 3 further comprising:

6. 6. The computer-implemented method of claim 5, wherein the decoder states represent acoustic features obtained by the encoder as a result of processing the data and text dependencies obtained by the decoder as a result of processing the data.

7. 2. The computer-implemented method of claim 1, wherein the text-based representation of the one or more semantic labels comprises one or more questions, each associated with a respective one of the one or more semantic labels.

8. The computer-implemented method of claim 1 , wherein the input further comprises the text corresponding to the content of the speech.

9. 1. A computer-implemented method for performing spoken language understanding, comprising: receiving data representing audio with speech; processing the data using a model to determine text corresponding to the audio content; receiving an input for performing a language understanding task, the input comprising a text-based representation of one or more semantic labels; processing the input using at least a portion of the model to extract semantic information from the text that corresponds to the content of the speech for the language understanding task; obtaining the extracted semantic information for the language understanding task; Equipped with the language understanding task comprises an intent classification task; the one or more semantic labels comprise one or more intent labels; the extracted semantic information comprises extracted intent information. Computer-implemented methods.

10. The computer-implemented method of claim 9 , wherein the extracted intent information indicates a respective positive or negative association with a respective intent label.

11. 1. A computer-implemented method for performing spoken language understanding, comprising: receiving data representing audio with speech; processing the data using a model to determine text corresponding to the audio content; receiving an input for performing a language understanding task, the input comprising a text-based representation of one or more semantic labels; processing the input using at least a portion of the model to extract semantic information from the text that corresponds to the content of the speech for the language understanding task; obtaining the extracted semantic information for the language understanding task; Equipped with the language understanding task comprises a slot-filling task; the one or more semantic labels comprise one or more slot labels; the extracted semantic information comprises one or more words extracted from the text that are associated with at least one of the one or more slot labels; Computer-implemented methods.

12. receiving a further input for performing a further language understanding task, the further input comprising a text-based representation of one or more further semantic labels; processing the further input using at least a portion of the model to extract further semantic information from the text that corresponds to the content of the speech; obtaining the extracted further semantic information; Furthermore, the further language understanding task is different from the language understanding task; The computer-implemented method of claim 1 .

13. The computer-implemented method of claim 12 , wherein the one or more further semantic labels are based on the extracted semantic information.

14. 13. The computer-implemented method of claim 12, wherein the text-based representation of the one or more further semantic labels comprises one or more questions, each associated with a respective one of the one or more further semantic labels.

15. The computer-implemented method of claim 12 , wherein the further input further comprises the text corresponding to the content of the speech.

16. the further language understanding task comprises a slot-filling task; the one or more further semantic labels comprise one or more slot labels; the extracted further semantic information comprises one or more words extracted from the text that are associated with at least one of the one or more slot labels; 13. The computer-implemented method of claim 12.

17. The computer-implemented method of claim 12 , wherein the processing of the further input is performed using the at least part of the model used to process the input.

18. The computer-implemented method of claim 1 , wherein the data representing the audio comprises a visual representation of the audio.

19. The computer-implemented method of claim 1 , wherein the data representing the audio comprises an audio file.

20. The computer-implemented method of claim 1 , wherein the model is trained based on prefix tuning.

21. 10. A system comprising one or more processors configured to perform the computer-implemented method of claim 1.

22. 10. A carrier medium carrying a plurality of computer-readable instructions adapted to cause one or more processors to perform the computer-implemented method of claim 1.

Images