Apparatus and method for correcting errors in transcribed voice commands

Abstract

An apparatus and method for correcting errors in transcribed voice commands, wherein the method comprises: receiving a voice command transcript; running a trained joint intent classification and slot filling model to generate contextualized word features and / or embeddings from word sequences detected in the transcript and subject them to intent classification and slot filling to generate predicted intent and slot value; running a trained phonetic based phrase confusion-aware model for correcting a transcription error in the predicted slot value; comparing the predicted slot value and / or an encoding or embedding generated from the predicted slot value against a plurality of predetermined valid slot values and / or a plurality of predetermined encodings or embeddings; and determining phonetic similarity scores for the predicted slot value against each valid slot value; and replacing the predicted slot value containing the transcription error with the valid slot value with highest phonetic similarity score.

Description

[0001] Apparatus and Method for Correcting Errors in Transcribed Voice Commands

[0002] Field

[0003] The present invention relates to an apparatus and a method for correcting errors in transcribed voice commands. In particular, such apparatus and method are suitable for, for example, deciphering voice commands for ordering of cargo / shipment and / or retrieval of cargo / shipment status in warehousing.

[0004] Background

[0005] A spoken language understanding (SLU) engine can be used to extract information from a user’s voice commands (or orders) and natural language queries, where such information can be served as input for other tasks. Some applications for such SLU engine and the use cases involving SLU systems that have such SLU engine include virtual assistants, voice ordering applications, and automated phone systems.

[0006] A common issue with deploying SLU systems in real world applications is that the performance of these SLU systems in real world environments differ significantly from that of systems in controlled and testbed environments. One of the reasons behind the discrepancies observed in the performances of the SLU systems in these environments can be attributed to the way these SLU engines were trained. These engines, or more precisely, the underlying natural language processing (NLP) models on which these engines are built on, are usually trained on clean text transcripts or utterances that are free of any transcription errors.

[0007] On the other hand, the SLU systems in real world environments involve inputs that are voice transcripts of voice utterances typically generated by an Automatic Speech Recognition (ASR) engine. These transcripts generated by ASR engines contain transcription errors that are not usually factored in the training of the NLP models, which leads to degradation in the performance of the SLU engines deployed.

[0008] Summary

[0009] According to an example of the present disclosure, there are provided an apparatus and a method for correcting errors in transcribed voice commands as claimed in the independent claims. Some optional features are defined in the dependent claims. Brief Description of the Drawings

[0010] Examples in the present disclosure will be better understood and readily apparent to one skilled in the art from the following written description, by way of example only and in conjunction with the drawings, in which:

[0011] Fig. 1 shows components of an example of an apparatus or system of the present disclosure. Fig. 2 shows a flowchart illustrating an example of a method of the present disclosure.

[0012] Fig. 3 illustrates correction of voice-to-text errors in data extracted from a transcript generated by Automatic Speech Recognition.

[0013] Fig. 4 shows a system architecture of an example of an apparatus or system of the present disclosure.

[0014] Detailed Description

[0015] Examples of the present disclosure involve an apparatus for operating a spoken language understanding (SLU) engine and a method of operation of the SLU engine. Such spoken language understanding (SLU) engine may be used for voice order automation in order processing systems, which includes voice orders for retrieval of cargo status from port warehouses using automated phone systems. Information extracted by the SLU engine from the voice commands (or orders) has to be valid and error-free to ensure that the correct commands (or orders) are acted upon. As the extracted information is used for subsequent tasks, any errors in the extracted information would undesirably limit the applicability of SLU systems for such voice order processing systems.

[0016] In one example of the present disclosure, a phrase confusion-aware Spoken Language Understanding (SLU) engine is developed to extract user intent and key information from transcripts of noisy voice commands generated by an Automatic Speech Recognition (ASR) engine.

[0017] The phrase confusion-aware Spoken Language Understanding (SLU) engine is a phonetic based phrase confusion-aware model that had learnt pairs of similar / dissimilar phrase representations for pairs of phonetically similar / confusable phrases. In operation, the phonetic based phrase confusion-aware model is used to correct ASR errors present in the extracted information from the ASR text transcript containing voice commands (or orders). The phonetic based phrase confusion-aware model corrects the ASR errors by comparing semantic representations of the extracted information against a set of predetermined valid slot values. Another term for “valid slot value” is “reference word”. The valid slot values or reference words may be customised and prepared by taking into account the application, context, environment, local culture, field and / or industry.

[0018] In the abovementioned example, given a voice command, there are typically two main tasks that to undertake to extract information out of (the voice transcript of) the voice command generated (or produced) by the Automatic Speech Recognition (ASR) engine. They are:

[0019] 1) Intent classification: What is the user intent of the voice command?

[0020] 2) Slot filling: What are the key words / phrases corresponding to information fields (slots) in the voice command that need to be captured?

[0021] The slot filling task is a sequence labelling task, where given a word sequence w = (wi, W2, ... , wn) that represents the natural language query, a sequence of slot labels s = (si , S2, ... , sn) is assigned to w, where each slot label corresponds to each word token and is given a semantic label from {‘O’, h, .... IP}. Here, the slot label is equal to some if it carries a semantic meaning, and a label ‘O’ otherwise.

[0022] To ensure that the information extracted from the voice orders or commands (in the voice transcript) are valid and error-free, an additional error correction task is undertaken to correct any errors present in the extracted information, arising from transcription errors generated by the ASR engine.

[0023] An example of a joint intent classification-slot filling task performed by the phrase confusion- aware Spoken Language Understanding (SLU) engine of the present example for a given transcript of the voice command generated by the ASR engine is described below.

[0024] Input to the joint intent classification-slot filling task:

[0025] Original customer utterance: “can I get the shipment status for SIMTech for 8 March” Transcript generated by the ASR engine: “can i get the shipment status for some tech for ate match”

[0026] Note: In the above example, “SIMTech” is the name of an organization (“ORG”). Transcribing “SIMTech” to “some tech” is an ASR error. Transcribing “8 March”, which is a date, to “ate match” is also an ASR error.

[0027] Output of the joint intent classification-slot filling task: User intent: Get shipment status Slot labels from ASR transcript:

[0028] Table 1 :

[0029] A word with a slot label “O” is deemed as not carrying a semantic meaning. The label “ORG” means organization and the label “DATE” means date.

[0030] Extracted and corrected slot values from ASR transcript:

[0031] Table 2:

[0032] An overview of the model architecture for the joint intent classification-slot filling task with error correction is described as follows.

[0033] Fig. 1 shows an overview of an example of the model architecture for the joint intent classification-slot filling task, with error correction features. With reference to Fig. 1, an apparatus or system 100 comprising a Spoken Language Understanding (SLU)-based engine for extracting information from noisy voice commands is proposed. The apparatus 100 comprises the following features (A) to (D).

[0034] (A) An automatic speech recognition (ASR) engine 104 for converting an input audio stream 102 containing voice orders or commands into a text transcript, wherein voice- to-text technology is used for such conversion. An example of a model that may be used for such ASR engine 104 is Vosk (See below for details). Other suitable models or engines may also be used for the ASR engine 104.

[0035] Vosk is an offline open source speech recognition toolkit. It enables speech recognition for 20+ languages and dialects - English, Indian English, German, French, Spanish, Portuguese, Chinese, Russian, Turkish, Vietnamese, Italian, Dutch, Catalan, Arabic, Greek, Farsi, Filipino, Ukrainian, Kazakh, Swedish, Japanese, Esperanto, Hindi, Czech, Polish. More to come. Vosk models are small (50 Mb) but provide continuous large vocabulary transcription, zero-latency response with streaming API, reconfigurable vocabulary and speaker identification. They have speech recognition bindings implemented for various programming languages like Python, Java, Node.JS, C#, C++, Rust, Go and others.

[0036] Vosk supplies speech recognition for chatbots, smart home appliances, virtual assistants. It can also create subtitles for movies, transcription for lectures and interviews. Vosk scales from small devices like Raspberry Pi or Android smartphone to big clusters.

[0037] (B) A data normalization module 106 for normalizing or pre-processing of the text transcript to generate data for further processing. For example, there may involve removal of punctuation from the text transcript, generation of contextualized word features and / or embeddings from word sequences in the text transcript, etc. If text transcripts are inputted from different ASR engines and have different formats, the data normalization module 106 converts them into a consistent format. In one example, a pre-trained language model BERT may be used to generate contextualized word features and / or embeddings from word sequences in the text transcript.

[0038] Using a pre-trained language model like BERT (Bidirectional Encoder Representations from Transformers) to generate contextualized word features may involve several steps as follow:

[0039] 1. Pre-training: BERT is pre-trained on a large corpus of text data using two tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). This helps BERT understand the context of words in a sentence.

[0040] 2. Tokenization: The text transcript is tokenized into smaller units, typically words or subwords. BERT uses a WordPiece tokenizer, which breaks down words into subword units.

[0041] 3. Embedding: Each token is converted into a dense vector representation (embedding) that captures its meaning. BERT generates these embeddings by considering the context of each word in the sentence, both from the left and the right.

[0042] 4. Contextualization: BERT processes the entire sequence of tokens through multiple layers of transformers. Each layer refines the embeddings by incorporating information from surrounding tokens, resulting in contextualized word features. This means the representation of a word is influenced by the words around it, capturing its meaning in context.

[0043] 5. Output: The final output is a sequence of contextualized word embeddings, where each embedding represents a word in the context of the entire sentence. These embeddings can be used for various downstream tasks like text classification, named entity recognition, or sentiment analysis. In the present example, the embeddings can be used for intent classification and slot filling. More details are described later.

[0044] (C) A joint intent classification-slot filling model 108 for predicting intent (i.e. the intention of the user order or command), slot labels indicative of a classification or category of selected words (e.g. a label can show that a word or phrase is not in any particular class or category, relates to organization name, relates to date etc.) and slot values (i.e. extracted relevant words that may contain ASR errors) from the words in the text transcript. The slot labels can help to give meaning to the slot values. In some cases, the intent may also give meaning to the slot values.

[0045] (D) A trained phonetic based phrase confusion-aware model 1 10 that had learnt pairs of phonetically similar and dissimilar phrases by leveraging on extracted information and known valid values (or correct word) for each slot label. In other words, the phonetic based phrase confusion-aware model 1 10 is custom-trained to learn to recognise words similar to words known in a particular field or industry. For instance, names and abbreviated names of organizations, terms used in a specific item, service and / or product ordering process, etc. The trained phrase confusion-aware model 110 is used to correct ASR errors in the slot values containing extracted relevant words that contain ASR errors. In this manner, it can be said that the trained phrase confusion-aware model 110 adopts a semantic similarity-based approach to correct such ASR errors. The trained phonetic based phrase confusion-aware model 110 should output extracted information 1 12 containing no errors. Such error-free extracted information 112 can be used for further processing, for example, to convert them into an instruction to a machine to perform an action instructed by the voice orders or commands. The extracted information 1 12 may include vector representations of extracted words.

[0046] In summary, the apparatus 100 first produces a text transcript generated by the ASR engine from an input audio stream 102. In the present example, the input audio stream 102 contains noisy audio voice orders or commands. The audio voice orders or commands are noisy in the sense that there could be background noise, words are spoken in an imperfect manner that may contain slangs, phrases, unclear pronunciation, and / or terms specific to a particular field, industry and / or context. Next, downstream Natural Language Processing (NLP) tasks are performed on the text transcript to extract relevant information. The downstream NLP tasks involve a step of using the joint intent classification-slot filling model 108 to predict the intention of user orders or commands, slot labels indicative of a classification or category of selected words and slot values (slot values refer to the selected words). Thereafter, the trained phrase confusion-aware model 1 10 is used to correct any ASR error in the slot values. The trained phrase confusion-aware model 1 10 outputs extracted information 112 including the corrected slot values. Such extracted information 11 is also the output of the apparatus 100.

[0047] More details on the joint intent classification-slot filling model 108 and the phrase confusion- aware model 1 10 are described as follows.

[0048] Fig. 2 details a flowchart of an example for performing joint intent classification-slot filling with an error correction task performed on an input word sequence with ASR errors. The model illustrated by the flowchart can be broken down into two main parts, the joint intent classification-slot filling model 108 and the phonetic based confusion-aware model 110 for error correction. In Fig. 2, the joint intent classification-slot filling model 108 comprises two layers 218 (for slot filling) and 224 (for intent classification) and the phonetic based confusion- aware model 1 10 is represented by reference numeral 206.

[0049] Given an input customer voice command or utterance (voice input), an Automatic Speech Recognition (ASR) engine (e.g. 104 in Fig. 1 ) is used to generate a text transcript (also known as ASR transcript) from the voice input, where this transcript may contain transcription errors (ASR errors) from the ASR engine. Next, word sequences from this transcript with ASR errors 208 are subject to data normalization. Data normalization is performed to remove punctuation from the text transcript and to generate contextualized word features 212 from the input word sequences. For example, the contextualized word features 212 can be generated using a pretrained language model BERT. BERT word features 212 can be generated by the BERT word embedding layer 210. These BERT word features 212 are then fed into a bidirectional Long Short-Term Memory (Bi-LSTM) layer 214. The Bi-LSTM layer 214 outputs word sequence features 216, which will be used for intent classification by a Softmax layer for intent classification 224 and used for slot filling by a Condition Random Field (CRF) layer for slot filling 218 to generate the predicted intent 226 and the predicted slot values 220 respectively. The predicted slot values 220 will contain ASR errors if they are present. These ASR errors are corrected by a fine-tuned phonetically based confusion-aware model 206, which outputs ASR error-free extracted slot values 222. The ASR error-free extracted slot values 222 are essentially the corrected words. For instance, in an earlier example described, the corrected words are “Simtech” (name of an organization) and “eight march” (date).

[0050] The Bi-LSTM layer 214 is a type of recurrent neural network (RNN) layer that is designed to learn long-term dependencies in sequence data. Unlike a standard LSTM, which processes data in one direction (either forward or backward), a Bi-LSTM processes data in both directions. This means it has two LSTM layers: one that processes the input sequence from start to end, and another that processes it from end to start. The Bi-LSTM layer 214 can capture context from both past and future states, making it useful for tasks where the context of the entire sequence is important. For example, in natural language processing, understanding the meaning of a word can depend on both the words that come before and after it.

[0051] The softmax layer for intent classification 224 is basically an activation function that converts raw output scores of a neural network into probabilities, which sum up to 1. The softmax function takes a vector of raw scores (logits) and transforms them into a probability distribution. Each element in the output vector represents the probability of the corresponding class. The output probabilities can be interpreted as the likelihood of the input belonging to each class. The class with the highest probability is typically chosen as the predicted intent.

[0052] A Conditional Random Field (CRF) layer 218 is a type of probabilistic graphical model used for structured prediction tasks in machine learning and pattern recognition. Unlike traditional classifiers that predict labels for individual samples independently, CRFs consider the context by modeling the dependencies between predictions. CRFs are useful in tasks where the prediction of one label depends on the prediction of neighboring labels. For example, in natural language processing, linear chain CRFs can be used for tasks like part-of-speech tagging and named entity recognition, where each label depends on its immediate neighbors.

[0053] During machine training stage, parallel to training the joint intent classification-slot filling model comprising the two layers 218 and 224, a pre-trained sentence embedding model 204 is finetuned on pairs of phonetically similar and dissimilar phrases, along with score labels that trains the sentence embedding model to recognize phonetic similarity (and respectively, dissimilarity) of phonetically similar (and respectively, dissimilar) pairs (202 in Fig. 2). These pairs of phrases are generated using text transcripts of voice commands in the training data. The phonetically similar pairs were constructed by pairing each slot value extracted from the original input word sequence against the corrected valid slot value, while the phonetically dissimilar pairs were constructed by pairing each slot value extracted from the original input word sequence against a different valid slot value.

[0054] An example of the pre-trained sentence embedding model 204 is Sentence-BERT (SBERT), which is a modification of BERT (Bidirectional Encoder Representations from Transformers) that uses Siamese and triplet networks to derive semantically meaningful sentence embeddings. SBERT reduces the computational overhead of BERT for semantic similarity search and transfer learning tasks, while maintaining or improving the accuracy.

[0055] The score labels are defined as follows: for a given pair of phonetically similar phrase (p, q), the score label I is given by

[0056] I = max(cos(u, v), 0.9).

[0057] Here, u and v are the sentence embeddings corresponding to p and q respectively generated by a separate pre-trained Sentence-BERT model, and cos(u, v) is the cosine similarity of the sentence embeddings u and v. Similarly, for a given pair of phonetically dissimilar phrase (p’, q’), the score label I’ is given by

[0058] I’ = min(cos(u, v), 0.1).

[0059] Examples of these pairs of phrases, along with their score labels, are given in Table 3 below.

[0060] Table 3 (Examples of pairs of phonetically similar and dissimilar phrases):

[0061] “arte” is the name of another organization and it is phonetically dissimilar to “some tech”. Hence, it corresponds to a low score of 0.1. On the other hand, “simtech”, an organization phonetically similar to “some tech” has a high score of 0.9. If an extracted slot value, for instance, “eight march”, is identical to the corrected slot value (also “eight march”), the score label value is 1 .

[0062] The phonetic based phrase confusion-aware model 206 is created by fine-tuning a sentence embedding model (Sentence-BERT) based on pairs of phonetically similar and dissimilar phrases, using the mean squared loss function. More precisely, for given a batch size of n, together n pairs of phonetically similar and dissimilar phrases (pi, qi), (P2, q2), .... (pn, qn) along with their corresponding score labels h, l2, .... In, compute the pairs of sentence embeddings (ui, Vi), (u2, V2), ... , (un, vn) corresponding to (pi, qi), (p2, q2), .... (pn, qn), and seek to minimize the following equation directed tp mean squared loss function lmse: The sentence embedding model 204 fine-tuned using phonetically similar and dissimilar pairs of phrases becomes the phonetic based confusion aware model 206. The phonetic based confusion-aware model 206 is then trained together with the joint intent classification-slot filling model (i.e. the layers 218 and 224), where the loss function is the weighted sum of the intent classification, slot filling, and sentence similarity losses.

[0063] During operation, to correct any ASR errors that may be present in the slot values extracted from an ASR transcript using the joint intent classification-slot filling model (i.e. layers 218 and 224), each extracted slot value is compared against a list of valid slot values (i.e. a plurality of predetermined valid slot values or reference words) for the (predicted) slot label, by comparing the phonetically similarity score of the extracted slot value against each valid slot value. Specifically, the sentence embeddings of the extracted slot value and each valid slot value are first generated using the phonetic based confusion aware model 206 and the cosine similarity of the sentence embedding of the extracted slot value against that of each valid slot value is computed. Next, the valid slot value with the highest phonetic similarity score is chosen. Examples of ASR error correction using the phonetic based confusion-aware model 206 are illustrated by Fig. 3.

[0064] Fig. 3 shows a first table 302 containing:

[0065] (i) slot value “some tech” relevant to an organization (ORG) that is extracted from the ASR transcript;

[0066] (ii) a list of predetermined valid slot values for organization, which includes organization name abbreviations such as simtech, arte etc.; and

[0067] (iii) semantic similarity score of the valid slot values to the extracted slot value, which is 0.9 for “simtech” (indicative of high phonetic similarity to “some tech”) and 0.1 for “arte” (indicative of low phonetic similarity to “some tech”).

[0068] Fig. 3 also shows a second table 304 containing:

[0069] (i) slot value “ate march” for date that is extracted from the ASR transcript;

[0070] (ii) a list of predetermined valid slot values for date, which includes nine may, eight march, etc.; and

[0071] (iii) semantic similarity score of the valid slot values to the extracted slot value, which is 0.1 for “nine may” (indicative of low phonetic similarity to “ate march”) and 0.9 for “eight march” (indicative of high phonetic similarity to “ate march”).

[0072] Fig. 3 further shows a third table 306 containing the corrected values for the slot values pertaining to organization and date in the first table 302 and the second table 304 respectively. The third table 306 contains corrected slot value “simtech” for organization and corrected slot value “eight march” for date. These slot values have highest phonetic similarity score.

[0073] Experiments to assess performance of an example having features described with reference to Fig. 1 and Fig. 2 and a modified version of such example are conducted on an in-house dataset based on a Model Factory ordering application created by the organization, SIMTech. Such application allows user to order diffuser thumb drives, and the user can choose the scents, holder colour and a message. A synthetic dataset consisting of textual utterances, which includes utterances with disfluencies, is first created. Subsequently, voice utterances are generated using a Python text to speech module from the synthetic textual utterances, and the ASR transcripts are generated from the voice utterances using an ASR engine. For training, validation and testing, the synthetic and ASR utterances were mixed and sampled.

[0074] The preferred method of the example of the present disclosure is called an “Extract First, Correct Later” method in the description below. The “Extract First, Correct Later” method is compared against a baseline method that does not correct any ASR error. The “Extract First, Correct Later” method is also compared against another method called the “Correct First, Extract Later” method, where one first deduces the user intent and the corrected word sequence from the input word sequence that may contain ASR errors, before performing the slot filling task on the corrected word sequence. The “Extract First, Correct Later” method is further compared to an alternative (or modified) method. Such alternative method is another version of the “Extract First, Correct Later” method, in which the phonetic based confusion- aware model (e.g. 110 of Fig. 1) is replaced with a simpler model using a phonetic based fuzzy string matching algorithm. Such alternative method constitutes an example of the present disclosure. It is still a viable option as its performance is close to the performance of the preferred “Extract First, Correct Later” method. The experimental results are given in Table 4 below.

[0075] The “Correct First, Extract Later” method involves training a model on a new ASR word correction task together with the joint intent classification-slot filling task. Firstly, a sequence of features from the input word sequence in the ASR transcript is encoded using a Recurrent Neural Network (RNN) word encoder. Subsequently, intent is predicted from the word encoder features using an RNN intent decoder. From the word encoder features and the intent decoder features, the word decoder features and the corresponding word sequence are generated using an RNN word decoder. Finally, the intent decoder, word encoder, and word decoder features are used to generate slot labels corresponding to the corrected word sequence using an RNN slot decoder. The phonetic based fuzzy string matching algorithm may be implemented as follows. There may involve a fuzzy search, which uses lexical and phonetic similarity measures to match user or customer utterances with reference words. Such reference words are predetermined. For instance, “simtech” and “arte” of earlier examples are predetermined reference words under the “organization” class or category. The phonetic based fuzzy string matching algorithm can address ASR errors by comparing the phonetic representations of n-grams (substrings of n words) from the customer utterance and the phonetic representations of reference words. If an ASR error is detected through the comparison of phonetic representations, the wrong word is replaced by the reference word. Some key steps of the phonetic based fuzzy string matching algorithm may include the following.

[0076] Pre-processing (i.e. this is what the data normalization module 106 of Fig. 1 does): Removal of generic tokens (e.g., "hello", "ok") from the customer utterance.

[0077] Handling split words by joining continuous single-letter words (e.g., "t v" becomes "TV").

[0078] Direct Match: This step looks for exact text matches between query tokens and reference words tokens.

[0079] Partial Match: This step handles partial utterances and ASR errors by using n-gram based partial similarity measures.

[0080] Phonetic Match: This step compares the phonetic representations of n-grams from the customer utterance and reference words. Phonetic representations may be generated using a phonetic encoding algorithm (e.g., Soundex, Metaphone). A similarity score is calculated based on the phonetic similarity of n-grams.

[0081] The phonetic based fuzzy string matching algorithm may work as follows.

[0082] Phonetic Representation: Convert both the customer utterance and correct words or phrase into their phonetic representations. For example, a correct phrase "memory card" might be phonetically encoded as "MANRACAD".

[0083] N-gram Matching: Generate n-grams (e g., unigrams, bigrams, trigrams) from the phonetic representations of both the query and product titles. Compare the n-grams from the query with those from the product titles to find matches.

[0084] Similarity Scoring: Calculate a similarity score based on the proportion of matching n-grams. Higher n-grams (e.g., trigrams) are given more weight as they provide more context and reduce false positives.

[0085] Sequential Matching: The algorithm proceeds sequentially through direct match, partial match, and phonetic match until a matching order is found. If no match is found at a higher n-gram level, the algorithm falls back to the previous level. For example, consider a customer utterance "in clinics hot 94" and reference words "Infinix Hot 9 Pro" (which is a particular product name or title), the phonetic representations might be "INCLINICSHOT" and "INFINIXHOTPRO" respectively. The phonetic based fuzzy string matching algorithm would generate n-grams from these representations and compare them to find a match, even though the exact text does not match. This approach ensures that the system can accurately identify the product the customer is calling about, and thereafter correct ASR errors using the correct word associated with the phonetic representations.

[0086] To determine similarity score, the phonetic based fuzzy string matching algorithm may calculate Levenshtein distance, also known as edit distance, which is a string metric used to measure the difference between two word sequences. It calculates the minimum number of single-character edits (insertions, deletions, or substitutions) required to transform one string into another. In summary, similar to the preferred “Extract First, Correct Later” method, phonetically similarity score of an extracted slot value (e.g. word, encoded word, and / or embedding extracted from the ASR transcript) is compared against each valid slot value. Firstly, phonetic encodings are generated for the extracted slot value and each valid slot value. Thereafter, Levenshtein distance between these encodings are calculated to determine the valid slot value with the highest phonetic match.

[0087] In another example, it could be that both Levenshtein distance and the earlier described sentence embedding model (Sentence-BERT) trained based on pairs of phonetically similar and dissimilar phrases (or even more phonetic matching models) are used to determine phonetic similarity scores and the valid slot value with highest phonetic match is used to correct ASR errors.

[0088] Table 4 (accuracy comparison between examples of present disclosure with a baseline method and a “Correct First, Extract Later” method for extracting information from noisy voice commands): Examples of the output from the preferred method is compared against the outputs of the “Correct First, Extract Later” method and the alternative method below.

[0089] Example 1 :

[0090] Original customer utterance: “ah two”

[0091] Transcript generated by the ASR engine: “are too”

[0092] User intent: Set quantity

[0093] Corrected slot value-slot label pairs: (two, quantity)

[0094] Output:

[0095] Corrected word sequence generated by the “Correct First, Extract Later” method: “ah two”

[0096] Table 5 (comparison of outputted intents and slot values pertaining to setting quantity):

[0097] Example 2:

[0098] Original customer utterance: “ah replace the old holder colour brown with red”

[0099] Transcript generated by the ASR engine: “or replace the old holder color brown with read” User intent: Set colour

[0100] Corrected slot value-slot label pairs: (brown, old colour), (red, new colour)

[0101] Output:

[0102] Corrected word sequence generated by the “Correct First, Extract Later” method: “ah replace the old holder colour yellow with brown”

[0103] Table 6 (comparison of outputted intents and slot values pertaining to setting colour):

[0104] Examples 1 and 2 above illustrates that the “Extract First, Correct Later” method outputs the most accurate slot values and slot labels.

[0105] Fig. 4 shows an example of an apparatus or system 400 that can be used to implement the apparatus or system of Fig. 1 and the features illustrated by Fig. 2. The apparatus 400 can be a local computer or remote server (e.g. cloud server) in which users can submit an audio input 102 (or a video input containing audio to be transcribed) by the ASR engine 104.

[0106] The apparatus 400 may comprise a processing unit (or processor) 402 for processing software including one or more programs for running one or more computer / server / device applications to enable a backend logic flow or the method or methods for carrying out the relevant software steps required to implement the features of the earlier described Figures.

[0107] Furthermore, the processing unit 402 may include user input modules such as a computer mouse 436, keyboard / keypad 404, and / or a plurality of output devices such as a display device 408. The display device 408 may incorporate technologies like LCD, LED, OLED, and the like. It may also be a touch screen capable of receiving user input.

[0108] The processing unit 402 may be connected to a computer network 412 via a suitable transceiver device 414 (i.e. a network interface), to enable access to e.g. the Internet or other network systems such as a wired Local Area Network (LAN) or Wide Area Network (WAN). The processing unit 402 may be connected to one or more external wireless communication enabled devices 434 via a suitable wireless transceiver device 432, e.g. a WiFi transceiver, Bluetooth module, Mobile telecommunication transceiver suitable for Global System for Mobile Communication (GSM), 3G, 4G, 5G telecommunication systems, and the like. Through the computer network (i.e. internet) 412, the processing unit 402 can gain access to one or more storages i.e. data storages, databases, data servers and the like connectable to the computer network 412 to retrieve and / or store data in the one or more storages. The external wireless communication enabled devices 434 may be user computers or mobile devices (e g. tablet computer, smartphones etc.) and users may use such devices to connect to the processing unit 402.

[0109] The processing unit 402 may include a microprocessor 418, a Random Access Memory (RAM) 420 and a Read Only Memory (ROM) 422. In the case of a server, the processing unit 402 may also include a number of Input / Output (I / O) interfaces, for example I / O interface 438 to the computer mouse 436, a memory card slot 416, I / O interface 424 to the display device 408, and I / O interface 426 to the keyboard / keypad 404.

[0110] The components of the processing unit 402 typically communicate via an interconnected bus 428 and in a manner known to the person skilled in the relevant art.

[0111] The programs may be supplied to the user of the processing unit 402, or the processor (not shown) of one of the one or more external wireless communication enabled devices 434, encoded on a data storage medium such as a CD-ROM, on a flash memory carrier, Solid State Drive, or a Hard Disk Drive, and are to be read using a corresponding data storage medium drive of a data storage device 430. Such computer or application programs may also be downloaded from the computer network 412. The application programs are read and controlled in its execution by the processor 418. Intermediate storage of program data may be accomplished using RAM 420.

[0112] In more detail, one or more of the computer or application programs may be stored on any non-transitory machine- or computer- readable medium. The machine- or computer- readable medium may include storage devices such as magnetic or optical disks, memory chips, or other storage devices suitable for interfacing with a general-purpose computer. The machine- or computer- readable medium may also include a hard-wired medium such as that exemplified in the Internet system, or wireless medium such as that exemplified in the Wireless l_AN (WLAN) system and the like. The computer program when loaded and executed on such a general-purpose computer effectively results in an apparatus that implements the steps of the computing methods in examples herein described.

[0113] Some applications of examples of the present disclosure are as follows.

[0114] Automation of the order management processes: Currently, many voice and textual sales orders placed by customers are still converted manually by order management professionals into structured sales order data, which is inefficient and limits the scalability of the order management processes. With the technology of examples of the present disclosure, companies can automate and streamline their order management processes, and this technology can also be integrated with planning systems to facilitate real-time planning.

[0115] Retrieval of cargo status from port warehouses: The current freight forwarding workflow at some logistics companies involves daily manual phone calls by the transport coordinator to the automated phones at the port warehouses for updates on the cargo status, which is time consuming. With the technology of examples of the present disclosure, the retrieval of cargo status from port warehouses can be automated, thereby freeing the transport coordinator up for more important tasks, such as transport planning of the truck operators.

[0116] Examples of the present disclosure may include the following features. Reference numerals of the elements in the figures of the present disclosure that are examples of the features discussed are provided in brackets.

[0117] An apparatus (e.g. 100, 400) for correcting errors in transcribed voice commands, wherein the apparatus comprises: a processor (e.g. 402) configured to execute instructions in a memory to operate the apparatus to: receive a text transcript of a voice command (e.g. 102) transcribed by an Automatic speech recognition (ASR) engine (e.g. 104); run a trained joint intent classification and slot filling model (e.g. 108, 218, 224) configured to generate contextualized word features (e.g. 216) and / or embeddings from a word sequence (e.g. 208) detected in the text transcript and to subject the generated word features and / or embeddings to intent classification and slot filling to generate a predicted intent (e.g. 226) and a corresponding predicted slot value (e.g. 220); and run a trained phonetic based phrase confusion-aware model (e.g. 1 10, 206) for correcting a transcription error in the predicted slot value, wherein the trained phonetic based phrase confusion-aware model is configured to: compare the predicted slot value and / or an encoding or embedding generated from the predicted slot value against a plurality of predetermined valid slot values and / or a plurality of encodings or embeddings generated from the predetermined valid slot values; determine phonetic similarity scores for the predicted slot value against each valid slot value; and replace the predicted slot value containing the transcription error with the valid slot value with highest phonetic similarity score (e.g. 112, 222).

[0118] The contextualized word features and / or embeddings may be generated using a trained language model BERT (Bidirectional Encoder Representations from Transformers) (e.g. 210).

[0119] The trained phonetic based phrase confusion-aware model may be based on a sentence embedding model (SBERT) (e.g. 204) and trained based on pairs of phonetically similar and dissimilar phrases (e.g. 202), wherein the phonetically similar pairs were constructed by pairing each slot value extracted from an original input word sequence against the corrected valid slot value, while the phonetically dissimilar pairs were constructed by pairing each slot value extracted from the original input word sequence against a different valid slot value.

[0120] The contextualized word features may be inputted to a bidirectional Long Short-Term Memory (Bi-LSTM) layer (e.g. 214) to produce output word sequence features, wherein the output word sequence features are used for the intent classification and slot filling to generate the predicted intent and the corresponding predicted slot value.

[0121] The Bi-LSTM layer may be equipped with multi-head attention.

[0122] Punctuation removal may be performed on the text transcript using a data normalization module (e.g. 106) before processing the text transcript using the trained joint intent classification and slot filling model.

[0123] The trained phonetic based phrase confusion-aware model may be configured to generate sentence or word embeddings of the predicted slot value and each valid slot value and compute a cosine similarity of the sentence or word embedding of the predicted slot value against that of each valid slot value as the phonetic similarity score.

[0124] The apparatus may be used to decipher voice commands for ordering of cargo / shipment and / or retrieval of cargo / shipment status in warehousing.

[0125] Phonetic encodings may be generated for the predicted slot value and each valid slot value, and Levenshtein distance is calculated between these encodings to determine the phonetic similarity score, wherein the valid slot value with calculated Levenshtein distance indicative of highest phonetic match has the highest phonetic similarity score.

[0126] A method for correcting errors in transcribed voice commands, wherein the method comprises: receiving a text transcript of a voice command (e.g. 102) transcribed by an Automatic speech recognition (ASR) engine (e.g. 104); running a trained joint intent classification and slot filling model (e.g. 108, 218, 224) configured to generate contextualized word features (e.g. 216) and / or embeddings from a word sequence (e.g. 208) detected in the text transcript and to subject the generated word features and / or embeddings to intent classification and slot filling to generate a predicted intent (e.g. 226) and a corresponding predicted slot value (e.g. 220); and running a trained phonetic based phrase confusion-aware model (e.g. 110, 206) for correcting a transcription error in the predicted slot value, wherein the trained phonetic based phrase confusion-aware model is configured to: compare the predicted slot value and / or an encoding or embedding generated from the predicted slot value against a plurality of predetermined valid slot values and / or a plurality of encodings or embeddings generated from the predetermined valid slot values; determine phonetic similarity scores for the predicted slot value against each valid slot value; and replacing the predicted slot value containing the transcription error with the valid slot value with highest phonetic similarity score (e.g. 1 12, 222).

[0127] With regard to the method, the contextualized word features and / or embeddings may be generated using a trained language model BERT (Bidirectional Encoder Representations from Transformers) (e.g. 210).

[0128] With regard to the method, the trained phonetic based phrase confusion-aware model may be based on a sentence embedding model (SBERT) (e.g. 204) and trained based on pairs of phonetically similar and dissimilar phrases (e.g. 202), wherein the phonetically similar pairs were constructed by pairing each slot value extracted from an original input word sequence against the corrected valid slot value, while the phonetically dissimilar pairs were constructed by pairing each slot value extracted from the original input word sequence against a different valid slot value. With regard to the method, wherein the contextualized word features may be inputted to a bidirectional Long Short-Term Memory (Bi-LSTM) layer (e.g. 214) to produce output word sequence features, wherein the output word sequence features are used for the intent classification and slot filling to generate the predicted intent and the corresponding predicted slot value.

[0129] With regard to the method, wherein the Bi-LSTM layer may be equipped with multi-head attention.

[0130] With regard to method, wherein punctuation removal may be performed on the text transcript using a data normalization module (e.g. 106) before processing the text transcript using the trained joint intent classification and slot filling model.

[0131] With regard to the method, wherein the trained phonetic based phrase confusion-aware model may be configured to generate sentence or word embeddings of the predicted slot value and each valid slot value and compute a cosine similarity of the sentence or word embedding of the predicted slot value against that of each valid slot value as the phonetic similarity score.

[0132] With regard to the method, wherein the method may be used to decipher voice commands for ordering of cargo / shipment and / or retrieval of cargo / shipment status in warehousing.

[0133] With regard to the method, wherein phonetic encodings may be generated for the predicted slot value and each valid slot value, and Levenshtein distance is calculated between these encodings to determine the phonetic similarity score, wherein the valid slot value with calculated Levenshtein distance indicative of highest phonetic match has the highest phonetic similarity score.

[0134] A non-transitory computer-readable medium storing a transcription error correction program including instructions that, when executed by a processor (e.g. 402), causes an apparatus (e.g. 100, 400) to: receive a text transcript of a voice command (e.g. 102) transcribed by an Automatic speech recognition (ASR) engine (e.g. 104); run a trained joint intent classification and slot filling model (e.g. 108, 218, 224) configured to generate contextualized word features (e.g. 216) and / or embeddings from a word sequence (e.g. 208) detected in the text transcript and to subject the generated word features and / or embeddings to intent classification and slot filling to generate a predicted intent (e.g. 226) and a corresponding predicted slot value (e.g. 220); and run a trained phonetic based phrase confusion-aware model (e.g. 1 10, 206) for correcting a transcription error in the predicted slot value, wherein the trained phonetic based phrase confusion-aware model is configured to: compare the predicted slot value and / or an encoding or embedding generated from the predicted slot value against a plurality of predetermined valid slot values and / or a plurality of encodings or embeddings generated from the predetermined valid slot values; determine phonetic similarity scores for the predicted slot value against each valid slot value; and replace the predicted slot value containing the transcription error with the valid slot value with highest phonetic similarity score (e.g. 112, 222).

[0135] In the present disclosure, unless the context clearly indicates otherwise, the term “comprising” has the non-exclusive meaning of the word, in the sense of “including at least” rather than the exclusive meaning in the sense of “consisting only of’. The same applies with corresponding grammatical changes to other forms of the word such as “comprise”, “comprises” and so on.

[0136] While the invention has been described in the present disclosure in connection with a number of examples, embodiments and implementations, the invention is not so limited but covers various obvious modifications and equivalent arrangements, which fall within the purview of the appended claims. Although features of the invention are expressed in certain combinations among the claims, it is contemplated that these features can be arranged in any combination and order.

Claims

CLAIMS1. An apparatus for correcting errors in transcribed voice commands, wherein the apparatus comprises: a processor configured to execute instructions in a memory to operate the apparatus to: receive a text transcript of a voice command transcribed by an Automatic speech recognition (ASR) engine; run a trained joint intent classification and slot filling model configured to generate contextualized word features and / or embeddings from a word sequence detected in the text transcript and to subject the generated word features and / or embeddings to intent classification and slot filling to generate a predicted intent and a corresponding predicted slot value; and run a trained phonetic based phrase confusion-aware model for correcting a transcription error in the predicted slot value, wherein the trained phonetic based phrase confusion-aware model is configured to: compare the predicted slot value and / or an encoding or embedding generated from the predicted slot value against a plurality of predetermined valid slot values and / or a plurality of encodings or embeddings generated from the predetermined valid slot values; determine phonetic similarity scores for the predicted slot value against each valid slot value; and replace the predicted slot value containing the transcription error with the valid slot value with highest phonetic similarity score.

2. The apparatus of claim 1 , wherein the contextualized word features and / or embeddings are generated using a trained language model BERT (Bidirectional Encoder Representations from Transformers).

3. The apparatus of claim 1 or 2, wherein the trained phonetic based phrase confusion- aware model is based on a sentence embedding model (SBERT) and trained based on pairs of phonetically similar and dissimilar phrases, wherein the phonetically similar pairs were constructed by pairing each slot value extracted from an original input word sequence against the corrected valid slot value, while the phonetically dissimilar pairs were constructed by pairing each slot value extracted from the original input word sequence against a different valid slot value.

4. The apparatus of claims 1 , 2 or 3, wherein the contextualized word features are inputted to a bidirectional Long Short-Term Memory (Bi-LSTM) layer to produce output word sequence features, wherein the output word sequence features are used for the intent classification and slot filling to generate the predicted intent and the corresponding predicted slot value.

5. The apparatus of claim 4, wherein the Bi-LSTM layer is equipped with multi-head attention.

6. The apparatus of any one of the preceding claims, wherein punctuation removal is performed on the text transcript using a data normalization module before processing the text transcript using the trained joint intent classification and slot filling model.

7. The apparatus of any one of the preceding claims, wherein the trained phonetic based phrase confusion-aware model is configured to generate sentence or word embeddings of the predicted slot value and each valid slot value and compute a cosine similarity of the sentence or word embedding of the predicted slot value against that of each valid slot value as the phonetic similarity score.

8. The apparatus of any one of the preceding claims, wherein the apparatus is used to decipher voice commands for ordering of cargo / shipment and / or retrieval of cargo / shipment status in warehousing.

9. The apparatus of any one of the preceding claims, wherein phonetic encodings are generated for the predicted slot value and each valid slot value, and Levenshtein distance is calculated between these encodings to determine the phonetic similarity score, wherein the valid slot value with calculated Levenshtein distance indicative of highest phonetic match has the highest phonetic similarity score.

10. A method for correcting errors in transcribed voice commands, wherein the method comprises: receiving a text transcript of a voice command transcribed by an Automatic speech recognition (ASR) engine; running a trained joint intent classification and slot filling model configured to generate contextualized word features and / or embeddings from a word sequence detected in the text transcript and to subject the generated word features and / or embeddings to intentclassification and slot filling to generate a predicted intent and a corresponding predicted slot value; and running a trained phonetic based phrase confusion-aware model for correcting a transcription error in the predicted slot value, wherein the trained phonetic based phrase confusion-aware model is configured to: compare the predicted slot value and / or an encoding or embedding generated from the predicted slot value against a plurality of predetermined valid slot values and / or a plurality of encodings or embeddings generated from the predetermined valid slot values; determine phonetic similarity scores for the predicted slot value against each valid slot value; and replacing the predicted slot value containing the transcription error with the valid slot value with highest phonetic similarity score.11 . The method of claim 10, wherein the contextualized word features and / or embeddings are generated using a trained language model BERT (Bidirectional Encoder Representations from Transformers).

12. The method of claim 10 or 11 , wherein the trained phonetic based phrase confusion- aware model is based on a sentence embedding model (SBERT) and trained based on pairs of phonetically similar and dissimilar phrases, wherein the phonetically similar pairs were constructed by pairing each slot value extracted from an original input word sequence against the corrected valid slot value, while the phonetically dissimilar pairs were constructed by pairing each slot value extracted from the original input word sequence against a different valid slot value.

13. The method of claim 10, 11 or 12, wherein the contextualized word features are inputted to a bidirectional Long Short-Term Memory (Bi-LSTM) layer to produce output word sequence features, wherein the output word sequence features are used for the intent classification and slot filling to generate the predicted intent and the corresponding predicted slot value.

14. The method of claim 13, wherein the Bi-LSTM layer is equipped with multi-head attention.

15. The method of any one of claims 10 to 14, wherein punctuation removal is performed on the text transcript using a data normalization module before processing the text transcript using the trained joint intent classification and slot filling model.

16. The method of any one of claims 10 to 15, wherein the trained phonetic based phrase confusion-aware model is configured to generate sentence or word embeddings of the predicted slot value and each valid slot value and compute a cosine similarity of the sentence or word embedding of the predicted slot value against that of each valid slot value as the phonetic similarity score.

17. The method of any one of claims 10 to 16, wherein the method is used to decipher voice commands for ordering of cargo / shipment and / or retrieval of cargo / shipment status in warehousing.

18. The method of any one of claims 10 to 17, wherein phonetic encodings are generated for the predicted slot value and each valid slot value, and Levenshtein distance is calculated between these encodings to determine the phonetic similarity score, wherein the valid slot value with calculated Levenshtein distance indicative of highest phonetic match has the highest phonetic similarity score.

19. A non-transitory computer-readable medium storing a transcription error correction program including instructions that, when executed by a processor, causes an apparatus to: receive a text transcript of a voice command transcribed by an Automatic speech recognition (ASR) engine; run a trained joint intent classification and slot filling model configured to generate contextualized word features and / or embeddings from a word sequence detected in the text transcript and to subject the generated word features and / or embeddings to intent classification and slot filling to generate a predicted intent and a corresponding predicted slot value: and run a trained phonetic based phrase confusion-aware model for correcting a transcription error in the predicted slot value, wherein the trained phonetic based phrase confusion-aware model is configured to: compare the predicted slot value and / or an encoding or embedding generated from the predicted slot value against a plurality of predetermined valid slot valuesand / or a plurality of encodings or embeddings generated from the predetermined valid slot values; determine phonetic similarity scores for the predicted slot value against each valid slot value; and 20. replace the predicted slot value containing the transcription error with the valid slot value with highest phonetic similarity score.

Images