Training end-to-end speech understanding systems with disordered entities

Abstract

A computer-implemented method, system, and computer program product are disclosed. According to embodiments, training data can be received, the training data can include speech and pairs of meaning representations associated with the speech as ground truth data. The meaning representations include at least semantic entities associated with the speech, where an order of utterance of the semantic entities is unknown. The semantic entities of the meaning representations in the training data can be reordered into an order of utterance of the associated speech using an alignment technique. A spoken language understanding machine learning model can be trained using the pairs of speech and pairs of meaning representations with the reordered semantic entities. The meaning representations in the received training data can be perturbed, e.g., the semantic entities, to create a random order sequence variant of the semantic entities associated with the speech. The perturbed meaning representations together with the associated speech can augment the training data.

Description

Technical Field

[0001] This application generally relates to computers and computer applications, spoken language understanding, encoders, decoders, attention models, and speech recognition, and particularly to the training of unordered entities in spoken language understanding systems. Background Technology

[0002] Spoken language understanding (SLU) systems are traditionally cascades of automatic speech recognition (ASR) systems and natural language understanding (NLU) systems. The former converts speech into text, and the latter then interprets the meaning of the text. Generally, ASR and such traditional SLU systems are trained using verbatim transcripts. A drawback is the cost of accurately transcribing each word in verbatim transcripts. Summary of the Invention

[0003] The summary of this disclosure is provided to aid in understanding computer systems and methods for training entities in an end-to-end spoken language understanding system, which are not necessarily given in the order in which entities are spoken, and is not intended to limit the disclosure or the invention. It should be understood that various aspects and features of this disclosure may be used advantageously alone in some cases, or in combination with various aspects and features of this disclosure in others. Therefore, variations and modifications can be made to the computer system and / or its operation to achieve different effects.

[0004] On one hand, a computer-implemented method may include receiving speech and a pair of meaning representations associated with the speech, wherein the meaning representations include at least semantic entities associated with the speech, and wherein the spoken order of the semantic entities is not necessarily known, for example, is unknown. The method may further include using alignment techniques to reorder the semantic entities to the spoken order of the speech. The method may also include using the speech and the pair of meaning representations with the reordered semantic entities to train a spoken language understanding machine learning model.

[0005] On the other hand, a computer-implemented method may include receiving speech and a pair of meaning representations associated with the speech, wherein the meaning representations include at least semantic entities associated with the speech, and wherein the order in which the semantic entities are spoken is not necessarily known, for example, unknown. The method may further include using an alignment technique to reorder the semantic entities to the order in which the speech is spoken, wherein the alignment technique includes acoustic keyword recognition used in conjunction with a hybrid speech recognition model. The method may also include using the speech and the pair of meaning representations with the reordered semantic entities to train a spoken language understanding machine learning model.

[0006] On the other hand, a computer-implemented method may include receiving speech and a pair of meaning representations associated with the speech, wherein the meaning representations include at least semantic entities associated with the speech, and wherein the order in which the semantic entities are spoken is not necessarily known, for example, unknown. The method may further include using an alignment technique to reorder the semantic entities to the order in which the speech is spoken, wherein the alignment technique includes using time stamps derived from an attention model. The method may also include using the speech and the pair of meaning representations with the reordered semantic entities to train a spoken language understanding machine learning model.

[0007] On the other hand, a computer-implemented method may include receiving speech and a pair of meaning representations associated with the speech, wherein the meaning representations include at least semantic entities associated with the speech, and wherein the order in which the semantic entities are spoken is not necessarily known, for example, unknown. The method may further include using alignment techniques to reorder the semantic entities to the order in which the speech is spoken. The method may further include training a spoken language understanding machine learning model using the speech and the pair of meaning representations with the reordered semantic entities. The method may further include augmenting the received speech and meaning representation pairs to include randomized sequence variations of the semantic entities. Training the spoken language understanding machine learning model may include pre-training the spoken language understanding machine learning model using the augmented speech and meaning representation pairs, and training the pre-trained spoken language understanding machine learning model using the reordered semantic entities.

[0008] On one hand, a computer-implemented method may include receiving speech and a pair of meaning representations associated with the speech, wherein the meaning representations include at least semantic entities associated with the speech, and wherein the order in which the semantic entities are spoken is not necessarily known, for example, unknown. The method may further include using alignment techniques to reorder the semantic entities to the order in which the speech is spoken. The method may also include training a spoken language understanding machine learning model using the speech and the pair of meaning representations with the reordered semantic entities. This includes inputting a given speech into the trained spoken language understanding machine learning model, wherein the trained spoken language understanding machine learning model outputs a set of predictions including intention labels and semantic entities associated with the given speech.

[0009] On the other hand, a computer-implemented method may include receiving training data. The training data may include speech and pairs of meaning representations associated with the speech. The meaning representations include at least semantic entities associated with the speech, wherein the order in which the semantic entities are spoken is unknown, for example, not necessarily known. The method may further include augmenting the training data by perturbing the semantic entities to create random sequence variations of the semantic entities. The method may further include pre-training a spoken language understanding machine learning model using the augmented training data, wherein different random sequence variations of the semantic entities are used at different times during training. The spoken language understanding machine learning model can be pre-trained to output intention labels and semantic entities associated with a given input speech.

[0010] On the other hand, a computer-implemented method may include receiving training data. The training data may include speech and pairs of meaning representations associated with the speech. The meaning representations include at least semantic entities associated with the speech, wherein the order in which the semantic entities are spoken is unknown, for example, not necessarily known. The method may further include augmenting the training data by perturbing the semantic entities to create random sequence variations of the semantic entities. The method may further include pre-training a spoken language understanding machine learning model using the augmented training data, wherein different random sequence variations of the semantic entities are used at different times during training. The spoken language understanding machine learning model can be pre-trained to output intention labels and semantic entities associated with a given input speech. The method may further include fine-tuning the pre-trained spoken language understanding machine learning model using the alphabetically ordered semantic entities.

[0011] A system comprising at least a processor and a memory device may also be provided, wherein at least one or more processors may be configured to perform any one or more methods described herein.

[0012] A computer-readable storage medium may also be provided that stores machine-executable instructions to perform one or more of the methods described herein.

[0013] The structure and operation of each embodiment are described in detail below with reference to the accompanying drawings. In the drawings, the same reference numerals indicate elements that are the same or have similar functions. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the end-to-end (E2E) spoken language understanding (SLU) system in the embodiment.

[0015] Figure 2 The HMM corresponding to the constituent phonemes in the example keywords is shown in the embodiment.

[0016] Figure 3 An example attention map from an embodiment is shown.

[0017] Figure 4 This is a flowchart illustrating a method for training an end-to-end (E2E) spoken language comprehension (SLU) machine learning model in an exemplary embodiment.

[0018] Figure 5 This is a schematic diagram illustrating a method for training an end-to-end (E2E) spoken language comprehension (SLU) machine learning model in another embodiment.

[0019] Figure 6 This is a schematic diagram showing components of a system capable of training a spoken language understanding machine learning model or system in one embodiment.

[0020] Figure 7 A schematic diagram of a computer or processing system that can be implemented according to one embodiment is shown.

[0021] Figure 8 A cloud computing environment in one embodiment is described.

[0022] Figure 9 This illustrates a set of functional abstraction layers provided by a cloud computing environment in an embodiment of the present invention. Detailed Implementation

[0023] In one or more embodiments, systems, methods, and techniques may be provided that can improve the training of end-to-end language understanding. Figure 1 This is a schematic diagram illustrating an end-to-end (E2E) spoken language understanding (SLU) system in an exemplary embodiment. An E2E SLU system may include one or more computer-implemented components, such as components implemented and / or running on or coupled to one or more hardware processors. For example, one or more hardware processors may include programmable logic devices, microcontrollers, memory devices, and / or other hardware components that can be configured to perform the corresponding tasks described in this disclosure. Coupled memory devices may be configured to selectively store instructions executable by one or more hardware processors. A processor may be a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), another suitable processing component or device, or one or more combinations thereof. The processor may be coupled to a memory device. This memory device may include random access memory (RAM), read-only memory (ROM), or another memory device, and may store data and / or processor instructions for implementing various functions associated with the methods and / or systems described herein. The processor may execute computer instructions stored in memory or received from another computer device or medium.

[0024] End-to-end (E2E) SLU systems can process speech input directly without intermediate text transcription. These SLU systems can be trained on a set of entities and utterance-level intents rather than on word-for-word transcription, significantly reducing data collection costs. In one or more embodiments disclosed herein, the systems, methods, and techniques described herein allow E2E SLU systems to process training data in which entities or semantics are not necessarily given in the order they are spoken.

[0025] In an end-to-end (E2E) spoken language understanding (SLU) system, the input can be speech (such as audio or acoustic signals), and the output can be a semantic representation. For example, speech 102 can be input to SLU module 104, which can include machine learning models such as neural networks or deep learning models, such as, but not limited to, recurrent neural network transducers (RNN-T) and / or attention-based encoders / decoders. SLU module 104 can output a semantic representation 106 of the speech, such as one or more intentions and entities.

[0026] For example, Spoken Language Understanding (SLU) module 104 can provide meaning representations corresponding to the input speech, such as intent detection and entities. In one aspect, the SLU system does not need to provide each word, or the manner of pronunciation of the input (e.g., entity order, word selection), when providing output. Examples of outputs provided by SLU module 104 include:

[0027] Complete transcription + semantic tags: (INT-flight) I would like to make a reservation for a flight to Denver (B-toCity) from Philadelphia (B-fromCity) on this coming Sunday (B-departDate).

[0028] Semantic entities in the order they are spoken: (INT-flight)Denver(B-toCity)Philadelphia(B-fromCity)Sunday(B-departDate))((INT-flight)Denver(B-destination)Philadelphia(B-departure)Sunday(B-departureDate));

[0029] Semantic entity set (order unknown):

[0030]

[0031] SLU module 104 can be trained on a training data corpus, which includes pairs of speech and meaning (intent and entity). For example, a pair may include speech and its corresponding meaning. Entities are also called slots for slot filling. For example, user commands or queries (speech) are interpreted by extracting intent and relevant slots. Such a corpus can be generated by manual labeling or an automated process (e.g., outputting an SLU with such labels for a given speech or pronunciation). For example, a query such as "show flights from Seattle to San Diego tomorrow" could have the following meaning representation.

[0032] Intent: Flight Information

[0033] slot (solid):

[0034] fromloc(origin):Seattle

[0035] destination: San Diego

[0036] depart_date(departure_date):tomorrow(tomorrow)

[0037] Table 1 shows examples of intentions and slot filling corresponding to utterance or speech. The notation shown in Table 1 uses the Begin-Inside-Outside notation, also known as BIO notation. In BIO notation, semantic entities with multiple constituent words are marked with "B" and then marked with "I", for example, "New B-fromlocYork I-fromlocCity I-fromloc". Non-entity words are marked with "O" to indicate that they are "outside" entities.

[0038] Table 1

[0039]

[0040] For example, SLU module 104 predicts a set of semantic entities based on the provided speech. Different expressions can convey the same meaning. Consider the following example speech or expression:

[0041] "I want to fly to Dallas from Reno that makes a stop in Las Vegas."

[0042] - "Make reservation to Dallas from Reno with a stop in Las Vegas."

[0043] - "Depart Reno for Dallas with Las Vegas stopover."

[0044] "I'm currently in Reno and have my next client meetings in Dallas, so I need to book a flight, but I also want to have a stop in Las Vegas."

[0045] Table 2 shows examples of the entity sets and intentions of the aforementioned speech or expressions. In one or more embodiments, the systems, methods, and / or techniques of the present invention can improve E2E SLU models to perform predictions of semantic entity sets. All the examples above have the same meaning in general, and therefore can be mapped to the same simplified meaning representation: entity set and vocal level intention, examples of which are shown in Table 2.

[0046] Table 2

[0047] fromloc.city_name(origin.city_name) Reno stoploc.city_name(stopover point.city_name) Las Vegas toloc.city_name(destination.city_name) Dallas

[0048] exist Figure 1 In the E2E SLU system shown, training can be performed using semantic entities and phonological intentions without requiring word-for-word transcription. The set of entities to be modeled can be given in the order of speech (e.g., the order in which the entities are spoken in their respective speech), or the order can be left unspecified.

[0049] ASR technology generates word-for-word transcriptions and target words to ensure word accuracy. SLU systems attempt to infer the correct meaning from utterances (e.g., Table 2), regardless of factors such as entity order or word selection. For example, in an embodiment, while an SLU model could also be trained to output all spoken words, such as a complete transcription, the success of the SLU model can be determined by the set of semantic labels and values ​​extracted by the SLU. An example metric for measuring the success of an SLU model could be the F1 score. If the SLU model outputs all words, it can also be used as ASR; in this case, the success of the SLU model can be measured by the word error rate (WER). In one respect, SLU can be viewed as an ensemble prediction problem compared to a sequence prediction problem.

[0050] End-to-end sequence-to-sequence models can be flexibly trained on different types of ground truth. For speech recognition, the training data is speech with word-for-word transcription, as shown in Example (0) of Table 3. To train the SLU model, sentences are annotated with entity labels, as shown in Example (1) of Table 3, along with labels representing the intention of the entire utterance. In Example (2) of Table 3, entities are presented in the natural speaking order for training. The difference between Example (2) and Example (1) in Table 3 is that all words that are not entities are excluded. Entities can be considered more important key phrases; however, other words also play an important role. For example, “to” and “from” are obviously crucial for determining whether a city is the destination city or the departure city. The SLU model may not output such words, but the speech signals corresponding to these words can help the SLU model output the correct entity labels.

[0051] Table 3

[0052]

[0053] In one respect, if the order of the entity set in the training data is unknown, the task can be viewed as a set prediction task. Since training a sequence-to-sequence model requires a target output sequence, in example (3), the basic facts can be normalized by entities ordered alphabetically by label names (e.g., stoploc.city name).

[0054] In cascaded SLU systems, classic ASR or NLU models may be difficult to train on this type of data, but this data type can be very rich and the collection cost is much lower. Consider a human agent talking to a customer to make a travel reservation, while the agent simultaneously performs actions such as filling out web forms or other database transaction records, which can be translated into basic facts like in example (3). To train ASR and NLU separately, accurate verbatim transcription of speech data could take 5-10 times longer than a human transcriber, plus the additional cost of labeling entities. In contrast, transaction records containing sets of entities can be obtained from helping customers and may not incur additional costs.

[0055] In one aspect, the SLU system can be trained to predict a set of entities from speech. In embodiments, one or more speech models are such as, but not limited to, recurrent neural network (RNN)-transformers (RNN-Ts), or attention-based encoder-decoder models such as those with LSTM encoders and / or Conformer encoders. Due to monotonic input-output alignment constraints, RNN-Ts are likely to struggle to learn from the fundamental fact that entities are not arranged in the order they are spoken. Attention-based models are likely to perform better because they can focus on potentially discontinuous, relevant parts of the speech signal. As described more fully below, in one or more embodiments, for ensemble prediction, data augmentation and explicit entity alignment can be used as methods to improve performance.

[0056] In one aspect, one or more modeling techniques disclosed herein can handle various semantic entities and intention sequences on the output label side. In another aspect, it is not necessary to assume that the SLU training label sequences are in a spoken order. For example, the systems and methods disclosed herein can treat the target output sequence as a set.

[0057] In one aspect, the data augmentation method disclosed herein is performed at the output label level. The meaning of the input speech signal generated by the E2E SLU system can be represented as a set of entities and intentions, for example, it does not have to be a complete word-for-word transcription. Locating such a set of SLU tokens can be handled in a manner similar to keyword search, where a specific word or set of words is detected in spoken phonation. Another problem that the systems and / or methods disclosed herein can address is how the SLU model can automatically perform this task or discover SLU tokens without an explicit keyword search step. In one or more embodiments, the systems and / or methods can implement set-based data augmentation and / or set reordering for training the acoustic model.

[0058] In one respect, end-to-end models directly map acoustic feature sequences to symbol sequences without requiring conditional independence assumptions. Alignment issues arising from the lengths of the input and target sequences can be addressed differently depending on the end-to-end approach. Examples of models that can be used with SLU include the following models for speech recognition. Other models can be used or modified.

[0059] RNN converter model

[0060] RNN-T introduces special BLANK symbols and lattice structures to align input and output sequences. These models can include three distinct subnetworks: a transcription network, a prediction network, and a joint network. The transcription network produces acoustic embeddings, while the prediction network resembles a language model because it is conditioned on previously generated non-BLANK symbols. The joint network combines the two embedding outputs to produce a posterior distribution over the output symbols, including the BLANK symbols. An RNN-T-based SLU model can be created in two steps: constructing an ASR model; then, adapting it to an SLU model through transfer learning. In the first step, the model is pre-trained on a large amount of general ASR data to allow it to efficiently learn how to transcribe speech into text. Since the target in the pre-training step is only graphemic / phonetic tokens, semantic labels are added as additional output targets before adapting the model with SLU data. These new SLU labels are integrated by adjusting the size of the output and embedding layers of the prediction network to include the additional symbols. The new network parameters are initialized randomly, while the rest are initialized from a pre-trained network. Once the network is modified, it is trained on SLU data following a similar procedure to training the ASR model.

[0061] Attention-based LSTM encoder-decoder model

[0062] This model estimates the posterior probability of a sequence without introducing any explicit hidden variables. The alignment problem is handled internally by dynamically compressing the input stream with a trainable attention mechanism in sync with the output sequence. The model is capable of handling non-monotonic alignment problems. The RNN-T and attention encoder-decoder models have similar structures. The attention-based model also includes an LSTM-based encoder network to generate acoustic embeddings. The single-head LSTM decoder contains components similar to those in a language-like model, as well as an attention module that combines the acoustic embeddings and symbol sequence embeddings into a context vector to predict the next symbol. The attention-based encoder-decoder ASR model can be adapted to SLU using the same steps described for RNN-T.

[0063] Attention-based Conformer encoder-decoder model

[0064] In one embodiment, an attention mechanism can be added to the encoder of the encoder-decoder model. A Conformer is a combination of a convolutional neural network and a self-attention-based transformer, enabling speech recognition. In one embodiment of the attention model, the encoder can be a Conformer. In another embodiment, the decoder can be a Conformer.

[0065] In various embodiments, speech paired with basic fact semantic entities given in an unspecified order (e.g., not necessarily in the order they are spoken) can be used, for example, using data where semantic entities are provided in an unspecified order, to train (e.g., as...). Figure 1 The example illustrates an end-to-end speech understanding (SLU) system. In one embodiment, one or more SLU alignment methods may be provided to infer the speaking order of semantic entities in training data to prepare the speaking order data. In another embodiment, supplementary methods for data augmentation may be provided, wherein semantic entities are presented in a random order during model pre-training to desensitize the model to the variability of entity ordering in the underlying facts.

[0066] Advantageously, the systems and methods disclosed herein can reduce the cost of annotation, for example, when the underlying facts of the training data can be semantic entities whose speaking order is unknown or unspecified. In embodiments, even if the semantic entities provided during training may not be in speaking order, end-to-end models such as attention-based encoder-decoder models or recurrent neural network converter (RNN-T) models can be used to model semantic labels that are not directly associated with acoustic events present in the speech signal. Advantageously, for example, the systems and methods disclosed herein can allow the use of monotonic (non-reordered) models such as RNN-Ts, which can be used for ASR and SLU, improving SLU performance (F1 score) even when the speaking order of entities in the training data is unknown, for example, improving SLU performance to similar levels to SLU trained on full transcription or entities in speaking order.

[0067] In embodiments, the SLU alignment method disclosed herein may include inferring the speaking order and reordering the set of semantic entities to the speaking order for use in SLU model training. In embodiments, the set-based data augmentation techniques disclosed herein may include creating random order variations of spoken entities to make the SLU model more robust to the order of entities in the underlying facts used for training.

[0068] Different approaches can be used for SLU alignment. In one embodiment, an SLU alignment method for finding the underlying spoken order of an entity set can be used in the process used for keyword search. When performing acoustic keyword recognition, a combination of multiple (e.g., two) acoustic models can be used. For example, the keyword to be searched is modeled by its underlying phonetic string, while all non-keyword speech is modeled by a garbage model. For example, using a conventional hybrid ASR model, a cascade of Hidden Markov Models (HMMs) corresponding to the constituent phonemes in the keyword can be constructed for the keyword to be searched. A phoneme is a phonetic representation of a phoneme (the actual sound). The garbage model can be represented by general phonemes for spoken speech and including background noise, such as silence. The method then chains these models together in sequence: first the garbage model, then the keyword model, and finally the garbage model, then uses the ASR model to force alignment between the spoken and keyword models. Implementations of this SLU alignment method can be used to arrange semantic entities in spoken order to, for example, improve set predictions of SLU.

[0069] Figure 2 The HMM corresponding to the constituent phonemes in the example keywords is displayed. For example, reordering the set to spoken order using a hybrid ASR model can include explicit keyword-based alignment. In an embodiment, for each entity value, to find an approximate time, the alignment method can construct an HMM (garbage-keyword-garbage) and, for example, perform forced alignment (VN = vocal noise) for the entity value "Newark". Figure 2 In the example keywords (entity values) at positions 204, 206, 208, and 210, the constituent phonemes at positions 204, 206, 208, and 210 are represented. Noise is represented at positions 202 and 212. Using the temporal information of each entity, this alignment method can reorder them in the order they are spoken.

[0070] For example, consider the following example of a given set:

[0071]

[0072] Based on the spoken phrase "I would like to make a reservation for a flight to Denver from Philadelphia on Sunday," the set can be reordered as follows:

[0073] State the order: INT - flight Denver B - to City Philadelphia B - from City Sunday B - depart Date.

[0074] In another embodiment, the SLU alignment method can use attention values. In this embodiment, attention can be used to perform implicit internal alignment. The attention model can handle SLU entities that are not spoken in order, and single-head attention can have a clear focus on spoken tokens at corresponding temporal positions in the acoustic feature stream. Based on this observation, the speaking order of SLU phrases can be estimated. Then, the method can heuristically estimate the average temporal position of each SLU phrase when the speaking order of the phrases is unknown, and calculate the average temporal position of each SLU phrase, thereby re-establishing the speaking order of the phrases.

[0075] For example, in this embodiment, the SLU alignment method may include training an attention model based on the fundamental fact of alphabetical order, and using an attention map to determine the average temporal position of each SLU phrase. In this embodiment, when the order in which the phrases are spoken is unknown, the following heuristic estimates the average temporal position of each SLU phrase:

[0076]

[0077] Where, α t,n This represents the attention given to the nth output tag at each acoustic frame t. Let the i-th SLU phrase (including spoken BPE tags and entity tags) be at position n in the output sequence. i Begins at n i+1 -1, and let n i Locations containing only the BPE (verbal) sign. Figure 3 An example attention map is shown, where the x-axis represents time within the speech signal (corresponding to t), the y-axis contains the sequence of BPE flags and entity labels (corresponding to n, from top to bottom), and α... t,n The value is represented by the darkness of the pixel. In Figure 3 The image shows an attention graph of the phrase "I would like to make a reservation for a flight to Denver from Philadelphia on this coming Sunday," where the basic facts are entities arranged alphabetically by their label names. Considering only spoken markers, Equation 1 calculates the average temporal position of each SLU phrase, thereby re-establishing the phonological order of the phrases.

[0078] For set prediction problems, a system and / or method can be provided with a set of entities whose speaking order is unknown. Set prediction problems refer to predicting the meaning representation of a given speech utterance or input speech utterance, which can include intentions and entities. For example, basic fact data (given a set of entities whose speaking order is unknown) can be used to train a SLU model, such as a sequence-to-sequence model. In an embodiment, to train a sequence-to-sequence model, the system and / or method may arbitrarily choose to normalize the entity order by sorting the label names (e.g., fromCity) alphabetically. To further improve robustness, the system and / or method can use or implement data augmentation that randomizes the order of entity and intention labels in the basic facts used to pre-train various E2E models. During this pre-training phase, the model can be presented with different versions of basic facts at each epoch. As an example, the following illustrates the randomized ordering of entities and intent labels, which can be used, for example, to pre-train sequences with different orderings used at different times (e.g., using a different ordering sequence for each time period than for another time period): Sunday (B-departDate) Philadelphia (B-fromCity) Denver (B-toCity) INT_flight; Philadelphia (B-fromCity) INT_flight Sunday (B-departDate) Denver (B-toCity); INT_flight Denver (B-toCity) Sunday (B-departDate) Philadelphia (B-fromCity); and so on. In the example format of these sets, the entity labels are shown in parentheses after the entity values. A fine-tuning phase can follow the pre-training phase, where the model is trained on basic facts with entities arranged alphabetically. Exposing the model to instances where the entity order does not match the basic facts during pre-training allows the model to learn better during fine-tuning.

[0079] In one or more embodiments, the system and / or method can train a spoken language understanding system. SLU training data may be provided as an unordered set of semantic entities (such as labels and values). In one or more embodiments, the system and / or method may use an SLU alignment technique to reorder the unordered set of semantic entities. In embodiments, the SLU alignment technique reorders the data to speaking order, including an alignment scheme based on acoustic keywords suitable for use with a hybrid speech recognition model. In embodiments, the SLU alignment technique reorders the data to speaking order using time stamps derived from the attention mechanism of an end-to-end SLU model. In one or more embodiments, an attention model may be trained on the SLU data (a set of unordered semantic entities) before alignment and reordering the data. This may be useful, for example, if the SLU data has an acoustic mismatch with the original speech model, such as noisy speech. In one or more embodiments, the system and / or method may use the reordered data to train an SLU system. In one or more embodiments, the system and / or method may pre-train an SLU model using a set-based data augmentation scheme targeting semantic entities. In one embodiment, the set-based data augmentation method can randomize the order of entities and intention labels in the available training data. In one or more embodiments, the system and / or method can train an SLU system using reordered data after pre-training it with a set-based data augmentation scheme.

[0080] One or more SLU models can be trained using, for example, basic factual data that may be available. For instance, one or more SLU models can be trained based on an application-specific corpus of data for a particular application (e.g., a particular domain).

[0081] For example, in the example implementation of the embodiments, data such as that from the Air Travel Information System (ATIS) can be used for training (e.g. Figure 1The SLU model shown in Figure 104 uses the publicly available Language Data Consortium (LDC) corpus, ATIS. For example, there could be 4976 training audio files (~9.64 hours, 355 speakers) downsampled to 8kHz and 893 test audio files (~1.43 hours, 355 speakers). In this example, in the embodiment, to better train the E2E model, an additional copy of the corpus can be created using velocity / beat perturbation, resulting in ~140 hours of training time. In the example, in the embodiment, to simulate real-world operating conditions, a second noisy ATIS corpus can be created by adding street noise with a signal-to-noise ratio (SNR) between 5 and 15 dB to the clean recordings. This ~9.64 hours of noisy training data can also be extended to ~140 hours through data augmentation. A corresponding noisy test set can also be prepared by disrupting the original clean test set with additional street noise with a 5 dB SNR. In this embodiment, in an example example, intent recognition performance can be measured by intent accuracy, while slot filling performance can be measured by F1 score. When using voice input instead of text, words are also predicted and errors may occur. A true positive can be correct only if both the entity label and value are present. For example, if the reference is toloc.city name:new york, but the decoded output is toloc.cityname:york, then in this embodiment, both false negatives and false positives can be counted. Scoring does not require knowledge of the entity order, thus it is applicable to “entity set” prediction.

[0082] The following are examples of use cases for implementing SLU according to various embodiments. In the embodiments, SLU can be implemented using an RNN-T model. In one example, the RNN-T model for SLU can be pre-trained on task-independent ASR data. For example, an ASR model trained on data from an available corpus can be used. A connectionist temporal classification (CTC) acoustic model can be trained and used to initialize the transcriptional network of the RNN-T model. For example, the RNN-T model can have a transcriptional network containing 6 bidirectional LSTM layers, with 640 units per layer in each direction. The prediction network is a single unidirectional LSTM layer with 768 units. The joint network projects the 1280-dimensional stacked encoder vector from the last layer of the transcriptional network and the 768-dimensional prediction network embeddings to 256 dimensions respectively, multiplies and combines them, and applies hyperbolic tangent. After this, the output is projected to 46 logits corresponding to 45 characters plus BLANK, followed by a softmax layer. The model has a total of 57M parameters. These models can be trained for 20 epochs in PyTorch. Other design and implementation choices, and hyperparameters, are possible. During SLU adaptation, new network parameters are randomly initialized, while the rest of the network is copied from the pre-trained network. Depending on the entity / intention task, additional output nodes (e.g., 151) can be added as entity / intention targets to the pre-trained network.

[0083] In another embodiment, SLU can be implemented using an attention-based LSTM encoder-decoder SLU model. In this embodiment, the attention-based E2E model can have a 6-layer bidirectional LSTM encoder and a 2-layer unidirectional LSTM decoder, modeling the posterior probabilities of approximately 600 BPE units augmented with entity and intention labels. The number of nodes in each LSTM layer can be 768 per direction. The first LSTM of the decoder operates only on the embedded predicted symbol sequence, while the second LSTM processes both acoustic and symbolic information using a single-headed additional position-aware attention mechanism. The dropout rate and drop-connect rate are set to 0.3 in the encoder and 0.15 in the decoder. Additionally, a zoneout with a probability of 0.10 can be applied to the second LSTM layer of the decoder. Overall, the model can contain 57M parameters. For ASR pre-training, the standard Switchboard-300 corpus can be used, and the model can be optimized using AdamW in batches of 192 sequences over 450k update steps. SLU fine-tuning can be performed in batches of 16 sequences over approximately 100k steps. Other design and implementation options and hyperparameters are also possible.

[0084] In another embodiment, the SLU can be implemented using an attention-based Conformer encoder-decoder SLU model. In this embodiment, to add self-attention to the encoder, the LSTM encoder can be replaced with a Conformer encoder. Overall, the model can contain 68M parameters. Other design and implementation options, and hyperparameters, are possible.

[0085] Various experiments were conducted separately for training the SLU model using the following: 1) complete word-for-word transcripts with semantic labels for adapting the ASR model to the SLU model; 2) basic facts containing only entities with natural speaking order; and 3) basic facts containing entities with unknown speaking order, using data augmentation and / or one or more pre-alignment methods. The experiments demonstrate that accurate SLU models can be trained using one or more methods described in this paper, even when the basic fact entities have unknown speaking order.

[0086] For example, one approach is to apply data augmentation, where the model is exposed to the fundamental fact of entities with various random orderings during the pre-training phase, followed by fine-tuning with entities arranged alphabetically. For instance, in RNN-T models, random order augmentation can improve performance, such as under noisy conditions. Data augmentation can help the model compensate for the various types of noise that need to be processed during training. Data augmentation helps to better tune the model when dealing with acoustic noise and label mismatches. The diverse data introduced through data augmentation can improve the model. For example, for attention-based encoder-decoder models, consistent improvements can be observed using random order data augmentation under both clean and noisy conditions. Similarly, improvements can be seen using the Conformer encoder under both clean and noisy conditions.

[0087] This method can also infer the speaking order of entities by aligning entities with speech, and then use this basic fact to train an SLU model. In one embodiment, one approach for alignment can be based on a hybrid ASR model. In another embodiment, one approach for alignment can be based on an attention model. For RNN-T models, inferring the speaking order of entities and training on the basic fact of alignment helps improve performance. Improvements can also be observed in training on the basic fact of alignment for attention-based encoder-decoder models and Conformer encoders.

[0088] In the embodiments, both the data augmentation method and the pre-alignment method can be used to train the SLU model. The method can be initialized using a model pre-trained on entities in a random order, and then fine-tuned based on the reordered fundamental facts. Experiments also show performance improvements for the SLU model, for example, in different types of models such as attention-based encoder-decoder models, Conformer encoders, and RNN-T, and under both clean and noisy conditions.

[0089] Spoken language comprehension (SLU) systems can determine the meaning of input speech signals, while speech recognition, for example, aims to generate word-for-word transcriptions. End-to-end (E2E) speech modeling can be trained only on semantic entities whose collection cost is lower than word-for-word transcription. This ensemble prediction problem may have unspecified entity order. Systems and / or methods in one or more embodiments improve E2E models, such as RNN converters and attention-based encoder-decoders, to enable the use of training data in which the entity sequences trained are not necessarily arranged in the order of speech. In one or more embodiments, when the order of speech of entities is unknown, the system and methods can improve E2E models by leveraging data augmentation techniques and implicit attention-based alignment methods to infer the order of speech.

[0090] Figure 4 This is a flowchart illustrating a method for training an end-to-end speech comprehension machine learning model in an exemplary embodiment. The method may be executed by one or more computer processors (e.g., hardware processors) or implemented on one or more computer processors. At 402, the method may include receiving training data, such as speech and a pair of meaning representations associated with the speech. The meaning representations may include at least semantic entities associated with the speech, wherein the order in which the semantic entities are spoken is unknown. Examples of meaning representations associated with speech are shown above in Table 2. The meaning representations may also include intention tags associated with the speech. The speech may be received in the form of sound, acoustic, or audio signals.

[0091] At 404, the method may include using alignment techniques to reorder semantic entities to the spoken order of speech. In embodiments, the SLU alignment disclosed herein may be used with a model to infer the spoken order and readjust training data. In embodiments, alignment techniques may include acoustic keyword recognition used in conjunction with a hybrid speech recognition model. For example, as referenced above... Figure 2 The embodiment of the alignment technique may include the use of a hybrid ASR with a Hidden Markov Model (HMM). The acoustic model of the HMM hybrid ASR can convert input speech or words into speech sequences. Figure 2The image shows a speech sequence of example keywords. In an embodiment, the method may include generating an Hidden Markov Model (HMM) with speech units ordered by vocalized noise for each keyword (e.g., a semantic entity) in the speech. The method may align the HMM (e.g., the ordered speech units) with the speech and extract or acquire the approximate time or temporal position of the keyword in the speech. The keywords (e.g., semantic entities) in the speech may then be sorted according to their temporal position in the speech, for example, in chronological order (earlier times appear earlier). In this way, the method can infer the order in which the semantic entities in the speech are spoken. This inferred order of speech of the semantic entities can be used to train a SLU model.

[0092] In another embodiment, the alignment technique includes using time stamps derived from the attention model. The attention model can first be adapted to the domain SLU data: speech paired with the fundamental fact that the order of semantic entities in which they are unknown. For example, an attention-based speech recognition model or SLU model can be modeled and / or run, thereby generating an attention map. For example, the attention model can be trained alphabetically to generate and use the attention map to infer the speaking order. Figure 3 The image shows an example of an attention map. An attention map displays the attention value over time for each supposed marker in speech. For example, it shows a sub-word unit using Byte Pair Encoding (BPE), commonly used in speech recognition. For instance, decoding “D@@EN@@VER” allows the construction of the word “DENVER”. (Reference) Figure 3 In the example graph shown, "Sunday" appears around the 8-second time marker, "Philadelphia" appears around the 6-8 second time marker, and "Denver" appears between the 4-6 second time marker. Using attention maps, the maximum or average time marker or time stamp of a specific word or semantic entity can be calculated. For example, for "Sunday," all hypothetical time positions of the phonological units of "Sunday" can be extracted and averaged to generate an approximate time stamp for the word. The timing of keywords (e.g., semantic entities) can be inferred from these time markers or stamps. For example, semantic entities can be sorted based on their time stamps (e.g., sorted by increasing time stamps). Semantic entities sorted by the order in which they are spoken can be used to train an SLU model.

[0093] In one aspect, the speech may include noisy speech data, and the attention model may be adjusted to suit the noisy speech data.

[0094] In 406, the method may include training a spoken language understanding machine learning model using a pair of speech and meaning representations with reordered semantic entities. The spoken language understanding machine learning model is trained on the speech as input and the meaning representations as output as basic facts so that, for a given new speech, it can predict a meaning representation corresponding to or associated with the new speech. This meaning representation may include, for example, intention labels and semantic entities that can represent the meaning of the speech.

[0095] In embodiments, the method may further include augmenting the received speech and meaning representation pairs to include random order sequence variations of semantic entities. The method may include pre-training a spoken language understanding machine learning model with the augmented speech and meaning representation pairs. Training is performed at 406, and then the pre-trained language understanding machine learning model is trained with reordered semantic entities.

[0096] In this embodiment, the pre-trained spoken language understanding machine learning model can be further pre-trained, refined, or fine-tuned using alphabetically ordered semantic entities, for example, prior to training at point 406. For instance, the parameters of the pre-trained spoken language understanding machine learning model can be further adjusted based on training using alphabetically ordered semantic entities. In this embodiment, training at point 406 may include training this fine-tuned spoken language understanding machine learning model.

[0097] Spoken language comprehension machine learning models can be neural networks. Examples can include, but are not limited to, RNN-T and end-to-end encoder-decoder models.

[0098] In section 408, a trained spoken language understanding machine learning model can be used or run, wherein, for a given input speech (e.g., an acoustic signal), the trained spoken language understanding machine learning model outputs or predicts a semantic representation associated with that speech, which is, for example, referred to as ensemble prediction, including predicted intention labels and semantic entities associated with the given speech. On one hand, the training and running of the trained model can be performed on different processors (or sets of processors) or on the same processor (or set of processors). For example, a trained model can be imported or exported from the processor where it was trained to another processor and then run. A trained model can also be run on the processor or set of processors where it was trained.

[0099] Figure 5This is a schematic diagram of a method for training an end-to-end speech understanding system according to an exemplary embodiment. The method may be executed by one or more computer processors (e.g., hardware processors) or implemented on one or more computer processors. At 502, training data may be received, which may include speech and speech-associated meaning representation pairs. The meaning representations may include at least semantic entities associated with the speech, wherein the order in which the semantic entities are spoken is unknown. Examples of speech-associated meaning representations are shown above in Table 2. Meaning representations may also include intention tags associated with the speech. Speech may be received in the form of sound, acoustic, or audio signals.

[0100] In a 504 error, training data can be augmented by perturbing the semantic entities in the received training data to create randomized sequential variants of semantic entities. For example, as described above, consider the following semantic representation corresponding to the speech "I want to fly to Denver from Philadelphia on Sunday," which includes an intention label, entity labels, and a set of values:

[0101]

[0102] The entities that are spoken in the order of this speech are as follows.

[0103] State the order: INT - flight Denver B - to City Philadelphia B - from City Sunday B - depart Date.

[0104] The following sets illustrate instances of the randomized order of entities and intention tags.

[0105] Sunday(B-departDate)Philadelphia(B-fromCity)Denver(B-toCity)INT_flight;

[0106] Philadelphia(B-fromCity)INT_flight Sunday(B-departDate)Denver(B-toCity);

[0107] INT_flight Denver(B-toCity)Sunday(B-departDate)Philadelphia(B-fromCity).

[0108] In 506, a spoken language understanding machine learning model (e.g., a neural network model) can be pre-trained using augmented training data, where different randomized sequence variations of semantic entities can be used at different training epochs. For example, during training, different randomized orders of entities and intention labels can be used at each epoch. The spoken language understanding machine learning model can be pre-trained to output intention labels and semantic entities associated with a given input speech.

[0109] In section 508, a pre-trained spoken language understanding machine learning model can be further fine-tuned using alphabetically ordered semantic entities. For example, fine-tuning could include retraining the spoken language understanding machine learning model using alphabetically ordered semantic entities from the training data (received as part of the basic fact data). For instance, continuing the example above, a pre-trained SLU ML model could be fine-tuned using alphabetically ordered entities (e.g., entity labels arranged alphabetically): {INT_flight Sunday(B-departDate)Philadelphia(B-fromCity)Denver(B-toCity)}.

[0110] In an embodiment, at 510, a pre-trained spoken language understanding machine learning model can be run with new input, such as new speech utterances, so that the pre-trained spoken language understanding machine learning model outputs a meaning representation (e.g., SLU labels such as intention labels and entity labels, and their values). In an embodiment, for example, as referenced... Figure 4 As described, the meaning of the spoken sequence can be used to further train the pre-trained spoken language comprehension machine learning model. On the other hand, the data augmentation used for pre-training can use only alphabetical ordering, for example, without random sequence variations. Any combination of one or more data augmentations can be used.

[0111] In embodiments, the method may further include using alignment techniques to reorder semantic entities to the spoken order of speech, and using training data with the reordered semantic entities to further train a pre-trained spoken language understanding machine learning model, such as as referenced. Figure 4 As described in 404 and 406. For example, acoustic keyword recognition can be performed in conjunction with a hybrid speech recognition model to reorder semantic entities in the order of speech, as described above. In another embodiment, for example, semantic entities can be reordered in the order of speech using time stamps derived from an attention model. In embodiments, the attention model can be adapted to suit SLU labels (e.g., semantic entities).

[0112] Figure 6This is a schematic diagram illustrating system components that can train a spoken language understanding machine learning model or system in one embodiment. One or more hardware processors 602, such as a central processing unit (CPU), graphics processing unit (GPU) and / or field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), and / or another processor, may be coupled to a memory device 604 and generate predictive models and recommend communication opportunities. The memory device 604 may include random access memory (RAM), read-only memory (ROM), or another memory device and may store data and / or processor instructions for implementing various functions associated with the methods and / or systems described herein. One or more processors 602 may execute computer instructions stored in the memory 604 or received from another computer device or medium. For example, the memory device 604 may store instructions and / or data for the operation of one or more hardware processors 602 and may include an operating system and other instruction and / or data programs. One or more hardware processors 602 may receive training data, which may include, for example, pairs of speech and corresponding semantic representations (e.g., intent labels and / or semantic entities). For example, one or more hardware processors 602 can reorder semantic entities according to the phonological order of the corresponding speech, and use the speech and the semantic representation pairs with the reordered semantic entities to generate and / or train a spoken language understanding machine learning model. The spoken language understanding machine learning model can be trained to predict or output a semantic representation (e.g., intent labels and semantic entities) corresponding to or associated with a given input speech. Training data can be stored in storage device 606, or received from a remote device via network interface 608, and can be temporarily loaded into memory device 604 for building or generating the learning model, i.e., the spoken language understanding machine learning model. The learning model can be stored on memory device 604, for example, for execution by one or more hardware processors 602. One or more hardware processors 602 can be coupled to an interface device such as network interface 608 to communicate with a remote system, for example, via a network, and can be coupled to input / output interface 610 to communicate with input and / or output devices such as a keyboard, mouse, monitor, and / or other devices.

[0113] Figure 7 A schematic diagram of an example computer or processing system that can implement one embodiment of the system is shown. The computer system is merely one example of a suitable processing system and is not intended to impose any limitation on the scope of use or functionality of the embodiments of the methods described herein. The processing system shown can operate with many other general-purpose or special-purpose computing system environments or configurations. It may be suitable for use with... Figure 7Examples of well-known computing systems, environments, and / or configurations used in conjunction with the processing systems shown may include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or notebook devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments, including any of the systems or devices described above.

[0114] A computer system can be described in the general context of executable instructions (such as program modules) that run on the computer system. Generally, a program module can include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. The computer system can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside on local and remote computer system storage media, including in-memory storage devices.

[0115] The components of the computer system may include, but are not limited to, one or more processors or processing units 12, system memory 16, and a bus 14 coupling various system components (including system memory 16) to the processor 12. The processor 12 may include a module 30 that performs the methods described herein. The module 30 may be programmable into an integrated circuit of the processor 12, or loaded from memory 16, storage device 18, or network 24, or a combination thereof.

[0116] Bus 14 may represent one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. For example, but not limited to, such architectures include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0117] Computer systems can include a variety of computer system-readable media. Such media can be any available media accessible to the computer system, and can include volatile and non-volatile media, removable and non-removable media.

[0118] System memory 16 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache or others. The computer system may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 18 may be provided for reading and writing non-removable, non-volatile magnetic media (e.g., a "hard disk drive"). Although not shown, disk drives may be provided for reading and writing removable, non-volatile disks (e.g., "floppy disks"), and optical disk drives may be provided for reading or writing removable, non-volatile optical disks (e.g., CD-ROMs, DVD-ROMs, or other optical media). In this case, each may be connected to bus 14 via one or more data media interfaces.

[0119] The computer system can also communicate with one or more external devices 26, such as a keyboard, pointing device, and display 28; with one or more devices that enable a user to interact with the computer system; and / or with any device (e.g., a network card, modem, etc.) that enables the computer system to communicate with one or more other computing devices. Such communication can occur through the input / output (I / O) interface 20.

[0120] The computer system can also communicate with one or more networks 24, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), via network adapter 22. As shown, network adapter 22 communicates with other components of the computer system via bus 14. It should be understood that, although not shown, other hardware and / or software components may also be used in conjunction with the computer system. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archiving storage systems.

[0121] It should be understood that while this disclosure includes a detailed description of cloud computing, the implementation of the teachings given herein is not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter. Cloud computing is a service delivery model designed to enable convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0122] The features are as follows:

[0123] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0124] Extensive network access: Capabilities are available through networks and accessed via standard mechanisms that facilitate the use of heterogeneous thin client or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0125] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0126] Rapid flexibility: The ability to provide capacity quickly and flexibly, automatically scaling down and up rapidly in some situations to scale up rapidly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0127] Measuring services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0128] The service model is as follows:

[0129] Software as a Service (SaaS): This provides consumers with the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from different client devices via thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0130] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer using programming languages ​​and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environment.

[0131] Infrastructure as a Service (IaaS): The capabilities offered to consumers are processing, storage, networking, and other basic computing resources that enable consumers to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0132] The deployment model is as follows:

[0133] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0134] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0135] Public cloud: Makes cloud infrastructure available to the public or large industry groups and is owned by an organization that sells cloud services.

[0136] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0137] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure comprising a network of interconnected nodes.

[0138] See now Figure 8 This describes an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 to which local computing devices used by cloud consumers can communicate. These local computing devices include, for example, personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 8 The types of computing devices 54A-N shown are intended to be illustrative only, and computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0139] See now Figure 9 This demonstrates a cloud computing environment of 50 ( Figure 8 This provides a set of functional abstractions. It should be understood beforehand. Figure 9 The components, layers, and functions shown are intended to be illustrative only, and embodiments of the invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0140] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61; a RISC (Reduced Instruction Set Computer) based server 62; a server 63; a blade server 64; a storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0141] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0142] In one example, management layer 80 may provide the following functionalities: Resource Provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 83 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides cloud resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 85 provides pre-scheduling and procurement of cloud resources based on anticipated future needs according to the SLA.

[0143] Workload layer 90 provides examples of functionalities that can leverage a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom instruction delivery 93; data analysis and processing 94; transaction processing 95; and spoken language comprehension model processing 96.

[0144] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0145] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0146] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the corresponding computing / processing device.

[0147] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Java, Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may be personalized to execute computer-readable program instructions by utilizing state information from the computer-readable program instructions in order to perform aspects of this invention.

[0148] This document describes various aspects of the invention with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0149] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in the blocks of the flowchart and / or block diagram.

[0150] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0152] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. As used herein, the term “or” is an inclusive operator and may refer to “and / or” unless the context clearly indicates otherwise. It will also be understood that the terms “comprising,” “including,” “containing,” “having,” “comprising,” “having,” “comprising,” and “having” as used herein may specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. As used herein, the phrase “in one embodiment” does not necessarily refer to the same embodiment, although it may. The phrase “in one embodiment” as used herein does not necessarily refer to the same embodiment, although it may. The phrase “in another embodiment” as used herein does not necessarily refer to different embodiments, although it may. Furthermore, embodiments and / or components of embodiments may be freely combined with each other unless they are mutually exclusive.

[0153] All the means or steps plus functional elements (if any) in the following claims are intended to include corresponding structures, materials, actions, and equivalents for performing functions in combination with other claimed elements, such as those specifically claimed. The description of the invention is given for illustrative and descriptive purposes and is not intended to be exhaustive or limited to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments were chosen and described to best explain the principles and practical application of the invention and to enable others skilled in the art to understand various embodiments of the invention with various modifications suitable for the intended particular purpose.

Claims

1. A computer-implemented method, comprising: A received speech and a meaning representation associated with the speech are paired, wherein the meaning representation includes at least a semantic entity associated with the speech, wherein, The order in which the semantic entities are spoken is unknown; Alignment techniques are used to reorder the semantic entities into the spoken order of words associated with the semantic entities in the speech; The received speech and meaning representation pairs are expanded to include random sequential variations of the semantic entities; The speech comprehension machine learning model was pre-trained using the expanded speech and meaning representation pairs. as well as The pre-trained spoken language understanding machine learning model is trained using the spoken language and the meaning representations of the semantic entities with reordered semantic entities.

2. The method as described in claim 1, wherein, The alignment technique includes acoustic keyword recognition used in conjunction with a hybrid speech recognition model.

3. The method as described in claim 1, wherein, The alignment technique includes using time stamps derived from an attention model.

4. The method of claim 3, wherein, The speech includes noisy speech data, and the attention model is adapted to the noisy speech data.

5. The method of claim 1, further comprising: The pre-trained spoken language understanding machine learning model is fine-tuned using the alphabetically ordered semantic entities; and the training includes training the fine-tuned spoken language understanding machine learning model using reordered semantic entities.

6. The method of claim 1, wherein, The spoken language comprehension machine learning model includes neural networks.

7. The method of claim 1, further comprising inputting the given speech into a trained spoken language understanding machine learning model, wherein, The output of a trained spoken language understanding machine learning model includes a set of predictions of intentional labels and semantic entities associated with the given speech.

8. A computer system, comprising: processor; Storage devices coupled to the processor; The processor is configured to at least: A received speech and a meaning representation associated with the speech are paired, wherein the meaning representation includes at least a semantic entity associated with the speech, wherein, The order in which the semantic entities are spoken is unknown; Alignment techniques are used to reorder the semantic entities into the spoken order of words associated with the semantic entities in the speech; The received speech and meaning representation pairs are expanded to include random sequential variations of the semantic entities; Pre-training a spoken language comprehension machine learning model using expanded speech and meaning representation pairs; and The pre-trained spoken language understanding machine learning model is trained using the spoken language and the meaning representations of the semantic entities with reordered semantic entities.

9. The computer system as claimed in claim 8, wherein, The processor is also configured to fine-tune the pre-trained spoken language understanding machine learning model using the alphabetically ordered semantic entities.

10. The computer system of claim 8, wherein, The alignment technique includes acoustic keyword recognition used in conjunction with a hybrid speech recognition model.

11. The computer system of claim 8, wherein, The alignment technique includes using time stamps derived from an attention model.

12. The computer system of claim 11, wherein, The speech includes noisy speech data, and the attention model is adapted to the noisy speech data.

13. The computer system of claim 8, wherein, The spoken language comprehension machine learning model includes neural networks.

14. A computer program product comprising program instructions embodied therein, the program instructions being readable by a device to cause the device to perform the method as claimed in any one of claims 1 to 7.

Images