End-to-end speech understanding without complete transcripts

Abstract

A method and system of training a Spoken Language Understanding (SLU) model includes receiving natural language training data including (i) one or more speech recordings, and (ii) a set of semantic entities and / or intents for each corresponding speech recording. For each speech recording, one or more entity tags and corresponding values and one or more intent tags are extracted from the corresponding semantic entities and / or overall intents. A Spoken Language Understanding (SLU) model is trained based on the one or more entity tags and corresponding values and the one or more intent tags for the corresponding speech recordings without a transcript for the corresponding speech recordings.

Description

Technical Field

[0001] This disclosure relates generally to computer systems, and more specifically to computer systems configured to extract meaning from spoken audio data. Background Technology

[0002] In recent years, spoken language understanding (SLU) and automatic speech recognition (ASR) have become increasingly prominent in various applications, including but not limited to interactive spoken conversation systems and call center analytics that understand agent-customer dialogues. ASR is an interdisciplinary subfield of computer science and computational linguistics that provides techniques and methods that enable computers to recognize spoken language. It combines knowledge and research from computer science, linguistics, and computer engineering. ASR converts speech into text (e.g., words). Conversely, SLU converts speech into meaning (e.g., semantically annotated text). Training models for speech recognition typically involve corpora of speech data that include their transcripts. Summary of the Invention

[0003] According to various embodiments, computing devices, non-transitory computer-readable storage media, and methods for training spoken language understanding (SLU) models are provided. During the training phase, natural language training data is received for each corresponding speech record, including (i) one or more speech records and (ii) a set of semantic entities and / or overall intents. For each speech record, one or more entity labels and corresponding values, as well as one or more intent labels, are extracted from the corresponding semantic entities and / or overall intents. The spoken language understanding (SLU) model is trained based on the one or more entity labels and corresponding values ​​of the corresponding speech record and the one or more intent labels, without requiring the semantic entities to have transcripts of the corresponding audio speech records.

[0004] In one embodiment, semantic entities are not in spoken order. Semantic entities may be in alphabetical order.

[0005] In one embodiment, extraction and training are based on an attention-based encoder-decoder neural network model.

[0006] In one embodiment, natural language training data is based on a combination of different types of training data.

[0007] In one embodiment, preprocessing alignment is performed to align semantic entities to spoken order.

[0008] In one embodiment, the training data is based on transaction data between the user and the administrator who helps the user perform tasks.

[0009] In one embodiment, the training data includes records of transaction data, which includes entity packages.

[0010] In one embodiment, the extraction of one or more entity labels and corresponding values, as well as one or more intent labels, is performed through neural network processing. In various embodiments, connection-mechanism time classification (CTC), recurrent neural network transducers (RNN-T), and attention-based encoder-decoder neural network models may be used.

[0011] In one embodiment, training involves transfer learning, which includes initializing a spoken language (SLU) model on top of an ASR model.

[0012] In one embodiment, during the activity phase, raw spoken data including audio speech recordings but transcripts without audio speech recordings are received. A trained SLU model is applied to identify the meaning of the raw spoken data, where the meaning includes the intent and semantic entities of the raw spoken language. The SLU model is then applied to identify the overall intent and semantic entities in the spoken language.

[0013] By limiting computation to a smaller pool of relevant data, a more efficient computing platform is provided, saving computational demands on the computing devices performing the actions and the amount of training data in the repository.

[0014] These and other features will become apparent from the following detailed description of illustrative embodiments thereof, which should be read in conjunction with the accompanying drawings. Attached Figure Description

[0015] The accompanying drawings are illustrative embodiments. They do not show all embodiments. Other embodiments may be used additionally or alternatively. Details that may be obvious or unnecessary may be omitted to save space or for more effective illustration. Some embodiments may be practiced with additional components or steps and / or without all components or steps shown. When the same number appears in different drawings, it refers to the same or similar components or steps.

[0016] Figure 1 An exemplary architecture for an end-to-end language understanding system in the absence of a complete transcript is shown, consistent with the illustrative embodiments.

[0017] Figure 2 An example of training data used to determine the meaning of a utterance is shown.

[0018] Figure 3A A summary table is provided for the evaluation of entity packet slot filling F1 scores for speech input using connection-based temporal classification and attention-based models.

[0019] Figure 3B A summary table is provided for the F1 scores of the packet slot filling for voice input entities with additive street noise.

[0020] Figure 3CA table shows how the amount of data used to train the ASR model affects the final F1 score, which was used to initialize SLU training.

[0021] Figure 4 An attention map of utterances using the ASR model is shown.

[0022] Figure 5 Attention maps of utterances using SLU in spoken order are shown.

[0023] Figure 6 Attention graphs are shown for utterances using SLU in alphabetical order.

[0024] Figure 7A and Figure 7B Illustrative procedures for the end-to-end language understanding system during the training and activity phases, consistent with the illustrative embodiments, are presented respectively.

[0025] Figure 8 A functional block diagram of a computer hardware platform is provided, which can be used to implement a computing device that can host a semantic entity annotation engine with specific configuration.

[0026] Figure 9 A cloud computing environment consistent with the illustrative embodiments is described.

[0027] Figure 10 An abstract model layer consistent with the illustrative embodiments is depicted. Detailed Implementation

[0028] Overview

[0029] In the following detailed description, numerous specific details are illustrated by way of examples to provide a thorough understanding of the relevant teachings. However, it should be apparent, however, that practice can be carried out without such details. In other cases, well-known methods, procedures, components, and / or circuits have been described at a relatively high level without detail to avoid unnecessarily obscuring aspects of this teaching.

[0030] This disclosure generally relates to systems and computational methods for automatically training models to understand spoken language in a computationally efficient manner. A significant aspect of spoken language understanding (SLU) involves the concept of slot filling, where the meaning of spoken utterances is represented using semantic entity labels. The teachings of this paper provide an end-to-end (E2E) spoken language understanding system that directly converts speech input into semantic entities. In one aspect, the E2E SLU model discussed herein can be trained on semantic entity annotations without requiring a word-for-word transcript, compared to conventional methods. Training such a model is highly useful because it significantly reduces the computational complexity of data collection. In various embodiments, two types of such speech-to-entity models are provided by tuning the model initially used to train speech recognition: connection mechanism temporal classification (CTC) models and attention-based encoder-decoder models. Given that the experiments discussed herein involve speech input, these systems correctly recognize both semantic entity labels and words representing semantic entity values. The applicant has determined that there is a negligible degradation when training only on entities compared to training on a complete transcript. In one embodiment, entities are reordered, resulting in only about 2% degradation in the speech-to-entity packet F1 score.

[0031] This paper teaches that speech input is provided within an end-to-end (E2E) spoken language understanding (SLU) framework, taking speech as input and returning entity labels (sometimes referred to as semantic entity annotations in this paper) and values. In one respect, the trained SLU model of current systems is useful for understanding the meaning of what is said. Compared to Automatic Speech Recognition (ASR), which requires word-for-word accuracy, current SLU is not very sensitive to each word or even how it is spoken (e.g., entity order, word choice, etc.), as long as the meaning of the utterance is preserved. Therefore, current SLU systems may not require training data in the form of word-for-word transcripts, which is time-consuming and computationally expensive for acquiring and processing new domains.

[0032] Traditionally, SLU systems are cascades of automatic speech recognition (ASR) systems that convert speech to text, followed by natural language understanding (NLU) systems that interpret the meaning of the text. In contrast, in one embodiment, an E2E SLU system processes speech input directly into meaning without going through an intermediate text transcript. In one embodiment, the teachings herein can be trained on an E2E SLU system using a set (or package) of entities that do not match the spoken order of the utterance. This degree of freedom from the specific order of entities allows the system to be trained on speech data from, for example, conversations between a customer and an administrator paired with transaction data. In various embodiments, transaction data can be automatically provided by a computing device or generated by a human agent. For example, consider a chatbot or human agent assisting a client with a task such as booking a flight, generating a transaction record that includes a set of entities. This record can serve as weak supervision for training a model that understands the meaning of the utterance. When performed by a human, an accurate word-for-word transcript of the speech data typically involves 5-10 real-time transactions, not to mention the additional cost of labeling the entities. In contrast, a transaction record including a package of entities is acquired during the process of assisting a customer, whether by a human or a chatbot, without any additional costs. By using speech recordings and entity packets during training without the need for transcripts, there are technological improvements: the computational complexity and cost of data collection are significantly reduced, while the amount of training data is improved, thereby enhancing the accuracy of computing devices performing spoken language analysis. The techniques described herein can be implemented in various ways. Exemplary implementations are provided below with reference to the accompanying figures.

[0033] Example Architecture

[0034] Figure 1 An example architecture 100 for an end-to-end speech comprehension system in the absence of a complete transcript, consistent with the illustrative embodiments, is shown. Architecture 100 may include one or more session entities 101(1) to 101(N) that can participate in a conversation with each other via network 106 by means of sending and receiving electronic data packets 105(1) to 105(N). Session entities are typically individuals (e.g., human users), but may also include chatbots configured to communicate with human users. Electronic data packets 105(1) to 105(N) are sometimes referred to herein as spoken data or simple voice input. Spoken data includes raw voice data (e.g., voice recordings). In some embodiments, spoken data also includes a set of entities, discussed in more detail later.

[0035] Architecture 100 may also include a historical data repository 112 for providing training data 113 that can be used to train a speech recognition model. Architecture 100 includes a spoken language understanding (SLU) server 116 that hosts SLU 103. A network exists that allows various user devices 102(1) to 101(N) to communicate with each other and allows spoken language data to be acquired and stored at the historical data repository 112. Network 106 may be, but is not limited to, a local area network (“LAN”), a virtual private network (“VPN”), a cellular network, the Internet, or a combination thereof. For example, network 106 may include a mobile network communicatively coupled to a private network (sometimes referred to as an intranet) that provides various ancillary services, such as communication with various databases, potential stakeholders, the Internet, and the cloud 120.

[0036] For the purposes of discussion, different user / computing devices are shown in the accompanying figures to represent some examples of client devices that can be used by a user (e.g., 101(1) to 102(N)) to communicate over network 106. Today, user devices typically take the form of portable handheld devices, smartphones, tablets, personal digital assistants (PDAs), and smartwatches, although user devices can be implemented in other form factors, including consumer, medical, and commercial electronic devices.

[0037] Historical data repository 112 is configured to store and maintain a large collection of training data 113, which includes data related to previous conversations between various users from which the SLU engine 103 can learn. For example, historical data repository 112 can provide training data related to conversations that have been successfully segmented and whose meanings have been identified and labeled. In one embodiment, training data 113 serves as a corpus of data from which SLU 103 can learn to create and / or train a spoken language understanding model, which can then be used to evaluate the meaning of conversations between one or more entities 102(1) to 102(N) without their transcripts.

[0038] In one embodiment, during the training phase of the spoken language understanding system, spoken language data 105(1) to 105(N) is acquired via network 106 from one or more user devices 102(1) to 102(N). In various embodiments, the acquisition may be performed by the SLU engine 103 running on SLU server 116 or directly by the historical data repository 112. In one embodiment, the acquisition may be performed by a distributed computing architecture on cloud 120. Spoken language data (e.g., 105(1) to 105(N)) includes (i) audio speech recordings and (ii) a set of semantic entities and / or intents.

[0039] In one embodiment, the semantic entities and / or intents of the spoken data do not include transcripts of the audio speech recordings. In other embodiments, transcripts may be filtered out (e.g., by SLU engine 103) before being stored in historical data repository 112 or later when data is received from historical data repository 112. In other words, entity labels / values ​​and intent labels are extracted from semantic entities without requiring a verbatim transcript of the speech recordings. For example, entity labels / values ​​and intent labels are extracted from written transcripts (if provided), and words other than entity labels / values ​​and intent labels are removed.

[0040] At predetermined intervals or upon triggering an event, the SLU engine 103 receives training data 113, which it can use to train an SLU model without requiring a verbatim transcript of the speech recording, thereby significantly reducing the amount of historical data and the computational overhead of the SLU server 116. The trained SLU model can then be used by the SLU engine 103 during the active phase to facilitate the understanding of the meaning of the speech recording. Each of these features will be discussed in more detail below.

[0041] It will be understood that the sheer volume of training data 113 and electronic data packets 105(1) to 105(N) received by SLU 103 can pose technical challenges to the computational resources of SLU server 116 hosting SLU 103, including processing time and memory resources. In response, in one embodiment, SLU 103 is configured to filter utterances that fall outside the scope of entity and intent labels. For example, utterances such as irrelevant phrases (e.g., incoherent or irrelevant phrases such as “uhm,” “ahh,” “let me think,” “like,” etc.) (referred to herein as utterances not defined by the interaction) and other terms not directly related to entity or intent labels are removed by SLU engine 103. In this way, the technical effect of saving valuable computational and storage resources of SLU server 116 and / or historical database 112 is achieved. By limiting computation to a smaller pool of relevant data, computational requirements on SLU server 116 are reduced, resulting in a more efficient computational platform.

[0042] While the historical data repository 112, SLU server 116, and computing devices 102(1) to 102(N) are illustrated by way of example on different platforms, it will be understood that these platforms may be combined in different embodiments. In other embodiments, one or more of these computing platforms may be implemented by virtual computing devices in the form of virtual machines or software containers hosted in the cloud 120, thereby providing a resilient architecture for processing and storage. The cloud will be discussed in more detail later.

[0043] Example semantic entities

[0044] Now for reference Figure 2 , Figure 2 An example of training data 200 for determining the meaning of a utterance (i.e., intent and semantic entity labels and values) is shown. For speech recognition, training data is typically pairs of utterances (e.g., speech recordings) and word-for-word transcripts, as shown in... Figure 2 Transcript 210 in the example is shown. To train an SLU model capable of performing intent classification and semantic slot filling, such sentences are typically further labeled with intent and semantic entities, as shown in example 220. In other words, known methods typically use transcripts plus intent and semantic entity labels 220.

[0045] In contrast, the SLU engine 103 is configured to train speech paired only with semantic entities. As used in this paper, semantic entities consist of paired label-value pairs. For example, “toloc.city_name” is the label, and “Boston” is the value of the semantic entity. Example 230 shows the semantic entities used for training presented in natural spoken order. More specifically, Example 230 differs from Example 220 in that words that are not part of the semantic entities are excluded. Semantic entities can be considered as more salient keywords. It should be noted that this does not mean that the other words (excluded) do not carry any meaning. For example, “from” and “to” are used to determine whether a city is a destination city or a departure city. In our trained SLU model, such words will not be output, but the speech signals corresponding to those words will help the model output the correct semantic entity labels. Similarly, Example 230 differs from Example 220 in that certain salient words that might be related to determining the intent (e.g., “want a flight”) are not present. The SLU engine is able to be trained using only the intent label (“O-INT-flight”).

[0046] In one embodiment, semantic entities do not need to be output in spoken order, but can be in any or any predetermined order. In Example 240, the semantic entities are not given in spoken order, but are ordered alphabetically according to the entity label names. This simulates the concept of a semantic frame or entity package, where the order of entities does not affect their meaning: {{fromloc.city name: RENO}, {stoploc.city name: LAS VEGAS}, {toloc.city name: DALLAS}}.

[0047] Example of adapting ASR models to SLU systems

[0048] In various embodiments, because there are different ways to transcribe SLU data, this paper presents different methods for training the SLU system. Starting with a pre-trained ASR model, the architecture is discussed below to explain how two different types of E2E models perform when used to model various SLU data. Each possible training process may employ one or more of the following modules.

[0049] In one embodiment, an ASR model is adapted to domain data (ASR-SLU adaptation) module. Given that off-the-shelf ASR models may be trained on data that is acoustically different from SLU data, the initial step is to adapt the ASR system. For example, the parameters of the ASR model are adapted to perform better on domain data in ASR tasks (i.e., speech transcription to words). The model is adapted to new acoustic conditions, words, and language constructs present in the SLU domain data using only the action of word-for-word transcription. In model adaptation, both the original general ASR data (GP-ASR) and the domain data can be used to provide better coverage of ASR output units than adapting only the domain data. For example, this coverage may relate to the vocabulary or set of units modeled by ASR (e.g., telephone numbers, characters, words, etc.), some of which may not appear in the domain data. Coverage is improved by using a large amount of general data.

[0050] In one embodiment, a joint ASR and SLU model is used for training (e.g., joint ASR+SLU). Entity labels are introduced into the training pipeline along with the complete transcript. This module takes the form of curriculum learning, which progressively modifies an off-the-shelf ASR model into a mature SLU model. Unlike traditional methods, this model is now trained to output non-acoustic entity labels in addition to the usual glyph or speech output labels. For GP-ASR data, the objective is only glyph / speech labels, while for SLU domain data, the objective also includes entity labels. While this module is a natural progression in training the final SLU model, it can be skipped if sufficient SLU resources are available.

[0051] In one embodiment, the SLU model is fine-tuned (e.g., fine-tuning the SLU). In this final module, the model from the first or second module discussed above is fine-tuned only on the SLU data to create the final SLU model. As previously mentioned, the semantic entities to be identified by the final SLU model can take different forms: entities in spoken order only within the complete transcript, or entities in alphabetical order only.

[0052] Example training of end-to-end SLU model

[0053] Using the training process described above, two variants of an end-to-end SLU system are provided in various embodiments to attempt to directly identify entities in speech without intermediate text generation and text-based entity detection. As an example, a database that can be from a publicly available corpus from a consortium can be used, while it will be understood that other data can also be used.

[0054] In one embodiment, the SLU data and evaluation metric method are used in conjunction with standard training and testing assemblies. As an illustrative example and not a limitation, 4978 training utterances from Class A (i.e., context-independent) training data from one dataset and 893 test utterances from another dataset are used.

[0055] In this example, the 4976 training utterances comprise 9.64 hours of audio from 355 speakers. The 893 test utterances comprise 1.43 hours of audio from 55 speakers. To better train the proposed E2E model, additional corpus copies can be created using velocity / rhythm perturbations. The final training corpus after data augmentation consists of 140 hours of audio data. Slot filling performance is measured using an F1 score. When using speech input instead of text, word errors may occur. The F1 score specifies that both the slot label and value are correct. For example, if the reference is toloc.cityname:NewYork, and the decoded output is toloc.cityname:York, the SLU engine counts both false negations and false affirmations. In one embodiment, simply producing correct slot labels is insufficient: no "partial credit" is given for parts of the semantic entity value (York) being identified. This scoring can ignore the order of entities and is therefore applicable to "entity bundle" cases.

[0056] In one embodiment, a connection mechanism-based temporal classification (CTC) SLU model is used. To allow the SLU model to process entities and corresponding values ​​independently of an external language model, a word CTC model can be constructed on general ASR data. As an example and not a limitation, 300 hours of predetermined data, such as switchboard (SWB-300) data, can be used. As is known in the art, SWB-300 is readily available public data for speech recognition. In various embodiments, different training methods can be used to train the CTC-based SLU model.

[0057] The first experiment assumes that word-for-word transcripts and entity labels are available for the SLU data. All three training modules described above are used for this purpose. As an example, and not a limitation, the ASR-SLU adaptation step can be performed as follows: The output layer of the ASR model, estimating scores for 18,324 word targets and whitespace symbols, is replaced with a randomly initialized output layer estimating scores for 18,642 word / entity targets and whitespace symbols. The remaining six LSTM layers (640 units per direction) and the fully connected bottleneck layer with 256 units retain the same weights. The model is then trained on a combined dataset of 300 hours of Type I data (such as SWBGP-ASR data) and 140 hours of Type II data (such as clean Type II data). Note that in this step, although the output layer has units for entity labels, the training target is only words. In the joint ASR+SLU step, entity labels are introduced into the training transcripts, and the joint ASR-SLU model is trained on the SWB+SLU data, starting from the final weights from the ASR-SLU adaptation step. In the third and final SLU fine-tuning steps, the joint ASR-SLU model is fine-tuned only on 140 hours of SLU data.

[0058] Figure 3A A summary table of evaluations of entity packet slot filling F1 scores for speech input using CTC and attention-based models is provided. Figure 3A In Experiment [1A] in the table, the complete transcript model of Experiment [1A] was evaluated on clean test data. Given that the SLU model is a word CTC model, an external language model (LM) was not used during decoding; instead, a simple greedy decoding of the output was employed. The initial model achieved an F1 score of 91.7 for correctly detecting entity labels and their values.

[0059] In experiment [2A], a similar SLU model was developed using complete verbatim transcripts and entity tags, but the ASR-SLU adaptation module and the joint ASR+SLU adaptation module were skipped. The model was initialized with a pre-trained SWB ASR model, and the SLU model was trained directly. The model also achieved an F1 score of 91.7, suggesting that a course learning step may not always be necessary.

[0060] exist Figure 3AIn the next set of experiments, the importance of word-for-word transcripts was analyzed in relation to the training process. Following the combined ASR+SLU module in experiment [1A], in experiment [3A], the SLU model was trained to recognize only semantic entity labels and their values ​​in spoken order. It was observed that the model trained on spoken order semantic entities (without a complete word-for-word transcript) learned to ignore words in the signal that were not entity values, while retaining only semantic entity values ​​and their labels. This SLU model performed slightly better than the full transcript model in [1A].

[0061] This experiment was expanded in experiment [4A] by completely removing the use of transcripts during training. The SLU model, after being initialized with a pre-trained ASR model, was directly trained to recognize entity labels and their values ​​without any lesson learning steps or verbatim transcripts. Table 300A indicates a slight decrease in model performance, but it remains consistent with the baseline system.

[0062] Finally, the SLU system was trained on the more difficult task of recognizing entity labels and their values ​​in alphabetical order. Following the joint ASR+SLU module in experiment [1A], in experiment [5A], an SLU model was trained to recognize only semantic entity labels and their values, but now in alphabetical order.

[0063] In experiment [6A], a model similar to that in [5A] was trained, but without any curriculum learning step. For example, a curriculum learning step involves pre-training an ASR model on the complete transcript and using that model to initialize an SLU model. On this task, the CTC model's performance significantly degrades because it cannot effectively learn from targets that are not reordered according to spoken order. With the curriculum learning step, the results in [5A] are better, but still worse than the baseline. Evaluation of attention-based SLU models.

[0064] In one embodiment, the attention model for SLU is initialized using an ASR model developed for standard switchboard ASR tasks. This model employs an encoder-decoder architecture, where the encoder is an 8-layer Long Short-Term Memory (LSTM) network stack using batch normalization, residual connections, and linear bottleneck layers. The decoder models the sequence of byte-pair coding (BPE) units estimated on characters and includes two unidirectional LSTM layers. One is a dedicated, language-model-like component that operates only on the embedded sequence of predicted symbols, while the other jointly processes acoustic and symbolic information. In one example, the decoder applies additional, position-aware attention, and each layer has 768 unidirectional LSTM nodes. This single-head sequence-to-sequence model can be used to achieve high-level speech recognition performance by leveraging various regularization techniques, including SpecAugment, sequence-noise injection, velocity-tempo enhancement, and various random deactivation methods.

[0065] To identify entities, following the modules discussed above, the ASR model is adapted similarly to the CTC model. In one embodiment, compared to the CTC model which uses word units, the attention model uses a smaller stockpile (e.g., 600 BPEs) of units and relies on a decoder LSTM to model longer sequences—attention-based models inherently have long-span language models. After the initial ASR model is trained on a switchboard, subsequent adaptation and transfer learning steps use only data without any switchboard data. Because the attention model operates at the sub-word level, and all new words appearing in the transcript can be modeled using these sub-word units, no expansion of the output and embedding layers is required in the first ASR-SLU adaptation step. For simplicity, the joint ASR+SLU module is skipped, and the SLU engine can directly jump to the fine-tuning SLU module, where the decoder's output and embedding layers are expanded with semantic entity labels. The softmax layer and embedding weights corresponding to the semantic entity labels are randomly initialized, while all other parameters (including weights) corresponding to previously known symbols in the softmax and embedding layers are copied from the ASR model. Without words outside the lexicon, a sub-word level model might be well-suited to starting the adaptation process directly with the "entities in natural spoken order" example 230 discussed above. In this example, all adaptation steps use 5 training epochs.

[0066] In experiment [6A], Figure 3A The table shows the slot-filling F1 score for the attention-based SLU model. In the experiment [1A], the attention-based ASR model trained on the Switchboard 300h was first adapted on clean data to create a domain-specific ASR model. On the test set, the word error rate (WER) using the basic SWB-300 model was approximately 7.9%, which improved to 0.6% after adaptation. This ASR model was then used as the initial model for transfer learning to create the SLU model. The F1 score was comparable to that of the CTC model.

[0067] exist Figure 3A In the experiment [2A], the ASR adaptation step was skipped and the SWB-300ASR model was directly used to initialize the SLU model training. In this case, the F1 score did not degrade. There was no difference in SLU performance whether the model was initialized with the general SWB-300ASR model (WER = 7.9%) or the domain-adapted ASR model (WER = 0.6%).

[0068] exist Figure 3AExperiment [4A] in Table 3A considers the impact of training transcript quality or detail. In this example, using transcripts that only include entities in spoken order ([4A]), F1 scores were obtained that were substantially similar to those of the full transcripts used in Experiment [1A] in Table 3A. When the training transcripts included entities in alphabetical order (i.e., possibly different from spoken order), Experiment [6A] showed a 2% degradation in F1 scores, from 92.9 to 90.9. This result is significantly better than the result of the CTC model (73.5), reflecting the reordering capabilities of attention-based models. As previously mentioned, adding... Figure 3A The additional steps for adapting ASR models with word-for-word transcripts provided in experiments [3A] and [5A] yielded almost no difference.

[0069] Now, for reference Figures 4 to 6 These figures show the attention maps of three different attention models for the utterance "I would like to make a reservation for a flight to Denver from Philadelphia on this coming Sunday". More specifically, Figure 4 Attention maps of utterances using the ASR model are shown; Figure 5 Attention maps of utterances using SLU in spoken order are shown; and Figure 6 Attention graphs are shown for utterances using SLU in alphabetical order.

[0070] like Figure 5 As shown, attention to the discourse is largely monotonic with attention to the continuous portions of the audio signal corresponding to the BPE units of keywords in semantic entities. Gaps exist reflecting skipping of non-entity words.

[0071] like Figure 6 As shown, attention to the speech is segmented monotonic, with monotonic regions covering BPE units within keywords. Since semantic entities are presented in a different order than spoken order, this figure illustrates how the model associates the correct portions of the speech signal with semantic entities. Furthermore, at approximately 2 seconds, attention is given to the phrase "make areservation," which predicts the overall intent of the sentence, "flight."

[0072] Now for reference Figure 3B It provides a summary table 300B of entity packet slot-filled F1 scores for speech input with additive street noise (5dB SNR). More specifically, a noisy corpus is used as the SLU dataset, and the results are repeated in... Figure 3AThis set of experiments, based on CTC, was conducted against this background. The experiments introduced additional variability into the training process using real noise from both training and testing. Furthermore, it increased the acoustic mismatch between the transfer model and the target domain. Figure 3A The overall trend of the CTC model observed in Table 300A is also... Figure 3B Table 300B shows that: (a) curriculum training based on ASR transcripts is effective; and (b) entity labels are well identified in spoken order, but performance may be worse when the semantic entity order is different. In experiments such as [2B], the mismatch between SLU and ASR data affects the performance of models initialized only with the mismatched pre-trained model and without any other adaptation steps. Noise distortion typically causes a performance degradation in these systems compared to performance results under matched conditions.

[0073] like Figure 3B As shown in the example in Table 300B, for a more detailed attention-based SLU model, there is an absolute degradation of 4.3% in the F1 score when comparing a model trained on the full transcript (e.g., [1B] F1 = 92.0) with a model trained on entities alphabetically ([6B] F1 = 87.7%). While this is a performance drop, it is significantly better than the CTC result ([6B] F1 = 68.5). Different conclusions can be drawn regarding the utility of ASR adaptation compared to clean speech conditions. When the adapted ASR model, instead of the base SWB-300 model, is used to initialize SLU model training, there is an improvement of approximately 1% in the F1 score. On noisy test sets, using the base SWB-300 model results in a WER of 60%, while the adapted ASR model on noisy data provides a WER of 5%. Significantly, the difference in the F1 score of the final model is only 1% due to the use of these two very different ASR models to initialize SLU model training.

[0074] Figure 3C Table 300C shows how the amount of data used to train the ASR model that initializes the SLU training affects the final F1 score. Table 300C also shows the results of an attention-based SLU model trained on entities in spoken order for clean (i.e., noisy) speech. Figure 3CIn the examples, for the initial ASR model, using 2000h instead of 300h improved the F1 score by approximately 1%. This is likely due to the model's increased robustness to unseen data: the unfit WER on this test set was 3.1% (SWB2000h) versus 7.9% (SWB300h). In contrast, when the SLU model was trained directly from scratch (e.g., from random parameters, rather than initialized from a pre-trained ASR model), the experiments provided an F1 score of approximately 78.1. These experiments demonstrate the importance of ASR pre-training on a wide range of speech data when SLU data is limited, and are not necessarily relevant to the final SLU task.

[0075] Therefore, the teachings of this paper demonstrate that various E2E SLU models can be successfully constructed without verbatim transcripts. Based on these teachings, similar results can be obtained with other neural network-based E2E SLU models, including other sequence transduction models such as RNN-T (Recurrent Neural Network Transducer) and transducer-based models. In one aspect, these systems are trained using pre-trained acoustic models and curriculum learning. The effects of entity order and acoustic mismatch on the performance of these systems are explained using both clean and noisy versions of data. The E2E systems discussed in this paper are trained without verbatim transcripts and are able to reliably predict entities even when trained on transcripts where entities are not necessarily given in spoken order. These teachings provide useful insights for training better SLU systems in real-world settings where complete transcripts are often unavailable for training and the final SLU system will be deployed in a noisy acoustic environment. While we have presented one type of training data ( Figure 2 Details of the implementation of training SLU on training data (210, 220, 230, 240) are provided, but SLU can also be trained on combinations of different types of training data (210, 220, 230, 240).

[0076] Example process

[0077] Using example architecture 100 and Figures 3A to 3C The foregoing review of different training methods suggests that a more in-depth discussion of the example process may be helpful now. Therefore, Figure 7A and Figure 8A presents illustrative processes 700 and 760 of an end-to-port language understanding system during the training and activity phases, respectively, as described in the illustrative embodiment. Processes 700 and 760 are shown as a set of blocks in a logic flowchart, representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In a software environment, the blocks represent computer-executable instructions that perform the operations when executed by one or more processors. Generally, computer-executable instructions can include routines, programs, objects, components, data structures, etc., that perform functions or implement abstract data types. In each process, the order in which operations are described is not intended to be construed as limiting, and any number of described blocks can be combined and / or executed in parallel in any order to implement the process. For discussion purposes, references are made to... Figure 1 The architecture 100 describes the process 700 and 760.

[0078] Figure 7A Process 700 represents the training phase of the spoken language comprehension system. In block 702, spoken language data 105(1) is acquired via network 106 from one or more user devices 102(1) to 102(N). The spoken language data (e.g., 105(1) to 105(N)) includes (i) audio speech recordings and (ii) a set of semantic entities and / or overall intents for each corresponding speech recording. In one embodiment, the semantic entities do not include transcripts of the audio speech recordings. In other embodiments, transcripts are filtered out (e.g., by SLU engine 103) before being stored in the historical data repository 112 or later when data is received from the historical data repository 112. In other words, entity labels / values ​​and intent labels are extracted from the semantic entities without the verbatim transcripts of the speech recordings. For example, entity labels / values ​​and intent labels are extracted from written transcripts (if available), and terms other than entity labels / values ​​and intent labels are removed.

[0079] In box 706, spoken data is stored in a repository, which may be the storage of the SLU server and / or a historical data repository 112.

[0080] In box 708, SLU engine 103 receives historical data 113 from repository 112 as training data. Each semantic entity in the training data does not need to include a transcript of the corresponding audio speech record. In various embodiments, the training data can be received at predetermined intervals or at the triggering of events such as a threshold number of new spoken data available in data repository 112.

[0081] In box 710, the SLU model is trained based on entity labels / values ​​and intent labels from historical data. Notably, training does not require the inclusion of verbatim transcripts of the corresponding spoken language.

[0082] During the activity phase, the trained SLU model can be used to determine the meaning of spoken language, as discussed below by way of example.

[0083] In box 762, SLU engine 103 receives raw spoken data from user device (e.g., 102(1)) via network 106.

[0084] In box 764, the trained SLU model is used to identify one or more semantic entities and / or intentions from the raw spoken data without a transcript of the audio speech recording. In one embodiment, the set of semantic entities and / or intentions from the audio speech recording and the raw spoken data is stored in a historical data repository without their transcripts. In this way, the training set can be continuously improved while avoiding storing parameters of the dialogue that are not sufficiently significant for training the SLU model.

[0085] Example computer platform

[0086] As discussed above, such as Figure 1 As shown, and according to Figure 7A and 7B Processes 700 and 760, related to training the SLU model and determining the meaning of utterances, can be performed using one or more computing devices for data communication via wireless or wired communication connections. Figure 8 A functional block diagram description is provided for a computer hardware platform 800 that can be used to implement a computing device capable of hosting the SLU engine 840. Specifically, Figure 8 The diagram shows servers that can be used to achieve proper configuration (such as...) Figure 1 The network or hosted computer platform 800 of the SLU server 116).

[0087] The computer platform 800 may include a central processing unit (CPU) 804, a hard disk drive (HDD) 806, random access memory (RAM) and / or read-only memory (ROM) 808, a keyboard 810, a mouse 812, a display 814, and a communication interface 816 connected to a system bus 802.

[0088] In one embodiment, HDD 806 has the capability to store programs (such as SLU engine 840) capable of performing various processes in the manner described herein. SLU engine 840 may have various modules configured to perform different functions, such as... Figure 1And those modules discussed in the context of other contexts. For example, there may be a conversation module 842 for acquiring conversations between users and administrators and / or chatbots. There may be a text processing module 844 for converting speech content into text. There may be a filtering module for extracting entity tags and intent tags from written transcripts (if available) and removing all other terms. There may be an intent adaptation module 848 for converting semantic entities in random or alphabetical order into spoken order for further processing. There may be an ASR-SLU adaptation module 850 and / or an ASR+SLU adaptation module 852 to perform the functions described herein. There may be a fine-tuning module 854 for collaborating with the ASR-SLU adaptation module 850 and / or the ASR+SLU adaptation module 852 to fine-tune the model to create a final SLU model. There may be an intent module 856 for determining the intent of utterances during the activity phase without requiring a transcript of the utterances.

[0089] Although modules 842 to 856 are in Figure 8 While shown as part of HDD 806, in some embodiments, one or more of these modules may be implemented in the hardware of computing device 800. For example, the modules discussed herein may be implemented as a combination of hardware and software. Figure 8 One or more components of the SLU engine 840 shown can be implemented as electronic circuits having transistors, diodes, capacitors, resistors, inductors, varactor diodes, and / or memristors. In other words, the SLU engine 840 can be implemented using one or more specially designed electronic circuits that perform the specific tasks and functions described herein.

[0090] In one embodiment, the HDD806 may store programs for operating the system as a web server. In another embodiment, the HDD806 may store an application comprising one or more library software modules, which are executed with or without the assistance of a virtual machine, using a scripting language or otherwise.

[0091] Example cloud platform

[0092] As discussed above, functionalities related to managing compliance for one or more client domains may include Cloud 200 (see Cloud 200). Figure 1 It should be understood that while this disclosure includes a detailed description of cloud computing, the implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of this disclosure can be implemented in conjunction with any other type of computing environment now known or developed later.

[0093] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0094] The features are as follows:

[0095] On-demand self-service: Cloud consumers can automatically and unilaterally provide computing functions, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0096] Extensive network access: Functionality is available via the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0097] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned based on demand. There is a sense of location independence because consumers generally do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0098] Rapid and flexible: Features can be delivered quickly and flexibly, automatically scaling down and up in some cases to scale up rapidly. To the consumer, the available features often appear limitless and can be purchased in any quantity at any time.

[0099] Measuring services: Cloud systems automatically control and optimize resource usage by leveraging metering at a certain level of abstraction for appropriate service types (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0100] The service model is as follows:

[0101] Software as a Service (SaaS): The functionality provided to consumers is the use of the provider's applications running on cloud infrastructure. Applications can be accessed from various client devices through a thin client interface such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application functionality; possible exceptions include limited user-specific application configuration settings.

[0102] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer onto cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environments.

[0103] Infrastructure as a Service (IaaS): The functionality provided to consumers includes processing, storage, networking, and other basic computing resources that enable consumers to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and possibly limited control over selected network components (e.g., host firewalls).

[0104] The deployment model is as follows:

[0105] Private cloud: Cloud infrastructure operated solely by an organization. It can be managed by the organization or a third party, and can exist internally or externally.

[0106] Community cloud: Cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist internally or externally.

[0107] Public cloud: Enables cloud infrastructure to be used by the public or large industrial groups and is owned by organizations that sell cloud services.

[0108] Hybrid cloud: Cloud infrastructure is a combination of two or more clouds (private, community, or public) that maintain distinct entities but are bound together by standardization or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0109] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure of a network of interconnected nodes.

[0110] Now for reference Figure 9The diagram illustrates an illustrative cloud computing environment 900. As shown, the cloud computing environment 900 includes one or more cloud computing nodes 910 that can communicate with local computing devices used by cloud consumers, such as personal digital assistants (PDAs) or cellular phones 954A, desktop computers 954B, laptops 954C, and / or automotive computer systems 954N. The nodes 910 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 950 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 9 The types of computing devices 954A-N shown are intended to be illustrative only, and computing node 910 and cloud computing environment 950 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0111] Now for reference Figure 10 This demonstrates the 950 (cloud computing environment) Figure 9 This provides a set of functional abstractions. It should be understood beforehand that... Figure 10 The components, layers, and functions shown are intended to be illustrative only, and embodiments of this disclosure are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0112] The hardware and software layer 1060 includes hardware and software components. Examples of hardware components include: a mainframe 1061; a server 1062 based on a RISC (Reduced Instruction Set Computer) architecture; a server 1063; a blade server 1064; a storage device 1065; and a network and network components 1066. In some embodiments, the software components include network application server software 1067 and database software 1068.

[0113] The virtualization layer 1070 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 1071; virtual storage 1072; virtual network 1073, including virtual private network; virtual application and operating system 1074; and virtual client 1075.

[0114] In one example, management layer 1080 may provide the following functionalities: Resource Provisioning 1081 Provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 1082 Provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security Provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 1083 Provides access to the cloud computing environment for consumers and system administrators. Service Level Management 1084 Provides cloud computing resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 1085 Provides pre-scheduling and procurement of cloud computing resources based on the anticipated future requirements for cloud computing resources according to the SLA.

[0115] The workload layer 1090 provides examples of functionalities that can leverage a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 1091; software development and lifecycle management 1092; virtual classroom education delivery 1093; data analytics and processing 1094; transaction processing 1095; and an SLU engine 1096, as discussed herein.

[0116] in conclusion

[0117] Various embodiments of this teaching have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0118] While the foregoing has described what is considered the best state and / or other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein can be implemented in different forms and examples, and the teachings can be applied to many applications, of which only a few have been described herein. The appended claims are intended to claim any and all applications, modifications, and variations that fall within the true scope of this teaching.

[0119] The components, steps, features, objects, benefits, and advantages discussed herein are illustrative only. They, and the discussions relating thereto, are not intended to limit the scope of protection. While various advantages have been discussed herein, it will be understood that not all embodiments are necessarily intended to include all advantages. Unless otherwise stated, all measurements, values, scores, positions, amplitudes, sizes, and other specifications set forth in this specification (including in the appended claims) are approximate and imprecise. They are intended to have a reasonable range of functionality associated with them and consistent with functionality customary in the art to which they pertain.

[0120] Many other embodiments are also conceived. These embodiments include those with fewer, additional, and / or different components, steps, features, objects, benefits, and advantages. These embodiments also include those in which components and / or steps are arranged and / or ordered differently.

[0121] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0122] These computer-readable program instructions may be provided to a processor of a suitably configured computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that directs a computer, programmable data processing apparatus, and / or other device to function in a certain manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0123] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0124] The calling flows, flowcharts, and block diagrams in this document illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a non-diagrammatical order. For example, depending on the functions involved, two consecutively shown blocks may actually execute substantially simultaneously, or these blocks may sometimes execute in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0125] While the foregoing has been described in conjunction with exemplary embodiments, it should be understood that the term "exemplary" means only as an example, and not the best or optimal. Nothing described or shown beyond what has been directly stated above is intended or should not be construed as resulting in the public offering of any component, step, feature, object, benefit, advantage, or equivalent, whether or not it is stated in the claims.

[0126] It should be understood that the terms and expressions used herein have the general meanings assigned to them by the corresponding query and field of study, unless otherwise specified herein. Relational terms such as "first" and "second" may be used merely to distinguish one entity or action from another, without necessarily requiring or implying any actual relationship or order between these entities or actions. The terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, meaning that a process, method, article, or apparatus that includes a list of elements includes not only those elements but may also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element beginning with "a" or "an" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes that element.

[0127] An abstract of this disclosure is provided to allow the reader to quickly determine the nature of this technical disclosure. It is submitted under the understanding that it is not intended to interpret or limit the scope or meaning of the claims. Furthermore, as can be seen from the detailed description above, various features have been combined in various embodiments for the purpose of simplification. The methods of this disclosure should not be construed as reflecting an intention to have more features than expressly recited in each claim of the claimed embodiments. Rather, as reflected in the following claims, the inventive subject matter lies in fewer than all features of a single disclosed embodiment. Therefore, the following claims are thus incorporated into the detailed description, wherein each claim exists independently as a separate claim subject matter.

Claims

1. A computing device, comprising: processor; A network interface coupled to the processor to enable communication over a network; An engine, configured to perform actions, including, during the training phase: Natural language training data is received through the network, the natural language training data including (i) one or more speech records, and (ii) a set of semantic entities and / or overall intent corresponding to each speech record; For each voice recording, extract (i) one or more entity labels and corresponding values, and (ii) one or more intent labels from the corresponding semantic entities and / or overall intent; and The Spoken Language Understanding (SLU) model is trained based on the entity labels and corresponding values ​​of the one or more speech recordings and one or more intent labels, without the transcripts of the speech recordings.

2. The computing device of claim 1, wherein the semantic entities are not in spoken order.

3. The computing device of claim 2, wherein the semantic entities are in alphabetical order.

4. The computing device of claim 2, wherein the natural language training data is based on a combination of different types of training data.

5. The computing device of claim 2, wherein the engine is further configured to perform actions, the actions including performing preprocessing alignment to align the semantic entities to spoken order.

6. The computing device of claim 1, wherein the training data is based on transaction data between a user and an administrator who assists the user in performing tasks.

7. The computing device according to claim 1, wherein the training data includes records of transaction data, and the transaction data includes entity packets.

8. The computing device of claim 1, wherein the extraction of the one or more entity labels and corresponding values, and the one or more intent labels, is performed by neural network processing, the neural network comprising at least one of connection mechanism temporal classification (CTC), recurrent neural network transducer (RNN-T), or attention-based encoder-decoder neural network.

9. The computing device of claim 1, wherein the training involves transfer learning, the transfer learning comprising initializing the SLU model with an automatic speech recognition (ASR) model.

10. The computing device of claim 1, wherein the engine is further configured to perform actions, the actions including, during an activity phase: Receive raw spoken data including transcripts of audio speech recordings but without said audio speech recordings; and A trained SLU model is used to identify the meaning of the raw spoken data, wherein the meaning includes the intent and semantic entities of the raw spoken language.

11. A non-transitory computer-readable storage medium tangibly embodying computer-readable program code, said computer-readable program code having computer-readable instructions, which, when executed, cause a computer device to perform a method for training a spoken language comprehension SLU model, said method comprising: Receive natural language training data, which includes (i) one or more speech records and (ii) a set of semantic entities and / or overall intent corresponding to each speech record; For each voice recording, extract (i) one or more entity labels and corresponding values, and (ii) one or more intent labels from the corresponding semantic entities and / or overall intent; and The Spoken Language Understanding (SLU) model is trained based on the entity labels and corresponding values ​​of the one or more speech recordings and the one or more intent labels, without the need for transcripts of the speech recordings.

12. The non-transitory computer-readable storage medium of claim 11, wherein the semantic entities are not in spoken order.

13. The non-transitory computer-readable storage medium of claim 12, wherein the natural language training data is based on a combination of different types of training data.

14. The non-transitory computer-readable storage medium of claim 13, the method further comprising performing preprocessing alignment to align the semantic entities to spoken order.

15. The non-transitory computer-readable storage medium of claim 11, wherein the training data is based on transaction data between a user and an administrator who assists the user in performing a task.

16. The non-transitory computer-readable storage medium of claim 11, wherein the training data includes records of transaction data, and the transaction data includes entity packets.

17. The non-transitory computer-readable storage medium of claim 11, wherein the extraction of the one or more entity labels and corresponding values, and the one or more intent labels, is performed by neural network processing, the neural network comprising at least one of connection mechanism temporal classification (CTC), recurrent neural network transducer (RNN-T), or attention-based encoder-decoder neural network.

18. The non-transitory computer-readable storage medium of claim 11, wherein the training involves transfer learning, the transfer learning comprising initializing the SLU model with an automatic speech recognition (ASR) model.

19. The non-transitory computer-readable storage medium of claim 11, further comprising, during the active phase: Receive raw spoken data including transcripts of audio speech recordings but without said audio speech recordings; and A trained SLU model is used to identify the meaning of the raw spoken data, wherein the meaning includes the intent and semantic entities of the raw spoken language.

20. A computer-implemented method, comprising: Receive natural language training data, which includes (i) one or more speech records and (ii) a set of semantic entities and / or overall intent corresponding to each speech record; For each voice recording, extract (i) one or more entity labels and corresponding values, and (ii) one or more intent labels from the corresponding semantic entities and / or overall intent; and The Spoken Language Understanding (SLU) model is trained based on the one or more entity labels and corresponding values ​​of the one or more speech recordings, as well as one or more intent labels, without the need for transcripts of the speech recordings.

Images