generating event output

Abstract

A system for determining a subscription to data when a user requests to receive an output in the event of an occurrence in the future is provided. The system can determine an output type based on the capabilities of the output device and the trigger type. The system can determine the trigger type based on the priority of the triggering event. The system can also determine how many times the subscription is to be executed. Using this information, the system creates the subscription so that the user can receive a notification or announcement when the event occurs.

Description

[0001] Cross-referencing of relevant application data

[0002] This application claims priority to U.S. Patent Application No. 16 / 775,247, entitled “GENERATING EVENT OUTPUT”, filed January 28, 2020, in the name of Vinaya Nadig et al. Background Technology

[0003] Speech recognition systems have evolved to the point where humans can use their voices to interact with computing devices. These systems employ technology to recognize words spoken by human users based on received audio input of varying quality. Speech recognition processing, combined with natural language understanding processing, enables voice-based user control of computing devices to perform tasks based on the user's verbal commands. This combination of speech recognition processing and natural language understanding processing techniques is referred to herein as speech processing. Speech processing may also involve converting a user's speech into text data, which can then be provided to various text-based software applications.

[0004] Voice processing can be used by computers, handheld devices, telephone computer systems, self-service terminals, and a wide variety of other devices to improve human-computer interaction. Attached Figure Description

[0005] To gain a more complete understanding of this disclosure, reference is now made to the following description in conjunction with the accompanying drawings.

[0006] Figure 1 A system configured to create subscriptions based on user input is shown according to an embodiment of this disclosure.

[0007] Figure 2 This is a conceptual diagram of the components of a system according to an embodiment of this disclosure.

[0008] Figure 3 This is a conceptual diagram illustrating how natural language understanding processing is performed according to an embodiment of this disclosure.

[0009] Figure 4 This is a conceptual diagram of a system component for configuring subscriptions according to an embodiment of this disclosure.

[0010] Figure 5 This is a conceptual diagram of a system component for performing a subscription according to an embodiment of this disclosure.

[0011] Figure 6 This is a conceptual diagram illustrating how subscription data representing a created subscription can be stored according to an embodiment of this disclosure.

[0012] Figure 7Ais a conceptual diagram of a text-to-speech component according to embodiments of the present disclosure.

[0013] Figure 7B is a conceptual diagram of a speech model according to embodiments of the present disclosure.

[0014] Figure 8 is a block diagram conceptually illustrating example components of a device, according to embodiments of the present disclosure.

[0015] Figure 9 is a block diagram conceptually illustrating example components of a system, according to embodiments of the present disclosure.

[0016] Figure 10 An example of a computer network for use with a speech processing system is shown. DETAILED DESCRIPTION

[0017] Automatic Speech Recognition (ASR) is a field of computer science, artificial intelligence, and linguistics that involves converting audio data associated with speech into text data representing that speech. Similarly, Natural Language Understanding (NLU) is a field of computer science, artificial intelligence, and linguistics that involves enabling computers to derive meaning from text data containing natural language. ASR and NLU are often used together as part of a speech processing system. Text-to-Speech (TTS) is a field that involves converting text data into audio data that is synthesized to resemble human speech.

[0018] When a particular event occurs in the future, the system can receive user input requesting that the system perform a particular action. Thus, a user can subscribe to a service whereby the system can detect the event and can generate some output to indicate to the user that the event occurred. The form of the output can be specified by the user, but in some cases the system can proactively determine the form of the output based on the type of event, the priority of the event, the user's current activity, and / or other factors such as device capabilities. For example, even if the user is engaged in another activity, the system can determine that a certain event calls for interrupting the user (such as being on a device watching content, such that the system interrupts the content playback to output information about the event), while a different event can call for a more passive notification, such as a message to a companion application on the user's mobile device that the user can notice later, rather than causing an interruption to the user's content viewing. Thus, the system can create a proactive experience in response to such user requests that cause the system to generate and store subscription data, in order to later generate a proactive output (such as a notification, reminder, or other output, as explained below) when the event occurs. For example, a user input can be "Alexa, tell me when I receive an email from Joe," and the system can create and store subscription data that causes the system to generate and send an output (e.g., a notification) to the user when the event of receiving an email from Joe occurs. Another user input can be "tell me when the Seahawks play," and the system can create and store a subscription that causes the system to generate and send an output to the user when the event (the Seahawks playing) occurs. The stored subscription can also indicate the output type, which devices receive the output, the recurrence type of the output, and other data for presenting the output to the user. Alternatively or additionally, the system can use default settings or machine learning methods to determine such information to make dynamic determinations about such items.

[0019] The improved system of the present disclosure determines the output type of a subscription based on conditions such as the capabilities of the output device, determines the trigger type of a subscription based on whether the event time is determinable or unknown, and / or determines other information such as the user's activity at the time of the output. The system also determines recurrence information for a subscription to determine how many times the subscription is to be performed. (Although the term subscription is used herein to describe a user registering for a notification, such a subscription can be for a one-time single event or for a recurring event.) Using this information, the system creates the subscription and provides the user with a confirmation that the subscription has been created. The confirmation can include information about the subscription. For example, if the system determines that the output type of the subscription is an audio output, then in confirming the creation of the subscription, the system can output "I will announce when you receive an email from Joe." When the system determines that the output type is a push notification, the system can output "I will notify you when you receive an email from Joe."

[0020] When the time of the event is determinable, such as the start of a television program or sporting event, the system can determine the trigger type to be a reminder, and upon confirmation of the creation of the subscription, the system can output, “I will remind you when the program starts (or before the program starts).” When the time of the event is not determinable in advance, such as receiving an email or a prescription being ready, the system can determine the trigger type to be a notification, and upon confirmation of the creation of the subscription, the system can output, “I will notify you when the prescription is ready.”

[0021] The system can also confirm the recurrence of the subscription. For example, if the user input is, “tell me every time I receive an email from Joe,” the system can create a subscription accordingly, and can output the confirmation, “I will tell you every time you receive an email from Joe.” The system can determine whether the triggering event is to occur once or can recur. If the event occurs only once (e.g., the opening ceremony of the 2020 Summer Olympics), the system creates a subscription that executes once. If the event can recur, the system can determine whether the user’s intent is to create a one-time subscription or a recurring subscription.

[0022] Figure 1 A system configured to create a subscription for a user input according to an embodiment of the disclosure is shown. While the figures and discussion herein illustrate certain operational steps of the system in a particular order, the described steps can be performed in a different order (as well as remove or add certain steps) without departing from the intent of the disclosure. As Figure 1 As shown, the system can include a device 110 local to the user 5 that communicates with one or more systems 120 over one or more networks 199.

[0023] The system 120 receives (132) a user input to create a subscription. The user input can be audio data representing an utterance spoken by the user 5 and captured by the device 110. The system 120 can perform automatic speech recognition (ASR) processing on the audio data to determine text data corresponding to the user input. In some embodiments, the user input can be text data or another form of data. The system 120 can perform natural language understanding (NLU) on the text data / user input to determine the user’s intent to create a subscription to receive an output in the future in the event an event occurs. For example, the user input can be, “tell me when I receive an email from Joe,” “tell me when the Seahawks play,” “remind me to close the window every time it rains,” etc.

[0024] The system 120 determines 134 trigger data corresponding to the subscription. The trigger data represents a triggering event that causes the generation of output data. The system 120 determines 136 output data corresponding to the subscription, where the output data represents an action to be performed when the triggering event occurs. The system 120 can perform natural language understanding (NLU) processing on the text data representing the user input to determine an intent corresponding to the user input. The system 120 can determine that the intent of the user is to create a subscription to receive data or to cause the system to perform an action in the future in response to the occurrence of an event. The system 120 can use NLU and the text data to determine the trigger data for performing the action. For example, if the user input is “notify me when I receive an email,” the system 120 using NLU can determine that the intent is to receive a notification in the future, that the time slot for the trigger for receiving the notification is when the user receives an email, and that the action to be performed is to send a notification to the user when the user receives an email. As another example, the user input can be “turn on the light when I get home,” and the system 120 using NLU can determine that the intent is to perform a smart home action in the future, that the time slot for the trigger is when the user gets home, and that the action to be performed is to turn on the light. As another example, the user input can be “tell me when my prescription is ready,” and the system 120 using NLU can determine that the intent is to receive a notification in the future, that the trigger is when the user’s prescription is ready, and that the action to be performed is to send a notification to the user when the prescription is ready.

[0025] The system 120 determines (138) a trigger type corresponding to the trigger data. The trigger type can represent a priority indication for the subscription based on the trigger event represented by the trigger data and other data associated with the trigger data and / or the user profile. The priority data can be a priority indicator of high, medium-high, medium, medium-low, low, and other variations. In some embodiments, the trigger type can be a numerical value representing the priority data for the subscription, where the numerical value can be between 1 and 5. In some embodiments, the system 120 determines the trigger type based on a category corresponding to the trigger event. For example, the trigger event "receive an email" can correspond to the category "mail / email." In other examples, the trigger event "prescription ready" can correspond to the category "medical / health," the trigger event "football game starts" can correspond to the category "event / sport," the trigger event "when it rains" can correspond to the category "weather condition," the trigger event "hurricane warning" can correspond to the category "severe weather condition," etc. Each category can be assigned a system priority indication, e.g., the mail / email category can be assigned as low, medical / health can be assigned as high, event / sport can be assigned as low, weather condition can be assigned as medium, severe weather condition can be assigned as high, etc. Using the system priority data, the system 120 can determine the trigger type for the subscription. For example, if the user input is "tell me when I receive an email from Joe," the system 120 can determine the trigger type as low based on the corresponding email / mail category having a low priority indication. In another example, if the user input is "tell me when there is a severe weather alert," the system 120 can determine the trigger type as high based on the corresponding severe weather condition category having a high priority indication.

[0026] In some embodiments, using the user input data and / or the user profile data associated with the user 5, the system 120 can determine user-specific priority data for the category and / or the trigger event. The system 120 can adjust the system priority indication based on user preference data, user activity, or other user profile data to determine the user-specific priority indication. For example, the user profile data corresponding to the user 5 can include user preferences related to the priority indication for a particular trigger event / category, e.g., the user profile data can indicate that the "prescription ready" event has a medium priority. In another example, the user profile data can indicate that the "severe weather condition" event has a medium priority, however, based on the system priority indication being high and based on the severe weather alert affecting the location of the user, the system 120 can determine the priority data for the particular severe weather alert trigger event as high. In some embodiments, the system 120 can use user profile data corresponding to other users to determine the trigger type. For example, if the user profile data of other users indicates a high preference for the mail / email category to have a high priority, then the system 120 can determine the trigger type for the trigger event "receive an email" as high.

[0027] The system 120 can use data determined from the user input to determine the trigger type. The user input can include an indication of a priority or can include other data from which a priority can be derived, e.g., “you want to tell me when I receive an email from Joe, this is important” or “Alexa, my medication is running out. Tell me when my prescription is ready,” and in both cases the system 120 can determine the trigger type to be high. In the case of a voice input, the system 120 can determine, at the time the user input is provided, emotional data that is indicative of the user’s emotion / feeling, and can use the emotional data to determine the trigger type. For example, the user input can be “tell me when I receive an email from Joe!” the system 120 can determine the emotional data using audio data that is indicative of excitement or urgency, and the system 120 can determine the trigger type to be high.

[0028] In some embodiments, the trigger type can be based on whether the time of the triggering event is known, determinable, or preset before the event occurs. For example, the time that a sporting event or a television show begins is known or can be determined before the sporting event or television show begins. In contrast, the time that a user can receive an email cannot be determined until the email is received. For triggering events for which the time before the event occurs is determinable / known, the system 120 determines the trigger type to be a reminder operation. For triggering events for which the time before the event occurs cannot be determined, the system 120 can determine the trigger type to be a notification operation.

[0029] The system 120 determines (140) the output type of the subscription using the trigger type and the device capabilities of the output device. In some embodiments, the user input can specify the output device of the subscription. For example, the user input can be “remind me on my phone when the Seahawks game,” and the system 120 can determine that the output device is the device specified as “my phone,” “personal phone,” or otherwise associated with the user profile of the user 5, a smart phone device. In some embodiments, when the user input does not specify the output device of the subscription, the system 120 determines the device that received the user input (e.g., the device 110) to be the output device.

[0030] In some implementations, the user input can not specify the form of output for the subscription. For example, the user input "tell me when I receive an email" does not specify whether the user wants audio output or a push notification or another type of output. In such cases, the system 120 uses the trigger type to determine the output type. The output type can indicate which form of output (e.g., an announcement, a push notification, a visual notification, an audio notification, etc.) will be used to indicate to the user that the triggered event occurred. The output type can also indicate the timing of presenting the output (e.g., interrupting the user with the output as soon as the event occurs, waiting until the user interacts with the system, etc.). The output type can also indicate how many devices associated with the user profile will receive the output.

[0031] If the trigger data indicates a high priority for the triggered event, the system 120 can determine to announce at the time of the event, e.g., by generating synthesized speech. If the trigger type is medium, the system 120 can determine the output type to be a notification, including a push notification, a visual notification (e.g., displaying text and / or graphical elements on the screen of an output device, causing a light of the output device to turn on, flash, or blink, etc.), an audio notification (e.g., a chime or other sound), or other form of notification, and can cause multiple devices to output the notification. If the priority is low, the system 120 can determine the output type to be a notification, and can cause one device associated with the user profile to output the notification.

[0032] If the trigger type is high, the output type can indicate the timing of the output to be presenting the output as soon as the event occurs, and can cause one or more devices to turn on to present the output, even if the device was previously off. In some cases, the system 120 can interrupt a user's interaction with an output device to present the output. For example, if the output device is playing music and an event with a trigger type of high occurs, the system 120 can cause the output device to pause the music and output "There is a severe weather alert in your area." If the trigger type is medium, the output type can indicate the timing of the output to be presenting the output after the event occurs when the user interacts with the system, and the output can not cause any devices that were off to turn on. For example, the user can turn on or activate an output device, and the system 120 can present the output at that time. If the trigger type is low, the output type can indicate the timing of the output to be presenting the output when the user requests to receive any notifications / reminders / or other output. For example, the trigger type for an email receipt event can be low, and when the user receives the email, the system 120 can generate the output data but not send it to the output device for presentation. The user can speak or otherwise provide input indicating an intent to receive a notification, e.g., "Alexa, what are my notifications for today?", at which time the system 120 can cause the output device to present the output "You have received an email."

[0033] If the trigger type is high, the system 120 can determine the output type such that it is sent to multiple devices associated with the user profile. If the trigger type is low, the system 120 can determine the output type such that it is sent to one device associated with the user profile.

[0034] The system 120 can determine the output capabilities of the output device to determine the output type of the subscription. The system 120 can assign a higher preference to the ability to output synthesized speech, and if the output device is capable of outputting synthesized speech, set the output type to audio output that includes TTS output. If the output device is not capable of outputting synthesized speech, the system 120 can determine whether the output device is capable of receiving a push notification, and set the output type to a push notification. The output type can include audio output, TTS output, a push notification, a visual notification, or other form of output that the device can be capable of presenting.

[0035] The system 120 determines (142) frequency data for performing the subscription. The frequency data indicates the number of times the user wants to receive the output when the event occurs. For example, the user input can be “tell me every time a package is delivered,” and the system 120 determines the frequency data to be “every time.” As another example, the user input can be “tell me the next two times I receive an email,” and the system 120 determines the frequency data to be “two times.” In some cases, the user input can not specify the number of times the user wants to receive the output of the event occurring. The system 120 can determine the type of event whether the triggering event occurs once, a limited number of times, or an unlimited number of times. If the triggering event occurs once (e.g., the opening ceremony of the Summer Olympics in 2020), the system 120 determines the frequency data to be “one time.” If the triggering event occurs multiple times, the system 120 can determine the frequency data to be “unlimited,” and can generate the output each time the event occurs until the user disables or deletes the subscription. In some embodiments, the system 120 can ask the user for further information about how many times the user wants to receive the output when the event occurs, and use the user’s response to determine the frequency data.

[0036] The system 120 stores (144) the subscription data, including the trigger data, the output data, the output type, the trigger type, and the frequency data. The subscription data can include other information, such as the user profile identifier, the output device identifier, and other data. The system 120 can use the stored subscription data to determine when to trigger the output and how to present the output to the user.

[0037] The system can operate using various components as shown in FIG. 1. Figure 2 The various components can be located on the same or different physical devices. Communication between the various devices can be done directly or through the network 199.

[0038] An audio capture component, such as a microphone or microphone array of the device 110, captures audio 11. The device 110 processes audio data representing the audio 11 to determine whether speech is detected. The device 110 can use various techniques to determine whether the audio data includes speech. In some examples, the device 110 can apply voice activity detection (VAD) techniques. Such techniques can determine whether speech is present in the audio data based on various quantitative aspects of the audio data, such as a spectral slope between one or more frames of the audio data; an energy level of the audio data in one or more spectral bands; a signal-to-noise ratio of the audio data in one or more spectral bands; or other quantitative aspects. In other examples, the device 110 can implement a finite classifier configured to distinguish speech from background noise. The classifier can be implemented by techniques such as linear classifiers, support vector machines, and decision trees. In still other examples, the device 110 can apply hidden Markov model (HMM) or Gaussian mixture model (GMM) techniques to compare the audio data to one or more acoustic models in a storage device, which can include models corresponding to speech, noise (e.g., environmental noise or background noise), or silence. Still other techniques can be used to determine whether speech is present in the audio data.

[0039] Once speech is detected in the audio data representing the audio 11, the device 110 can perform wake-word detection using the wake-word detection component 220 to determine when a user intends to speak an input to the device 110. An example wake-word is "Alexa."

[0040] Wake-word detection is typically performed without performing language analysis, text analysis, or semantic analysis. Instead, the audio data representing the audio 11 is analyzed to determine whether specific characteristics of the audio data match a preconfigured acoustic waveform, audio signature, or other data, to determine whether the audio data "matches" stored audio data corresponding to a wake-word.

[0041] Accordingly, the wake-word detection component 220 can compare the audio data to stored models or data to detect a wake-word. One approach for wake-word detection applies a general large vocabulary continuous speech recognition (LVCSR) system to decode the audio signal, with a wake-word search performed in the resulting lattice or confusion network. LVCSR decoding can require relatively high computational resources. Another approach for wake-word detection builds HMMs for each wake-word and non-wake-word speech signal separately. Non-wake-word speech includes other spoken words, background noise, etc. One or more HMMs can be built to model non-wake-word speech characteristics, referred to as filler models. Viterbi decoding is used to search for the best path in the decoding graph, and the decoding output is further processed to make a decision about the presence of a wake-word. This approach can be extended to include discriminative information by incorporating a hybrid DNN-HMM decoding framework. In another example, the wake-word detection component 220 can be built directly on a deep neural network (DNN) / recurrent neural network (RNN) structure without involving HMMs. This architecture can estimate the posterior of a wake-word with contextual information by stacking frames within a context window of the DNN or using an RNN. Successive posterior thresholding or smoothing is applied to the decision. Other techniques for wake-word detection can also be used, such as those known in the art.

[0042] Once the wake-word is detected, the device 110 can "wake up" and begin transmitting audio data 211 representing the audio 11 to the system 120. The audio data 211 can include data corresponding to the wake-word, or the portion of the audio corresponding to the wake-word can be removed by the device 110 before the audio data 211 is sent to the system 120.

[0043] Upon receipt by the system 120, the audio data 211 can be sent to the coordinator component 230. The coordinator component 230 can include memory and logic that enable the coordinator component 230 to transmit various pieces of data and various forms of data to various components of the system, as well as perform other operations as described herein.

[0044] The coordinator component 230 sends the input audio data 211 to the ASR component 250, which transcribes the input audio data 211 into input text data representing one or more hypotheses representing the speech contained in the input audio data 211. Thus, the text data output by the ASR component 250 can represent one or more (e.g., in the form of an N-best list) ASR hypotheses representing the speech represented in the audio data 211. The ASR component 250 interprets the speech in the audio data 211 based on similarities between the audio data 211 and a pre-established language model. For example, the ASR component 250 can compare the audio data 211 to models of sounds (e.g., sub-word units, such as phonemes, etc.) and sequences of sounds to identify words matching the sequence of sounds represented in the audio data 211. The ASR component 250 outputs text data representing one or more ASR hypotheses. The ASR component 250 can also output respective scores for the one or more ASR hypotheses. Such text data and scores can be output, for example, following language model operations of the ASR component 250. Thus, the text data output by the ASR component 250 can include the highest scoring ASR hypothesis, or can include an N-best list of ASR hypotheses. The N-best list can additionally include respective scores associated with each ASR hypothesis represented therein. Each score can indicate a degree of confidence in the ASR processing performed to generate the ASR hypothesis associated with the score. Further details of the ASR processing are included below.

[0045] The device 110 can send the text data 213 to the system 120. Upon receipt by the system 120, the text data 213 can be sent to the coordinator component 230, which can send the text data 213 to the NLU component 260. The text data 213 can be derived from input provided by the user 5 via an application / app on the device 110, where the user 5 can use the application / app to create a subscription (as described in connection with Figure 1 For example, the text data 213 can be "notify me when I receive an email from ___" or "tell me when my ___ prescription is ready for pickup at the pharmacy."

[0046] The NLU component 260 receives one or more ASR hypotheses (i.e., textual data) and attempts to semantically interpret the phrases or sentences represented therein. That is, the NLU component 260 determines one or more meanings associated with the phrases or sentences represented in the textual data based on the words represented in the textual data. The NLU component 260 determines an intent representing an action that the user desires to perform and a segment of the textual data that allows the device (e.g., the device 110, the system 120, the skill 290, the skill system 225, etc.) to perform the intent. For example, if the textual data corresponds to "play music by Adele," the NLU component 260 can determine an intent of the system 120 to output music and can identify "Adele" as the artist. As another example, if the textual data corresponds to "what's the weather like," the NLU component 260 can determine an intent of the system 120 to output weather information associated with a geographic location of the device 110. In another example, if the textual data corresponds to "turn off the lights," the NLU component 260 can determine an intent of the system 120 to turn off a light associated with the device 110 or the user 5.

[0047] The NLU component 260 can send NLU result data (which can include tagged textual data, an indicator of an intent, etc.) to the coordinator component 230. The coordinator component 230 can send the NLU result to the skill 290. If the NLU result data includes a single NLU hypothesis, the coordinator component 230 can send the NLU result data to the skill 290 associated with the NLU hypothesis. If the NLU result data includes an N-best list of NLU hypotheses, the coordinator component 230 can send the highest scoring NLU hypothesis to the skill 290 associated with the highest scoring NLU hypothesis.

[0048] A "skill" can be software running on the system 120 that is analogous to a software application running on a traditional computing device. That is, the skill 290 can enable the system 120 to perform a particular function in order to provide data or produce some other requested output. The system 120 can be configured with more than one skill 290. For example, a weather service skill can enable the system 120 to provide weather information, a car service skill can enable the system 120 to book a ride with respect to a taxi or ride-sharing service, a restaurant skill can enable the system 120 to order a pizza with respect to an online ordering system of a restaurant, etc. The skill 290 can operate in coordination between the system 120 and other devices, such as the device 110, in order to accomplish certain functions. Input to the skill 290 can come from a speech processing interaction or through other interactions or input sources. The skill component 290 can include hardware, software, firmware, etc. that can be dedicated to a particular skill 290 or shared between different skills 290.

[0049] In addition to or instead of being implemented by the system 120, the skill 290 can be implemented by the skill system 225. This can enable the skill system 225 to perform particular functions in order to provide data or perform some other action requested by a user.

[0050] The types of skills include home automation skills (e.g., skills that enable a user to control home devices such as lights, door locks, cameras, thermostats, etc.), entertainment device skills (e.g., skills that enable a user to control entertainment devices such as smart televisions), video skills, news skills, and custom skills that are not associated with any preconfigured skill type.

[0051] The system 120 can be configured with a single skill 290 that is dedicated to interacting with more than one skill system 225.

[0052] Unless otherwise explicitly stated, a reference to a skill, skill device, or skill component can include a skill 290 operated by the system 120 and / or a skill operated by the skill system 225. In addition, many different terms (such as action, bot, app, etc.) can be used to refer to the functionality described herein as a skill.

[0053] The system 120 can include a TTS component 280 that generates audio data (e.g., synthesized speech) from text data using one or more different methods. The text data input to the TTS component 280 can come from a skill 290, the coordinator component 230, or another component of the system 120.

[0054] In one method of synthesis, called unit selection, the TTS component 280 matches text data to a database of recorded speech. The TTS component 280 selects matching units of recorded speech and concatenates the units together to form audio data. In another method of synthesis, called parametric synthesis, the TTS component 280 varies parameters such as frequency, volume, and noise to create audio data that includes artificial speech waveforms. Parametric synthesis uses a computerized voice generator, sometimes called a vocoder.

[0055] The system 120 can include a profile store 270. The profile store 270a can include various information about individual users, groups of users, devices, etc. that interact with the system 120. A "profile" refers to a set of data associated with a user, device, etc. The data of a profile can include preferences specific to the user, device, etc.; input and output capabilities of the device; internet connection information; user bibliographic information; subscription information; and other information.

[0056] The profile storage 270 can include one or more user profiles, where each user profile is associated with a different user identifier. Each user profile can include various user identifying information. Each user profile can also include preferences of the user and / or one or more device identifiers representing one or more devices registered with the user.

[0057] The profile storage 270 can include one or more group profiles. Each group profile can be associated with a different group identifier. A group profile can be specific to a group of users. That is, a group profile can be associated with two or more individual user profiles. For example, a group profile can be a family profile associated with user profiles of multiple users of a single family. A group profile can include preferences shared by all user profiles associated therewith. Each user profile associated with a group profile can additionally include preferences specific to the user associated therewith. That is, each user profile can include preferences that are unique with respect to one or more other user profiles associated with the same group profile. A user profile can be a standalone profile or can be associated with a group profile. A group profile can include one or more device profiles representing one or more devices associated with the group profile.

[0058] The profile storage 270 can include one or more device profiles. Each device profile can be associated with a different device identifier. Each device profile can include various device identifying information. Each device profile can also include one or more user identifiers representing one or more user profiles associated with the device profile. For example, a profile of a home device can include user identifiers of the home users.

[0059] The subscription storage 455 can include data related to subscriptions created for users identified by respective user profiles. For example, the subscription storage 455 can include trigger information (indicating when an active action is to be performed) and action information (indicating an action to be performed). The subscription storage 455 can also include information indicating an output type, a trigger type, and a recurrence type.

[0060] The system can be configured to incorporate user permissions, and the activities disclosed herein can only be performed with the approval of the user. As such, the systems, devices, components, and techniques described herein will generally be configured to limit processing where appropriate, and to process user information only in a manner that ensures compliance with all appropriate laws, regulations, standards, etc. The described systems and techniques can be implemented on a geographic basis to ensure compliance with laws in various jurisdictions and entities in which components of the system and / or users are located.

[0061] The ASR engine 258 can return a list of N-best paths corresponding to the top N paths determined by the ASR engine 258 and their respective recognition scores. Given the list and associated recognition scores, an application, such as a program or component, internal or external to the ASR component 250 that receives the N-best list can then perform further operations or analysis on the list. For example, the N-best list can be used to correct errors and train various options and processing conditions of the ASR module 250. The ASR engine 258 can compare the actual correct utterance with the best result and with other results on the N-best list to determine why the incorrect recognition received certain recognition scores. The ASR engine 258 can correct its methods (and can update information in the ASR model 252) to reduce the recognition scores of incorrect methods in future processing attempts.

[0062] The system 120 can also include a subscription manager 275 that can process user input to create subscriptions for users, storing corresponding trigger data and action data in the subscription storage 455, as described with respect to Figure 4 The subscription manager 275 can process event triggers from the skills 290 to determine whether to perform the actions of a subscription, and can update recurrence information for the subscription, as described with respect to Figure 5

[0063] One or more content systems 226 can be in communication with the system 120 and can provide event data 227. The event data can represent events that occur. The content systems 226 can relate to various systems that provide different types of data, including event data that indicates the occurrence of a particular event. For example, the content systems 226 can be associated with weather conditions, sports-related information, email management systems, prescription / medical data systems, package delivery systems, etc., and can provide data related thereto. For example, the event data 227 can represent weather conditions that occur, sports-related events that occur, receipt of an email, an indication that a prescription is ready, an indication that a package has been or will be delivered, etc. As described with respect to Figure 5 The subscription manager 275 can use the event data 227 to determine that an output according to a subscription is triggered, as described with respect to

[0064] ​In some implementations, the system 120 can monitor content databases, knowledge databases, information available on the Internet, and other sources of information to determine occurrences of events, determine changes in content / information, and determine availability of new content / information. When the system 120 detects an occurrence of an event, a change in content / information, or availability of new content / information, the system 120 can generate event data indicating these. The system 120 can use keywords to monitor various sources of information to determine occurrences of events corresponding to the keywords, determine changes in content / information corresponding to the keywords, and determine availability of new content / information corresponding to the keywords. Using the event data, the system 120 can determine that an output (such as an announcement, a push notification, a visual notification, an audible notification, or other type of output) is triggered in accordance with a subscription.

[0065] Figure 3 How NLU processing is performed on textual data is shown. Generally, the NLU component 260 attempts to semantically interpret textual data input therein. That is, the NLU component 260 determines the meaning behind the textual data based on individual words and / or phrases represented therein. The NLU component 260 interprets the textual data to derive an intent of the user and a piece of textual data that allows a device (e.g., the device 110, the system 120, the skill system 225, etc.) to complete the action.

[0066] The NLU component 260 can process textual data that includes several ASR hypotheses. The NLU component 260 can process all (or a portion) of the ASR hypotheses input therein. Even though the ASR component 250 can output multiple ASR hypotheses, the NLU component 260 can be configured to process only the highest scoring ASR hypothesis.

[0067] The NLU component 260 can include one or more recognizers 363. Each recognizer 363 can be associated with a different domain (e.g., smart home, video, music, weather, custom, etc.). Each recognizer 363 can operate at least partially in parallel with other recognizers 363 of the NLU component 260 on textual data input to the NLU component 260.

[0068] Each recognizer 363 can include a named entity recognition (NER) component 362. The NER component 362 attempts to identify syntactic and lexical information that can be used to interpret meaning with respect to textual data input therein. The NER component 362 identifies portions of textual data that correspond to named entities that can be applicable to processing performed on a domain-by-domain basis. The NER component 362 (or other components of the NLU component 260) can also determine whether a word refers to an entity whose identity is not explicitly mentioned in the textual data, e.g., “he,” “she,” “it,” or other anaphora, cataphora, etc.

[0069] Each recognizer 363, and more specifically each NER component 362, can be associated with a particular grammar model and / or database 373, a particular set of intents / actions 374, and a particular personalized lexicon 386. Each geoname lexicon 384 can include skill index vocabulary information associated with a particular user and / or device 110. For example, geoname lexicon A (384a) includes skill index vocabulary information 386aa-386an. For example, a user's music skill vocabulary information can include album titles, artist names, and song names, while a user's contacts list skill vocabulary information can include contact names. Because each user's music collection and contacts list can be different, such personalized information improves entity resolution.

[0070] The NER component 362 applies the grammar models 376 and vocabulary information 386 to determine a mention of one or more entities in the text data. In this way, the NER component 362 identifies a "slot" (corresponding to one or more particular words in the text data) that can be used for later processing. The NER component 362 can also tag each slot with a type (e.g., noun, place, city, artist name, song name, sender name, recipient name, entity name, sports team name, sports event, weather condition, prescription name, etc.).

[0071] Each grammar model 376 includes names of entities (i.e., nouns) that are common in speech about the particular domain to which the grammar model 376 relates, while the vocabulary information 386 is personalized to the user and / or device 110 that generates the user input. For example, a grammar model 376 associated with the shopping domain can include a database of words that people commonly use when discussing shopping.

[0072] Each recognizer 363 can also include an intent classification (IC) component 364. The IC component 364 parses the text data to determine an intent. An intent represents an action that the user wishes to perform. The IC component 364 can be in communication with a database 374 of words that are linked to intents. For example, a music intent database can link words and phrases such as "quiet," "volume off," and "mute" to the intent to turn off the music. The IC component 364 can also be in communication with a database 375 of actions that are linked to intents. For example, a music intent database can link the intent to turn off the music to the action of sending a command to the music application to turn off the music. <mute>Intents. For example, a subscription intent database can link words and phrases such as "tell me when," "notify me," and "keep me in the loop" to <tellmewhen>Intents. The IC component 364 identifies potential intents by comparing words and phrases in the text data to words and phrases in an intent database 374.

[0073] The particular IC component 364 can identify an intent that links to a domain-specific syntax framework 376 with "slots" to be filled. Each slot of the syntax framework 376 corresponds to a portion of the text data that the system believes corresponds to an entity. For example, corresponding to the text data "I want to go to the movies this weekend," the syntax framework 376 can have a slot for a movie title and a slot for a date. <playmusic>The grammatical framework of intent 376 can correspond to sentence structures such as "play {album name}", "play {song name}", "play {artist name}'s {song name}", and so on. For example, corresponding to <tellmewhen>The syntactic framework 376 of intent can correspond to sentence structures such as "remind me of {trigger event}", "notify me of {trigger event}", "remind me of {trigger event} by {action}", "notify me of {trigger event} by {action}", and the like. However, to make parsing more flexible, the syntactic framework 376 can not be structured as a sentence, but rather based on associating slots with syntactic tags.

[0074] For example, the NER component 362 can parse text data to identify words as subjects, objects, verbs, prepositions, and the like based on syntactic rules and / or models prior to recognizing named entities in the text data. The IC component 364 (e.g., implemented by the same recognizer 363 as the NER component 362) can use the identified verbs to identify an intent. The NER component 362 can then determine a syntactic model 376 associated with the identified intent. For example, corresponding to the sentence structure "remind me of {trigger event}", the NER component 362 can identify the intent as "remind" and determine the syntactic model 376 as "remind me of {trigger event}". <playmusic>The syntax model 376 of the intent of "play" can specify a list of slots applicable to playing the identified "object" and any object modifiers (e.g., prepositional phrases), such as {artist name}, {album name}, {song name}, etc. For example, corresponding to the example query "play {artist name}'s {album name}" the syntax model 376 can specify the following slots: <tellmewhen>The intent-specific grammar model 376 for the intent of "remind me to" can specify a list of time slots and any event modifiers that are applicable to notifying the user when the recognized "event" occurs (e.g., an email is received, a prescription is ready, a particular weather condition, etc.), such as {email sender}, {prescription name}, {raining}, etc. The NER component 362 can then search the corresponding fields in the lexicon 386, attempting to match the words and phrases in the text data previously tagged by the NER component 362 as grammatical objects or object modifiers to those identified in the lexicon 386.

[0075] The NER component 362 can perform semantic tagging, which is tagging of words or combinations of words according to their type / semantic meaning. The NER component 362 can use heuristic grammar rules to parse the text data, or can build a model using techniques such as Hidden Markov Models, Maximum Entropy Models, Log-Linear Models, Conditional Random Fields (CRF), etc. For example, the NER component 362 implemented by a music recognizer can parse and tag text data corresponding to "play mother's little helper by the Rolling Stones" as {verb}: "play", {object}: "mother's little helper", {object preposition}: "by", and {object modifier}: "the Rolling Stones". The NER component 362 identifies "play" as a verb, and the IC component 364 can determine that it corresponds to the semantic type of "action" or "activity". The NER component 362 identifies "mother's little helper" as an object, and the IC component 364 can determine that it corresponds to the semantic type of "song" or "album". The NER component 362 identifies "by" as an object preposition, and the IC component 364 can determine that it corresponds to the semantic type of "instrument" or "artist". The NER component 362 identifies "the Rolling Stones" as an object modifier, and the IC component 364 can determine that it corresponds to the semantic type of "band" or "group". <playmusic>intent. For example, the NER component 362 implemented by the subscription recognizer can parse and label the text data corresponding to "notify me when I receive an email from Joe" as {verb}: "notify," {object}: "receive an email," {preposition}: "from," and {object modifier}: "Joe." The NER component 362 identifies "notify" as a verb, and the IC component 364 can determine that it corresponds to an intent to be notified of an event <tellmewhen>intent. At this stage, the meanings of "mother's little helper" and "The Rolling Stones" have not been determined, but based on the grammatical rules and models, the NER component 362 has determined that the text of these phrases is related to the grammatical object (i.e., the entity) of the user input represented in the text data.

[0076] The framework linked to the intent is then used to determine which database fields should be searched to determine the meanings of these phrases, such as searching the user's gazetteer 384 for similarities to the framework slots. For example, <playmusic>The framework of the intent can indicate to attempt to resolve the identified argument based on {artist name}, {album name}, and {song name}, and another framework of the same intent can indicate to attempt to resolve the argument modifier based on {artist name} and resolve the argument based on {album name} and {song name} linked to the identified {artist name}. If the search of the place name dictionary 384 does not resolve the time slot / field using the place name dictionary information, the NER component 362 can search a database of general purpose words (e.g., in the knowledge base 372). For example, if the textual data includes "play a song by the Rolling Stones," after failing to determine an album name or song name called "a song" by "the Rolling Stones," the NER component 362 can search the database for the word "song." In the alternative, the general purpose words can be checked before the place name dictionary information, or both can be attempted, resulting in possibly two different results.

[0077] The NER component 362 can tag the textual data to impart meaning thereto. For example, the NER component 362 can tag "play mother's little helper by the Rolling Stones" as: {domain} music, {intent} play, {argument modifier} by the Rolling Stones, {argument} mother's little helper <playmusic>, {artist name} The Rolling Stones, {media type} song, and {song title} mother's little helper. As another example, the NER component 362 can label "play a song by The Rolling Stones" as: {domain} music, {intent} <playmusic>, {artist name} The Rolling Stones, and {media type} song. For example, the NER component 362 can tag "notify me when I receive an email from Joe" as: {domain} subscription, {intent} <tellmewhen>, {Trigger Event} received email, {Action} notify, {Sender Name} "Joe".

[0078] Figure 4 is a conceptual diagram of system components for configuring a subscription according to embodiments of the present disclosure. The system 120 can include a subscription manager 275, a subscription configuration component 410, a trigger type component 420, a device compatibility component 425, a delivery preference component 430, and a recurrence manager 435. The subscription manager 275 can receive a request to create a subscription. As described herein, a subscription refers to a request for an action to be performed by the system when a particular event occurs (e.g., an output / data / information is received in the future, a device is caused to perform an operation in the future, etc.). A user, such as a user 5 interacting with a device 110, a developer or skill / experience creator interacting with the system 120 to create / manage skills and applications, or other type of user, can request that the system 120 create a subscription. For example, a user can create a subscription to receive a notification, reminder, or another type of output when an event occurs (e.g., when an email is received, when a package is delivered, when a prescription is ready, when a television show is starting to play, when a person is home, etc.). As another example, a user can create a subscription to perform an action when an event occurs, such as turning on a light when a person is home, adjusting a temperature in a home at a particular time, etc.

[0079] The system 120 can receive user input data 402 including information for creating a subscription (e.g., trigger data indicating when to perform an action, action / output data indicating an output to be performed at the trigger, etc.). The system 120 can receive the user input data 402 from the user 5 via the device 110 as voice input / utterances, text input, graphical user interface input, and / or other forms of input. The system 120 can receive the user input data 402 from a skill / application developer or administrator to create a subscription for an end user, such as the user 5. The system 120 can determine (using NLU as described above) that the user input is related to creating or managing a subscription, and can send the user input data 402 to the subscription manager 275 for further processing. The subscription manager 275 can determine that the user input data 402 is related to creating a subscription, and can send the user input data 402 to the subscription configuration component 410.

[0080] In some embodiments, the user input data 402 can include text data representing a user request to create a subscription. In some embodiments, the user input data 402 can include NLU data / hypotheses corresponding to a user request to create a subscription. In some embodiments, the user input data 402 can include trigger information, output information, device information, user profile information, user preferences, and other data.

[0081] The subscription configuration component 410 can process the user input data 402 to create a subscription to perform an action when a particular event occurs in the future. The subscription configuration component 410 can determine, from the user input data 402, trigger data representing a change in state or an occurrence of an event, such as "receive an email," "a medical prescription is ready," "the Seahawks are playing," etc. The subscription configuration component 410 can determine, from the user input data 402, output data representing an action to be performed or an output to be generated when triggered according to the trigger data. The output data can indicate that a notification is to be generated, that a particular device is to be operated / controlled, etc. The subscription configuration component 410 can determine, from the user input data 402, metadata representing other information related to the subscription, such as a trigger type, an output type, delivery preference information, frequency / re-occurrence data, etc.

[0082] The trigger type component 420 can process the trigger data to determine a corresponding trigger type. The trigger type can represent an indication of a priority of the subscription based on the triggering event represented by the trigger data and other data associated with the trigger data and / or a user profile. The trigger type can be a priority indicator of high, medium-high, medium, medium-low, low, and other variations. The trigger type can be a priority indicator of high, medium-high, medium, medium-low, low, and other variations. In some embodiments, the trigger type can be a numerical value representing a priority of the subscription, where the numerical value can be between 1 and 5. The trigger type component 420 can determine the trigger type based on a category of the triggering event. For example, the triggering event "receive an email" can correspond to the category "mail / email." In other examples, the triggering event "a prescription is ready" can correspond to the category "medical / health," the triggering event "a football game starts" can correspond to the category "event / sport," the triggering event "when it rains" can correspond to the category "weather condition," the triggering event "a hurricane warning" can correspond to the category "severe weather condition," etc. Each category can be assigned a system priority indication, for example, the mail / email category can be assigned as low, medical / health can be assigned as high, event / sport can be assigned as low, weather condition can be assigned as medium, severe weather condition can be assigned as high, etc.

[0083] In some embodiments, the trigger type component 420 can use user input and / or user profile data to determine the trigger type. User profile data corresponding to user 5 can include user preferences related to priority indications for particular trigger events / categories, e.g., the user profile data can indicate that the "prescription ready" event has a medium priority. In another example, the user profile data can indicate that the "severe weather condition" event has a medium priority. In some embodiments, the trigger type component 420 can use user profile data corresponding to other users to determine the trigger type. The trigger type component 420 can use data determined from user input to determine the trigger type. The user input can include an indication of priority or can include other data from which a priority can be derived, e.g., "you tell me when I receive an email from Joe, that's important" or "Alexa, my medication is running out. Tell me when my prescription is ready," and in both cases the trigger type component 420 can determine the trigger type to be high.

[0084] In some embodiments, the trigger type indicates whether the time of the triggering event is known or unknown. In some cases, the time of the event is known because the event occurs at a determinable preset time, e.g., when a football game starts, when a television show is broadcast, when a weather condition occurs, etc. In this case, the trigger type component 420 determines that the trigger type corresponding to the subscription is a reminder operation. The trigger type component 420 can determine the trigger type to be a reminder operation even if the event time has not been announced or is not yet known, but can be determined before the event occurs.

[0085] In other cases, the time of the event is unknown or cannot be determined before the event occurs, e.g., when an email is received, when a medical prescription is ready, when a package is delivered, etc. In this case, the trigger type component 420 determines that the trigger type corresponding to the subscription is a notification operation.

[0086] In some embodiments, the trigger type component 420 can determine the trigger type of a trigger by using stored data indicating the trigger type of particular triggers. For example, the stored data can indicate that a trigger related to a broadcast event corresponds to a reminder operation, while a trigger related to receiving information from a skill or other system corresponds to a notification operation.

[0087] The device compatibility component 425 can determine the output type of the subscription based on the trigger type and the output capabilities of the device that will perform the action or present the output at the time of the event. In some embodiments, the user input data 402 can specify the output device for the subscription (e.g., "notify me on my mobile phone when my prescription is ready" or "turn on the hallway light when I get home"). In some embodiments, the user input data 402 can not specify the output device for the subscription (e.g., "tell me when the Seahawks game is on" or "remind me to close the window when it rains"). In that case, the device compatibility component 425 can determine the output capabilities of the device (e.g., the device 110) that received the request to create the subscription.

[0088] In some cases, the device that received the subscription request can be a voice-enabled device that can be configured to perform spoken language understanding and can be capable of outputting synthesized speech. In other cases, the device that received the subscription request (e.g., a mobile device) can include a voice-enabled application that is capable of performing spoken language understanding using user input but is not capable of outputting synthesized speech. In some cases, the device can need to enable the ability to receive push notifications from the voice-enabled application.

[0089] The device compatibility component 425 can determine whether the output capabilities of the output device for the subscription include the ability to output synthesized speech generated by the TTS processing component 280 described above. The device compatibility component 425 can determine whether the output capabilities of the output device for the subscription include the ability to receive push notifications, the ability to present visual output (e.g., a light, etc.), or the ability to present other types of output. The device compatibility component 425 can determine whether the output device is capable of performing the requested action, such as controlling a smart device according to a user request.

[0090] In some embodiments, when the output device is capable of outputting synthesized speech, the device compatibility component 425 can determine the output type to be an audio output that includes synthesized speech. If the output device is not capable of outputting synthesized speech, the device compatibility component 425 can determine the output type to be a push notification. In some embodiments, the device compatibility component 425 can determine whether the output device is enabled to receive push notifications. If the output device is not capable of outputting synthesized speech, a push notification, or other type of output, the output type determined by the device compatibility component 425 can be an error or can otherwise indicate that the output device does not support the output. The output types that are made include audio output, TTS output, visual output, push notifications, email messages, SMS messages, and other forms of output types.

[0091] The device compatibility component 425 can determine a set of devices associated with the user profile that will receive the output according to the subscription. When the subscription is created, the subscription configuration component 410 can associate multiple devices with the subscription data 450 as potential output devices that will receive the output when the event occurs. The form of the output for each device can also be stored. For example, the device compatibility component 425 can determine the potential output devices and their respective output forms as smart speaker device 110a: TTS output, smart phone 110b: push notification, smart watch 110c: visual / light notification, display device 110f: TTS output and visual notification, etc. When the subscription is executed (when event data is received indicating that the triggering event occurred), the subscription manager 275 can determine which of the potential output devices can receive the output. In some cases, all of the potential output devices can receive the output, and in some cases, only one potential device can receive the output. The subscription manager 275 can use various factors to make this determination, including but not limited to presence data (indicating that the user is in proximity to one or more devices when the output is to be presented), location data (indicating the user’s location (e.g., home vs. work)), interaction data (indicating that the user is interacting with a particular device when the output is to be presented), and other factors. These factors can indicate that the user is more likely to receive / view / access the output (if presented on a particular device).

[0092] The output type can indicate which form of output (e.g., announcement, push notification, visual notification, audio notification, etc.) will be used to indicate to the user that the triggering event occurred. The output type can also indicate the timing of presenting the output (e.g., interrupt the user with the output as soon as the event occurs, wait until the user interacts with the system, etc.). The output type can also indicate how many devices associated with the user profile will receive the output.

[0093] The delivery preference component 430 can determine how the output is presented when the event is triggered based on preferences defined by the user, skill / application developer, or the system. For example, the profile store 270 can include data indicating a user preference to receive notifications or reminders at particular devices associated with the user profile. The delivery preference component 430 can determine which devices to send the output to when the subscription is triggered, the type / form of the output, and other information related to presenting the output to the user. For example, a developer can specify that the output for a particular subscription be presented as a push notification, and not an audio output / announcement. As another example, a system preference can specify that the output for a particular subscription cannot be presented via a device associated with a child user or designated (by the user, manufacturer, system, etc.) as a child device.

[0094] The recurrence manager 435 can determine frequency data representing a number of times that output is to be generated or an action is to be performed. Some events can occur multiple times, such as receiving an email from a particular sender, a score change in a football game, a weather condition, a package delivery, etc. The user input data 402 can indicate a number of times that output (e.g., a notification, a reminder, an announcement, etc.) is to be presented. For example, the user input can be "tell me next time I receive an email from Joe," "remind me to close the windows every time it rains this week," "notify me next two times I receive a package delivery," etc. The recurrence manager 435 can process the user input data 402 to determine the frequency data. In the example of "tell me next time I receive an email from Joe," the recurrence manager 435 determines the frequency data to be one or a one-time occurrence. In the example of "remind me to close the windows every time it rains this week," the recurrence component 425 determines the frequency data to be recurring over the next seven days. In some embodiments, the recurrence manager 435 can determine a number of times that it is forecasted to rain at the user's location this week, and can determine the frequency data to be the number of times it is forecasted to rain this week. In the example of "notify me next two times I receive a package delivery," the recurrence component 435 can determine the frequency data to be two times / two times / two.

[0095] In some embodiments, the NLU component 260 can determine the frequency data by processing the user input to determine how many times the user intends to receive output. The recurrence component 435 / NLU component 260 can determine the frequency data by identifying words in the user input that indicate a time period. Such words can include, but are not limited to, when, every time, next N times, each time, whenever, next N days, next time, next time, etc.

[0096] In some embodiments, the recurrence manager 435 can determine that a triggering event occurs only once or only once during a given time period. For example, a triggering event like the opening ceremony of the 2020 Summer Olympics (e.g., the user input can be "remind me when the opening ceremony of the Olympics starts") occurs once during a given limited time period, such as within a year or a month. In this case, the recurrence manager 435 can determine the frequency data to be one / single / one time.

[0097] In some embodiments, the recurrence manager 435 can use user profile data, user preferences, past user interactions, past subscriptions, similar subscriptions by different users, and other data to determine the frequency data.

[0098] In some embodiments, if the recurrence manager 435 cannot determine the frequency data for a subscription with a certain level of confidence, the system 120 can ask the user how many times the subscription is to be performed, or confirm with the user how many times the subscription is to be performed. For example, the system 120 can output via synthesized speech or displayed text / dialog box "How many times do you want to be notified” or "Next time I will notify you when you receive a package” etc.

[0099] The data determined by the components 420, 425, 430, and 435, as well as the trigger data and output data determined by the subscription configuration component 410, are stored in the subscription store 455 as subscription data 450. In some embodiments, the subscription data 450 can include a user profile identifier to identify which user profiles are to receive the output when the subscription is triggered. The subscription data 450 can include a device identifier to identify which device is to receive the output when the subscription is triggered. Example data that can be included in the subscription store 455 is shown in Figure 6

[0100] In some embodiments, if the trigger type is a reminder operation, the subscription configuration component 410 can store the subscription data 450 in a reminder store 440. The output generated for a reminder subscription / operation can be available for use by other components or the user at a later time. For a reminder subscription, the subscription manager 275 can generate the output at the time of the event and present it to the user via the output device once. The user can snooze or otherwise indicate not to delete / reject the output / reminder, and the system 120 can present the output / reminder of the event to the user again at a later time. The reminder store 440 can store data indicating that the user snoozed the reminder output, and can cause the system 120 to present the reminder output again after a period of time or when the user requests to view / listen to his activities or snoozed reminders. For example, the system 120 can cause a device to output "The Sea Hawks game starts in 10 minutes,” the user can snooze the reminder, and the system 120 can output "Reminder: The Sea Hawks game has started” at a later time. In another example, the initial reminder output can be provided via a first output device (a smart speaker), the user can request to view his reminders via a second device (a mobile device), and in response, the system 120 can present the reminder output via the second device.

[0101] In some embodiments, if the trigger type is a notification operation, the subscription manager 275 can present the output generated for a notification subscription / operation only once, rather than continuing to remind the user of the event.​

[0102] In some embodiments, the subscription configuration component 410 can determine filter data and entity values corresponding to a subscription by processing the user input data 402. The filter data and entity values can be stored as subscription data 450. The filter data can represent data or information that can be used to identify a triggering event for the subscription. The entity values can indicate one or more entities associated with the triggering event. For example, if the user input is "tell me when I receive an email from Joe," the filter data can include an email address associated with Joe, and the entity value can include "Joe." Thus, when an email from Joe is received, the subscription manager 275 can use the email address and the entity value "Joe" to recognize that output is to be generated in accordance with the subscription associated with the event. As another example, if the user input is "notify me when my asthma prescription is ready," the filter data can be a user identifier associated with the user's medical information / prescription, and the entity value can be "asthma." As another example, if the user input is "remind me when the Seahawks play," the filter data can be a television show identifier associated with a football game, and the entity value can be "the Seahawks."

[0103] In some embodiments, the system 120 can generate output confirming the creation of a subscription. The subscription manager 275 can generate output including information related to the subscription to notify the user that the subscription has been successfully created. The confirmation output can be based on and / or can include an output type, trigger data, output data, device identifier, recurrences information, etc. For example, once a subscription is created, the subscription manager 275 can present the following output: "I will notify you when you receive an email from Joe." In some cases, the subscription manager 275 can generate output requesting the user to provide more information or perform an action to facilitate the creation of the subscription. For example, if the recurrences manager 435 is unable to determine frequency data for the subscription, the subscription manager 275 can present the following output: "Do you want to be notified every time you receive an email from Joe?" As another example, if the device compatibility component 425 determines that the output device does not support output in accordance with the subscription, the subscription manager 275 can generate the following output: "The device does not support notifications." As part of the functionality performed by the TTS component 280, the confirmation output can be generated using the natural language generation techniques described in connection with the output generation component 270. Figure 7A and Figure 7B

[0104] The output generated by the subscription manager 275 can be provided to the coordinator 230 for output to the device 110. If the device that received the subscription request is capable of outputting synthesized speech and / or audio data, the output can be processed by the TTS component 280 to generate synthesized speech. The output can be displayed as text data or another form on the device that received the subscription request.​

[0105] The system 120 can receive user input in response to the system presenting a confirmation of the subscription being created, where the user input can indicate a change to the created subscription. For example, if the system presents the confirmation "I will notify when you receive an email from Joe," the user can reply "No, please instead send me a notification on my phone," or the user can provide input in another form (e.g., text, selection of an option via a graphical user interface, etc.) indicating that the user wants to receive the notification on the mobile device instead of the notification. The subscription manager 275 then updates the corresponding subscription data 450 accordingly to indicate the output type of the push notification and the device identifier corresponding to the user's mobile device.

[0106] Further non-limiting examples of confirmation outputs that the system 120 can present to a user are described below. In one example, if the user input is "Tell me when I receive an email from Joe," and the output device is capable of outputting synthesized speech, the subscription configuration component 410 can create a notification subscription (because the time of occurrence is not determinable) with the frequency data set to "one-time," and the subscription manager 275 can output the confirmation "I will notify when you receive an email from Joe."

[0107] In another example, if the user input is "Tell me every time I receive an email from Joe," and the output device is capable of outputting synthesized speech, the subscription configuration component 410 can create a notification subscription (because the time of occurrence is not determinable) with the frequency data set to "reoccur / each time," and the subscription manager 275 can output the confirmation "I will notify every time you receive an email from Joe."

[0108] In another example, if the user input is "Tell me when I receive an email from Joe," and the output device is not capable of outputting synthesized speech and is enabled to receive push notifications, the subscription configuration component 410 can create a notification subscription with the frequency data set to "one-time," and the subscription manager 275 can output the confirmation "I will notify you when you receive an email from Joe."

[0109] In another example, if the user input is "Tell me when I receive an email from Joe," and the output device is not capable of outputting synthesized speech and is disabled to receive push notifications, the subscription manager 275 can output the request "To receive notifications, please enable push notifications for the app" prior to creating the subscription.

[0110] In another example, if the user input is "tell me when I receive an email from Joe" and the output device is not capable of outputting synthesized speech and is not capable of receiving push notifications (e.g., a smart television or an input device only), the subscription configuration component 410 can not create a subscription and the subscription manager 275 can output "device does not support annunciation or notification."

[0111] In another example, if the user input is "tell me when the Seahawks play" and the output device is capable of outputting synthesized speech, the subscription configuration component 410 can create an alert subscription (because the time of occurrence is determinable) with the data set to "one-time" and the subscription manager 275 can output the confirmation "I will alert you when the Seahawks play."

[0112] In another example, if the user input is "tell me when the Seahawks play" and the output device is not capable of outputting synthesized speech and is enabled to receive push notifications, the subscription configuration component 410 can create an alert subscription with the frequency data set to "one-time" and the subscription manager 275 can output the confirmation "I will alert you when the Seahawks play."

[0113] In another example, if the user input is "tell me when the Seahawks play" and the output device is not capable of outputting synthesized speech and is disabled to receive push notifications, the subscription configuration component 410 can create an alert subscription with the frequency data set to "one-time" and the subscription manager 275 can output the confirmation "enable push notifications for the app to receive notifications."

[0114] Figure 5 is a conceptual diagram of system components for performing a subscription according to embodiments of the disclosure. The skill component 290 can provide event data 502 to the subscription manager 275 (directly or via the coordinator 230). The event data 502 can indicate an occurrence of an event, such as receiving an email, a prescription being ready for pickup, a score change / update in a football game, etc. The subscription manager 275 can retrieve subscription data 450 from the subscription storage 455 and determine that the event data 502 triggers an output according to the subscription. The subscription manager 275 can generate output data 504 determined using the subscription data. The output data 504 can include content data representing content / information to be presented to a user or representing an action to be performed. The output data 504 can also include a device identifier associated with an output device that is to receive the output data. The output data 504 can also include an output type of how to present the output to the user (e.g., TTS, push notification, audio, visual, etc.).

[0115] The subscription manager 275 can send the output data 504 to the coordinator 230 for further processing so that the output can be presented to the user in response to the subscription being triggered. Using the combination of the subscription data 450 and the event data 502, the subscription manager 275 can determine the output data 504 to be presented to the user. Figure 2 The system 120 describing one or more components can determine an output 505 to send to the device 110 to present to the user. For example, the output data 504 can indicate that audio data including synthesized speech is to be sent to a smart speaker device associated with the user profile, and the system 120 can send the output 505a including the audio data to the device 110a. In another example, the output data 504 can indicate that a push notification is to be sent to a mobile device associated with the user profile, and the system 120 can send the output 505b including the notification data to the device 110b. In another example, the output data 504 can indicate that a visual indication is to be sent to a smart speaker device associated with the user profile, and the system 120 can send the output 505a including instructions for presenting the visual indication (e.g., a yellow ring) to the device 110a. The coordinator 230 can send a confirmation to the subscription manager 275 that the output / action of the subscription has been completed. The coordinator 230 can store execution data in the execution history storage 520. The execution data can be associated with the user profile for which the output was presented, and can represent information related to the execution of the stored subscription. The execution data can include an indication that the subscription was executed, a time that the subscription was executed, and other data.

[0116] After receiving the confirmation from the coordinator 230, the subscription manager 275 can use the recurrence manager 435 to determine whether the subscription is to be deleted or disabled. If the frequency of the subscription is one-time, the subscription manager 275 can determine to delete the subscription (at least with respect to the user profile). If the frequency of the subscription is recurring, the subscription manager 275 can update the subscription storage 455 accordingly.

[0117] The recurrence manager 435 can use the frequency data / subscription data 450 to determine whether the subscription is a one-time execution or a recurring execution (decision component 510). If the subscription is determined to be a one-time execution, the recurrence manager 435 can delete the corresponding subscription data record from the subscription storage 455. In some embodiments, the corresponding subscription data record can be indicated as disabled, completed, or executed.

[0118] If the subscription is determined to be reoccurring execution, the reoccurring manager 435 can retrieve data from the execution history storage 520 indicating whether the subscription was executed. The reoccurring manager 435 can determine whether the subscription was recently executed or executed within a given time period (to ensure that a previous execution of the subscription is not considered). After determining that the subscription was executed, the reoccurring manager 435 can use the frequency data corresponding to the subscription to determine (decision component 515) that the subscription has been executed the number of times indicated by the frequency data, and the corresponding subscription data record can be deleted / disabled. The reoccurring manager 435 can use the frequency data to determine that the subscription still needs to be executed in the future, and can update the frequency data of the corresponding subscription data record to reflect the remaining number of times the subscription is to be executed. For example, if the user input is "tell me the next three times I receive a package delivery," the frequency data can be "three" when the subscription is created. After the first time the user is notified of a package delivery, the reoccurring manager 435 can update the frequency data of the corresponding subscription data record to "two" to indicate that the user is to be notified of a package delivery two more times. After the third time the user is notified of a package delivery, the reoccurring manager 435 can delete / disable the corresponding subscription data record.

[0119] In some embodiments, the reoccurring manager 435 can use data stored at the execution history storage 520 to determine the number of times a subscription is executed. The reoccurring manager 435 can check the data at the storage 520 stored after the subscription is created. Using this information, the reoccurring manager 435 can determine to delete or update the subscription data record to indicate whether the system should continue to generate output for the subscription in the future.

[0120] In Figure 6 An example subscription data for various subscriptions that can be included in the subscription store 455 is shown. For example, for a first subscription represented by data record 602, the user input can be "tell me immediately when I receive an email from Joe," and as shown, the subscription data can be <user identifier = ABC>; <trigger = EmailReceived>; <trigger type = high>; <recurrence = single>; <output device = smart speaker>; <output type = audio / TTS>. For an example second subscription represented by data record 604, the user input can be "notify me the next two times my prescription is ready," and as shown, the subscription data can be <user identifier = ABC>; <trigger = PrescriptionReady>; <trigger type = medium>; <recurrence = two times>; <output device = mobile device>; <output type = push notification>. For an example third subscription represented by data record 606, the user input can be "remind me the next two weeks when the Seahawks play," and as shown, the subscription data can be <user identifier = BCD>; <trigger = SportsStart>; <trigger type = low>; <recurrence = next two Saturdays and Sundays>; <output device 1 = smart TV>; <output type 1 = error>; <output device 2 = mobile device>; <output type 2 = push notification>. For an example fourth subscription represented by data record 608, the user input can be "tell me when it rains this week," and as shown, the subscription data can be <user identifier = CDE>; <trigger = WeatherCondition>; <trigger type = medium>; <recurrence = each of 7 days>; <output device = smart speaker>; <output type = visual notification>. For an example fifth subscription represented by data record 610, the user input can be "notify me whenever there is a server weather alert in my area," and as shown, the subscription data can be <user identifier = CDE>; <trigger = SevereWeatherAlert>; <trigger type = high>; <recurrence = each time>; <output device 1 = smart speaker>; <output type 1 = audio / TTS>; <output device 2 = mobile device>; <output type 2 = push notification>.

[0121] One or more of the system 120 components described herein can implement one or more trained machine learning models. Such models can be trained and operated using a variety of machine learning techniques. Models can be trained and operated according to a variety of machine learning techniques. Such techniques can include, for example, neural networks such as deep neural networks and / or recurrent neural networks, inference engines, trained classifiers, and the like. Examples of trained classifiers include support vector machines (SVMs), neural networks, decision trees, AdaBoost (short for "Adaptive Boosting") in combination with decision trees, and random forests. Using SVMs as an example, SVMs are supervised learning models with an associated learning algorithm that analyzes data and identifies patterns in the data, and are commonly used for classification and regression analysis. Given a set of training examples, each marked as belonging to one of two categories, an SVM training algorithm builds a model that assigns new examples into one category or the other, making it a non-probabilistic binary linear classifier. More complex SVM models can be built using a training set that identifies more than two categories, where the SVM determines which category is most similar to the input data. The SVM model can be mapped such that examples of individual categories are separated by a clear margin. New examples are then mapped into the same space and predicted to belong to one category based on which side of the margin they fall on. A classifier can issue a "score" indicating which category the data most closely matches. The score can provide an indication of how closely the data matches a category.

[0122] To apply machine learning techniques, the machine learning process itself needs to be trained. Training a machine learning component, such as one of the trained models in this case, requires establishing "ground truth" for the training examples. In machine learning, the term "ground truth" refers to the accuracy of the training set classification for supervised learning techniques. Models can be trained using a variety of techniques, including backpropagation, statistical learning, supervised learning, semi-supervised learning, stochastic learning, or other known techniques.

[0123] Neural networks can also be used to perform ASR processing, including acoustic model processing and language model processing. In cases where the acoustic model uses a neural network, each node of the neural network input layer can represent an acoustic feature of a feature vector of acoustic features, such as can be output after a first pass of performing speech recognition, and each node of the output layer represents a score corresponding to a sub-word unit, such as a phoneme, triphone, or the like, and / or possibly corresponding to an associated state of the sound represented by the feature vector. For a given input to the neural network, multiple potential outputs are output, each with an assigned score representing a probability that the particular output is the correct output given the particular input. The highest scoring output of the acoustic model neural network can then be fed into an HMM, which can determine transitions between sounds before passing the results to a language model.

[0124] In the case of a language model using a neural network, each node of the input layer of the neural network can represent a previous word, and each node of the output layer can represent a potential next word determined by the trained neural network language model. Because the language model can be configured as a recurrent neural network that incorporates some history of words processed by the neural network, the prediction of the potential next word can be based on previous words in the utterance, not just the most recent word. The language model neural network can also output a weighted prediction of the next word.

[0125] The processing of the neural network is determined by the learned weights on each node input and the structure of the network. Given a particular input, the neural network determines the output of one layer at a time until the output layer of the entire network is computed.

[0126] The connection weights can be initially learned by the neural network during training, where a given input is associated with a known output. In a set of training data, various training examples are fed into the network. Each example will typically set the weights of the correct connections from input to output to 1, and assign weights of 0 to all connections. In another embodiment, the initial connection weights are assigned randomly. As the examples in the training data are processed by the neural network, the inputs can be sent to the network and compared to the associated outputs to determine how the performance of the network compares to the target performance. Using training techniques such as backpropagation, the weights of the neural network can be updated to reduce the errors produced by the neural network in processing the training data.

[0127] System components that can be used to perform unit selection, parametric TTS processing, and / or model-based audio synthesis are shown in Figure 7A As shown in Figure 7A , the TTS component / processor 780 can include a TTS front end 716, a speech synthesis engine 718, a TTS unit store 772, a TTS parameter store 780, and a TTS back end 734. The TTS unit store 772 can include, among other things, voice bank stores 778a-288n, which can include pre-recorded audio segments (referred to as units) to be used by the unit selection engine 730 in performing unit selection synthesis, as described below. The TTS parameter store 780 can include, among other things, parameter settings 768a-268n, which can be used by the parametric synthesis engine 732 in performing parametric synthesis, as described below. A particular set of parameter settings 768 can correspond to a particular voice profile (e.g., a whispering voice, an excited voice, etc.).

[0128] In various implementations of the disclosure, model-based audio data synthesis can be performed using the speech model 722 and the TTS front end 716. The TTS front end 716 can be the same as the front end used in traditional unit selection or parametric systems. In other implementations, some or all of the components of the TTS front end 716 are based on other trained models. However, the disclosure is not limited to any particular type of TTS front end 716. The speech model 722 can be used to synthesize speech without the need for TTS unit storage 772 or TTS parameter storage 780, as described in more detail below.

[0129] The TTS front end 716 converts input text data 710 (from, for example, an application, a user, a device, or other text source) into a symbolic language representation that can include linguistic contextual features, such as phoneme data, punctuation data, syllable-level features, word-level features, and / or emotion, speaker, accent, or other features, for processing by the speech synthesis engine 718. Syllable-level features can include syllable stress, syllable pace, syllable pitch variation, or other such syllable-level features; word-level features can include word stress, word pace, word pitch variation, or other such word-level features. Emotion features can include data corresponding to an emotion associated with the input text data 710, such as surprise, anger, or fear. Speaker features can include data corresponding to a type of speaker, such as gender, age, or profession. Accent features can include data corresponding to an accent associated with a speaker, such as a southern accent, a Boston accent, an English accent, a French accent, or other such accents.

[0130] The TTS front end 716 can also process other input data 715, such as text tags or text metadata, which can indicate, for example, how certain words should be pronounced, for example, by tags formatted according to the Speech Synthesis Markup Language (SSML) or in some other form indicating a desired quality of speech. For example, a first text tag can be included in the text to mark where the whispered text should begin (e.g., <begin whisper>) and a second tag can be included in the text to mark where the whispered text should end (e.g., <end whisper>). The tags can be included in the input text data 710 and / or the text for the TTS request can be accompanied by separate metadata indicating what text should be whispered (or some other indicated audio characteristic). The speech synthesis engine 718 can compare the annotated phonetic unit models and information stored in the TTS unit storage 772 and / or the TTS parameter storage 780 to convert the input text to speech. For example, the TTS front end 716 and speech synthesis engine 718 can include their own controllers / processors and memory, or they can use the controllers / processors and memory of the server 120, device 110, or other device. Similarly, the instructions for operating the TTS front end 716 and speech synthesis engine 718 can be located within the TTS components 780, within the memory and / or storage of the server 120, device 110, or in an external device.

[0131] Text data 710 input to the TTS components 780 can be sent to the TTS front end 716 for processing. The front end can include components for performing text normalization, language analysis, language prosody generation, or other such components. During text normalization, the TTS front end 716 can first process the text input and generate a standard text, converting, for example, numbers, abbreviations (such as Apt., St., etc.), symbols ($, %, etc.), and the like to written-out equivalents of the words.

[0132] During language analysis, the TTS front end 716 can analyze the language in the normalized text to generate a sequence of phonetic units corresponding to the input text. This process can be referred to as grapheme-to-phoneme conversion. Phonetic units include symbolic representations of sound units to be ultimately combined by the system and output as speech. For purposes of speech synthesis, text can be divided using a variety of sound units. The TTS component 780 can process speech based on phonemes (individual sounds), demi-phonemes, biphones (the latter half of one phoneme coupled with the former half of an adjacent phoneme), triphones (two consecutive phonemes), syllables, words, phrases, sentences, or other units. Each word can be mapped to one or more phonetic units. This mapping can be performed using a language dictionary stored by the system, for example, in the TTS storage component 772. Language analysis performed by the TTS front end 716 can also identify different grammatical components, such as prefixes, suffixes, phrases, punctuation, syntactic boundaries, and the like. Such grammatical components can be used by the TTS component 780 to produce audio waveform outputs that sound natural. The language dictionary can also include letter-to-sound rules and other tools that can be used to pronounce previously unrecognized words or combinations of letters that the TTS component 780 can encounter. Generally, the more information contained in the language dictionary, the higher the quality of the speech output.

[0133] Based on the language analysis, the TTS front end 716 can then perform language prosody generation, in which the phonetic units are annotated with desired prosodic characteristics (also referred to as acoustic features) that indicate how the desired phonetic units are to be pronounced in the final output speech. During this phase, the TTS front end 716 can consider and incorporate any prosodic annotations that accompany the text input to the TTS component 780. Such acoustic features can include syllable-level features, word-level features, emotion, speaker, accent, language, pitch, energy, duration, and the like. Application of the acoustic features can be based on prosody models available to the TTS component 780. Such prosody models indicate how a particular phonetic unit is to be pronounced in certain circumstances. The prosody models can take into account, for example, the position of a phoneme in a syllable, the position of a syllable in a word, the position of a word in a sentence or phrase, adjacent phonetic units, and the like. As with the language dictionary, a prosody model with more information can result in higher quality speech output compared to a prosody model with less information. In addition, the prosody models and / or phonetic units can be used to indicate particular voice qualities for the speech to be synthesized, where these voice qualities can match the voice qualities of the input speech (e.g., the phonetic units can indicate prosodic characteristics to make the final synthesized speech sound like a whisper based on the input speech being a whisper).

[0134] The output of the TTS front end 716, which can be referred to as a symbolic language representation, can include a sequence of phonetic units annotated with prosodic characteristics. This symbolic language representation can be sent to a speech synthesis engine 718 (which can also be referred to as a synthesizer) for conversion to an audio waveform of speech for output to an audio output device and ultimately to a user. The speech synthesis engine 718 can be configured to convert input text to high-quality, natural-sounding speech in an efficient manner. Such high-quality speech can be configured to sound as much like a human speaker as possible, or can be configured to be intelligible to a listener without attempting to mimic precise human vocalizations.

[0135] The speech synthesis engine 718 can use one or more different methods to perform speech synthesis. In one synthesis method, referred to as unit selection, the unit selection engine 730 matches the symbolic language representation created by the TTS front end 716 to a database of recorded speech, such as a database (e.g., TTS unit storage 772) that stores information about one or more vocal corpora (e.g., vocal libraries 778a-n). Each vocal library can correspond to various segments of audio recorded by a speaking person, such as a voice actor, with the segments stored as acoustic units (e.g., phonemes, diphones, etc.) in separate libraries 778. Each stored audio unit can also be associated with an index that lists various acoustic characteristics or other descriptive information about the unit. Each unit includes an audio waveform (such as a short.wav file of a particular sound) corresponding to a phonetic unit as well as a description of various features associated with the audio waveform. For example, an index entry for a particular unit can include information such as the pitch, energy, duration, harmonics, center frequency, position of the phonetic unit in a word, sentence, or phrase, neighboring phonetic units, etc. of the particular unit. The unit selection engine 730 can then use the information about each unit to select units to connect together to form the speech output.

[0136] The unit selection engine 730 matches the symbolic language representation to the information about the spoken audio units in the database. The unit database can include multiple examples of a phonetic unit to provide the system with many different options for connecting units into speech. The matching units that are determined to have the desired acoustic quality to create the desired output audio are selected and connected together (e.g., by the synthesis component 720) to form output audio data 790 representing the synthesized speech. Using all of the information in the unit database, the unit selection engine 730 can match units to the input text to select units that can form a natural-sounding waveform. One benefit of unit selection is that, depending on the size of the database, a natural-sounding speech output can be generated. As noted above, the larger the unit database of a vocal corpus, the more likely the system is to be able to construct a natural-sounding speech.

[0137] In another synthesis method, known as parametric synthesis, parameters such as frequency, volume, noise are altered by a parametric synthesis engine 732, digital signal processor, or other audio generation device to create an artificial speech waveform output. Parametric synthesis uses a computerized voice generator, sometimes referred to as a vocoder. Parametric synthesis can use acoustic models and various statistical techniques to match symbolic linguistic representations with desired output speech parameters. Using parametric synthesis, a computing system (e.g., the synthesis component 720) can generate audio waveforms with desired acoustic characteristics. Parametric synthesis can include the ability to be accurate at high processing speeds, and to process speech without large databases related to unit selection, but can also produce output speech quality that can not match unit selection. Unit selection and parametric techniques can be performed separately or combined together and / or combined with other synthesis techniques to produce speech audio output.

[0138] The TTS component 780 can be configured to perform TTS processing in multiple languages. For each language, the TTS component 780 can include specially configured data, instructions, and / or components to synthesize speech in the desired language. To improve performance, the TTS component 780 can modify / update the contents of the TTS storage 780 based on feedback of TTS processing results, thus enabling the TTS component 780 to improve speech recognition.

[0139] The TTS storage component 780 can be customized for individual users based on his / her personalized desired speech output. In particular, the speech units stored in the unit database can be taken from input audio data of the user speaking. For example, to create customized speech output of the system, the system can be configured with multiple voice inventories 778a-278n, where each unit database is configured with a different "voice" to match the desired speech quality. Such voice inventories can also be linked to a user account. The voice is selected by the TTS component 780 to synthesize speech. For example, one voice inventory can be stored for synthesizing a whisper (or speech that approximates a whisper), another voice inventory can be stored for synthesizing excited speech (or speech that approximates excited speech), and so on. To create different voice inventories, a large number of TTS training utterances can be spoken by a person (such as a voice actor) and recorded by the system. The audio associated with the TTS training utterances can then be segmented into small audio segments and stored as part of the voice inventory. The person speaking the TTS training utterances can speak with different voice qualities to create customized voice inventories, e.g., the person can whisper the training utterances, speak the training utterances with excitement, and so on. Thus, the audio of each customized voice inventory can match the corresponding desired voice quality. The customized voice inventories 778 can then be used during run-time to perform unit selection to synthesize speech with a speech quality that corresponds to the input speech quality.

[0140] Additionally, parametric synthesis can be used to synthesize speech with a desired speech quality. For parametric synthesis, the parametric characteristics can be configured to match the desired speech quality. If excited speech is desired to be simulated, the parametric characteristics can indicate an increased speech rate and / or pitch of the resulting speech. Many other examples are possible. The desired parametric characteristics for a particular speech quality can be stored in a "voice" profile (e.g., parameter settings 768) and used for speech synthesis when a particular speech quality is desired. Customized speech (for unit selection or parametric synthesis) can be created based on a combination of multiple desired speech qualities. For example, one voice can be "shouting" and another voice can be "shouting and emphasizing." Many such combinations are possible.

[0141] Unit selection speech synthesis can be performed as follows. Unit selection includes a two-step process. First, the unit selection engine 730 determines what speech units to use and then combines them such that the particular combination of units matches the desired phonetic and acoustic characteristics and creates the desired speech output. The units can be selected based on a cost function that represents how well a particular unit matches a speech segment to be synthesized. The cost function can represent a combination of different costs that represent different aspects of how well a particular speech unit works for a particular speech segment. For example, a target cost indicates how well a given speech unit matches the characteristics of the desired speech output (e.g., pitch, prosody, etc.). A connection cost represents how well a particular speech unit matches adjacent speech units (e.g., speech units that occur immediately before or immediately after the particular speech unit) in order to connect the speech units together in the final synthesized speech. The total cost function is a combination of the target cost, the connection cost, and other costs that can be determined by the unit selection engine 730. As part of unit selection, the unit selection engine 730 selects speech units that have the lowest total combined cost. For example, a speech unit that has a very low target cost can not necessarily be selected (if its connection cost is high).

[0142] The system can be configured with one or more speech corpora for unit selection. Each speech corpus can include a database of speech units. The database of speech units can be stored in the TTS unit storage 772 or another storage component. For example, different unit selection databases can be stored in the TTS unit storage 772. Each database of speech units (e.g., a speech bank) includes recorded speech utterances with corresponding text of the utterances aligned with the utterances. The database of speech units can include many hours of recorded speech (in the form of audio waveforms, feature vectors, or other formats) that can take up a large amount of storage. The unit samples in the database of speech units can be categorized in a variety of ways, including phonetic units (phonemes, diphones, words, etc.), language prosody labels, sequences of acoustic features, speaker identity, etc. The sample utterances can be used to create mathematical models that correspond to the desired audio output for a particular phonetic unit. When matching symbolic language representations, the speech synthesis engine 718 can attempt to select units in the database of speech units that most closely match the input text, including phonetic units and prosody annotations. Generally, the larger the speech corpus / database of speech units, the better the speech synthesis that can be achieved by the larger number of unit samples that can be selected to form the precise desired speech output.

[0143] Parameter speech synthesis based on vocoders can be performed as follows. The TTS component 780 can include an acoustic model or other model that can convert symbolic language representations to synthesized acoustic waveforms of the text input based on audio signal manipulation. The acoustic model includes rules that can be used by the parameter synthesis engine 732 to assign particular audio waveform parameters to input phonetic units and / or prosody annotations. The rules can be used to calculate scores that represent the likelihood that a particular audio output parameter (such as frequency, volume, etc.) corresponds to a portion of the input symbolic language representation from the TTS front end 716.

[0144] The parameter synthesis engine 732 can use a variety of techniques to match the speech to be synthesized with the input phonetic units and / or prosodic annotations. One common technique is to use a Hidden Markov Model (HMM). The HMM can be used to determine the probability that an audio output should match a text input. The HMM can be used to convert parameters from a linguistic and acoustic space into parameters used by a vocoder (digital speech encoder) to artificially synthesize the desired speech. Using an HMM, a number of states are presented, where the states collectively represent one or more potential acoustic parameters to be output to a vocoder, and each state is associated with a model, such as a Gaussian Mixture Model. Transitions between states can also have an associated probability that represents the likelihood that a state can be reached from a previous state. The sound to be output can be represented as a path between states of the HMM, and multiple paths can represent multiple possible audio matches for the same input text. Each portion of the text can be represented by multiple potential states that correspond to different known pronunciations of the phonemes and their parts, such as phoneme identity, stress, accent, position, etc. An initial determination of the probability of a potential phoneme can be associated with one state. As the speech synthesis engine 718 processes new text, the state can change or remain the same based on the processing of the new text. For example, the pronunciation of a previously processed word can change based on a later processed word. A Viterbi algorithm can be used to find the most likely sequence of states based on the processed text. The HMM can generate speech in a parameterized form that includes parameters that are converted into an audio segment by a vocoder, such as a fundamental frequency (f0), noise envelope, spectral envelope, etc. The output parameters can be configured for a particular vocoder, such as a STRAIGHT vocoder, a TANDEM-STRAIGHT vocoder, a WORL D vocoder, an HNM (Harmonic plus Noise) based vocoder, a CELP (Code Excited Linear Prediction) vocoder, a GlottHMM vocoder, an HSM (Harmonic / Stochastic Model) vocoder, or other vocoders.

[0145] In addition to calculating potential states of one audio waveform as potential matches to phonetic units, the parameter synthesis engine 732 can also calculate potential states of other potential audio outputs, such as various ways of making a particular phoneme or diphthong pronunciation, as potential acoustic matches to acoustic units. In this way, multiple states and state transition probabilities can be calculated.

[0146] The possible states and possible state transitions computed by the parameter synthesis engine 732 can result in multiple potential audio output sequences. Based on the acoustic model and other potential models, the potential audio output sequences can be scored according to the confidence level of the parameter synthesis engine 732. The highest scoring audio output sequence that includes the parameter stream to be synthesized can be selected, and digital signal processing can be performed by a vocoder or similar component to create an audio output that includes synthesized speech waveforms corresponding to the parameters of the highest scoring audio output sequence and, if the correct sequence was selected, the input text. Different parameter settings 768 that can represent acoustic settings that match a particular parameter "voice" can be used by the synthesis component 722 to ultimately create the output audio data 790.

[0147] When unit selection is performed, after the unit selection engine 730 selects a unit, the audio data corresponding to that unit can be passed to the synthesis component 720. The synthesis component 720 can then process the audio data of the unit to create modified audio data, where the modified audio data reflects a desired audio quality. The synthesis component 720 can store various operations that can convert the unit audio data to the modified audio data, where different operations can be performed based on a desired audio effect (e.g., lower, shout, etc.).

[0148] As an example, the input text can be received with metadata, such as SSML tags, that indicate that a selected portion of the input text should be whispered when output by the TTS module 780. For each cell corresponding to the selected portion, the synthesis component 720 can process the cell’s audio data to create modified cell audio data. The modified cell audio data can then be concatenated to form the output audio data 790. The modified cell audio data can also be concatenated with unmodified audio data, depending on when the whispered speech is desired to begin and / or end. While the modified audio data can be sufficient to cause the output audio data to have the desired audio quality, other factors can also affect the final output of the audio, such as playback speed, background effects, etc., which can be outside the control of the TTS module 780. In that case, other output data 785 can be output with the output audio data 790 so that the final playback device (e.g., the device 110) receives playback instructions that can help create the desired output audio. Thus, the other output data 785 can include instructions or other data that indicate playback device settings, such as volume, playback rate, etc., or other data that indicate how the output audio data, including synthesized speech, should be output. For example, for a whispered speech, the output audio data 790 can include other output data 785 that can include a prosody tag or other indicator that instructs the device 110 to slow the playback of the output audio data 790, thus making the final audio sound more like a whispered speech, which is typically slower than normal speech. In another example, the other output data 785 can include a volume tag that instructs the device 110 to output the speech at a volume level that is less than the current volume setting of the device 110, thus improving the effect of a quiet whisper.

[0149] Figure 7B An embodiment of a speech model 722 is shown. The speech model can include an encoder 750, an attention mechanism 752, and a decoder 754. This arrangement of components can be referred to as a sequence-to-sequence model. The encoder 750 and / or the decoder 754 can be a neural network with one or more layers. The layers can be composed of one or more nodes; each node can receive input data or output of a node from a previous layer and process the data according to one or more model weights. For example, a node can multiply an input value by a model weight to produce an output. The neural network can be a deep neural network (DNN), a convolutional neural network (CNN), and / or a recurrent neural network (RNN). The neural network can be trained using training data, such as recordings of utterances and corresponding text.

[0150] Encoder 750 can receive data, such as input text data 710 corresponding to input data from device 110, NLU result data, and / or other data. Encoder 750 can encode this information into a context vector, which is input to decoder 754. Optionally, attention mechanism 752 can receive this context vector, as well as outputs of other nodes of encoder 750, and weight (e.g., "pay attention to") different outputs of encoder 750 differently. Decoder 754 can then use the context vector and / or outputs of attention mechanism 752 to generate output audio data 790, which can include response data.

[0151] Figure 8 FIG. 1 is a block diagram conceptually illustrating a device 110 that can be used with the system. Figure 9 FIG. 2 is a block diagram conceptually illustrating example components of a remote device, such as system 120 and skill system 225, which can assist with ASR processing, NLU processing, etc. System (120 / 225) can include one or more servers. As used herein, a "server" can refer to a traditional server as understood in a server / client computing structure, but can also refer to a number of different computing components that can assist with operations discussed herein. For example, a server can include one or more physical computing components (such as a rack server) that are physically and / or through network connections to other devices / components and capable of performing computing operations. A server can also include one or more virtual machines that emulate a computer system and run on one device or across multiple devices. A server can also include other combinations of hardware, software, firmware, etc. to perform operations discussed herein. A server can be configured to operate using one or more of a client-server model, a computer cluster model, grid computing techniques, fog computing techniques, mainframe techniques, utility computing techniques, peer-to-peer model, sandbox techniques, or other computing techniques.

[0152] A number of systems (120 / 225) can be included in the overall system of the present disclosure, such as one or more systems 120 for performing ASR processing, one or more systems 120 for performing NLU processing, one or more skill systems 225 for performing actions in response to user input, etc. In operation, each of these systems can include computer-readable and computer-executable instructions that reside on a respective device (120 / 225), as will be discussed further below.

[0153] Each of these devices (110 / 120 / 225) can include one or more controllers / processors (804 / 904), which can each include a central processing unit (CPU) for processing data and computer-readable instructions, and a memory (806 / 906) for storing the data and instructions of the respective device. The memory (806 / 906) can individually include volatile random access memory (RAM), non-volatile read-only memory (ROM), non-volatile magnetic resistance memory (MRAM), and / or other types of memory. Each device (110 / 120 / 225) can also include a data storage component (808 / 908) for storing data and controller / processor-executable instructions. Each data storage component (808 / 908) can individually include one or more types of non-volatile storage devices, such as magnetic storage devices, optical storage devices, solid-state storage devices, etc. Each device (110 / 120 / 225) can also be connected to removable or external non-volatile memory and / or storage devices (such as removable memory cards, storage key drives, networked storage devices, etc.) through a respective input / output device interface (802 / 902).

[0154] Computer instructions for operating each device / system (110 / 120 / 225) and its various components can be executed by the controller / processor (804 / 904) of the respective device at runtime using the memory (806 / 906) as temporary "working” storage. The device’s computer instructions can be stored in non-volatile memory (806 / 906), storage (808 / 908), or external devices in a non-transitory manner. Alternatively, some or all of the executable instructions can be embedded in hardware or firmware on the respective device, in addition to or instead of software.

[0155] Each device (110 / 120 / 225) includes an input / output device interface (802 / 902). The various components can be connected through the input / output device interface (802 / 902), as will be discussed further below. In addition, each device (110 / 120 / 225) can include an address / data bus (824 / 924) for transmitting data among the components of the respective device. In addition to (or instead of) being connected to other components across the bus (824 / 924), each component within a device (110 / 120 / 225) can also be directly connected to other components.

[0156] Reference Figure 8 , the device 110 can include an input / output device interface 802 that connects to various components, such as an audio output component, such as a speaker 812, wired or wireless earphones (not shown), or other components capable of outputting audio. The device 110 can also include an audio capture component. The audio capture component can be, for example, a microphone 820 or microphone array, wired or wireless earphones (not shown), or the like. If a microphone array is included, an approximate distance to a point of origin of sound can be determined through acoustic localization based on time and amplitude differences between sounds captured by different microphones in the array. The device 110 can additionally include a display 816 for displaying content. The device 110 can also include a camera 818.

[0157] Via the antenna 814, the input / output device interface 802 can connect to one or more networks 199 via a wireless local area network (WLAN) (such as WiFi) radio, Bluetooth, and / or wireless network radio, such as a radio capable of communicating with a wireless communication network such as a Long Term Evolution (LTE) network, WiMAX network, 3G network, 4G network, 5G network, and the like. Wired connections, such as Ethernet, can also be supported. Through the networks 199, the system can be distributed in a network environment. The I / O device interface (802 / 902) can also include communication components that allow for the exchange of data between devices, such as different physical systems or other components in a collection of systems.

[0158] The components of the device 110, system 120, or skill system 225 can include their own dedicated processors, memories, and / or storage devices. Alternatively, one or more of the components of the device 110, system 120, or skill system 225 can utilize the I / O interface (802 / 902), processor (804 / 904), memory (806 / 906), and / or storage device (808 / 908) of the device 110, system 120, or skill system 225, respectively. Thus, the ASR component 250 can have its own I / O interface, processor, memory, and / or storage device; the NLU component 260 can have its own I / O interface, processor, memory, and / or storage device; and so on for the various components discussed herein.

[0159] As noted above, multiple devices can be employed in a single system. In such multi-device systems, each of the devices can include different components for performing different aspects of the system processing. Multiple devices can include overlapping components. As described herein, the components of the device 110, system 120, and skill system 225 are illustrative and can be positioned as standalone devices or can be included, in whole or in part, as components of larger devices or systems.

[0160] As Figure 10 As shown, multiple devices (110a-110j, 120, 225) can include components of a system, and devices can connect through network 199. Network 199 can include a local or wide-area network, or can include the Internet. Devices can connect to network 199 through wired or wireless connections. For example, speech detection device 110a, smart phone 110b, smart watch 110c, tablet 110d, vehicle 110e, display device 110f, smart television 110g, washing machine / dryer 110h, refrigerator 110i, and / or toaster 110j can connect to network 199 through a wireless service provider, through a WiFi or cellular network connection, etc. Other devices are included as supporting devices that connect to the network, such as system 120, skill system 225, and / or other devices. Supporting devices can connect to network 199 through wired connections or wireless connections. Network devices can use one or more built-in or connected microphones or other audio capture devices to capture audio, with processing performed by an ASR component, NLU component, or other component of the same device or another device connected via network 199, such as ASR component 250, NLU component 260, etc. of one or more systems 120.

[0161] The foregoing can also be understood in view of the following clauses:

[0162] 1. A computer-implemented method comprising:

[0163] during a first time period:

[0164] receiving, from a device, audio data corresponding to an utterance, the audio data being associated with a user profile;

[0165] processing the audio data to determine intent data, the intent data indicating a request to receive an output when a future event occurs;

[0166] determining trigger data representing the event;

[0167] determining priority data corresponding to the event, the priority data indicating a high priority; and

[0168] storing the trigger data and the priority data;

[0169] during a second time period after the first time period:

[0170] receiving event data indicating an occurrence of the event;

[0171] determining, using the trigger data, that the event data triggers an output with respect to the user profile;

[0172] determine, using the priority data, to output an announcement of the event occurrence;

[0173] determine that the device is capable of outputting synthesized speech;

[0174] determine output text data representing the announcement;

[0175] process the output text data using text-to-speech (TTS) processing to determine output audio data; and

[0176] send the output audio data to the device.

[0177] 2. The computer-implemented method of clause 1, further comprising:

[0178] during the first time period:

[0179] process the audio data using automatic speech recognition (ASR) to determine input text data;

[0180] process the input text data using natural language understanding (NLU) to determine first frequency data indicative of an intent to receive the output a plurality of times when the event occurs; and

[0181] store the first frequency data;

[0182] during the second time period:

[0183] determine, using the first frequency data and a number of times the event has occurred, second frequency data indicative of a remaining number of times the output is to be generated; and

[0184] associate the second frequency data with the trigger data.

[0185] 3. The computer-implemented method of clause 1 or 2, further comprising:

[0186] during a third time period:

[0187] receive, from a second device, second audio data corresponding to a second utterance, the second audio data being associated with the user profile;

[0188] process the second audio data to determine second intent data indicative of a second request to receive an output when a second event occurs;

[0189] determine, using NLU, second trigger data representing the second event;

[0190] determine second priority data corresponding to the second trigger data, the second priority data indicating a high priority; and

[0191] storing the second trigger data and the second priority data;

[0192] during a fourth time period after the third time period:

[0193] receiving second event data,

[0194] determining, using the second trigger data, that the second event data triggers output with respect to the user profile;

[0195] determining that the second device is unable to output synthesized speech;

[0196] determining, using the second priority data, to output a push notification to the second device and a third device associated with the user profile, the push notification representing an indication that the second event occurred;

[0197] generating notification data representing the push notification; and

[0198] sending the notification data to the second device and the third device.

[0199] 4. A computer-implemented method comprising:

[0200] receiving input data representing natural language, the input data being associated with a user profile;

[0201] processing the input data to determine intent data, the intent data indicating a request to receive output when an event occurs;

[0202] determining, using the input data, trigger data representing the event;

[0203] determining a priority corresponding to the trigger data; and

[0204] associating the trigger data and the priority with the user profile.

[0205] 5. The computer-implemented method of clause 4, further comprising:

[0206] determining a category corresponding to the event;

[0207] determining first priority data corresponding to the event;

[0208] determining second priority data represented in the input data; and

[0209] determining the priority using the first priority data and the second priority data.

[0210] 6. The computer-implemented method of clause 4 or 5, further comprising:

[0211] determining that the priority is high;

[0212] using the priority to determine that the output is an announcement;

[0213] determining that an output capability of an output device includes the ability to output synthesized speech;

[0214] generating output data representing a confirmation that an announcement will be generated when the event occurs;

[0215] processing the output data using text-to-speech (TTS) processing to determine output audio data; and

[0216] sending the output audio data to the output device.

[0217] 7. The computer-implemented method of clause 4, 5, or 6, further comprising:

[0218] determining that the priority is high;

[0219] determining that a first output device is not capable of outputting synthesized speech;

[0220] using the priority to determine that a push notification is to be output to the first output device and a second output device;

[0221] generating output data representing a confirmation that a notification will be generated when the event occurs; and

[0222] sending the output data to the first output device.

[0223] 8. The computer-implemented method of clause 4, 5, 6, or 7, further comprising:

[0224] determining a trigger type associated with the trigger data, the trigger type indicating a time at which the event will occur;

[0225] associating the trigger type and the time with the trigger data;

[0226] generating output data representing a confirmation that a reminder of the event occurring will be generated;

[0227] processing the output data using TTS processing to determine output audio data; and

[0228] sending the output audio data to a device.

[0229] 9. The computer-implemented method of clause 4, 5, 6, 7, or 8, further comprising:

[0230] determining that the priority is low;

[0231] using the priority to determine that the output is a push notification;

[0232] generating output data representing a confirmation that a notification will be generated when the event occurs; and

[0233] sending the output data to a device.

[0234] 10. The computer-implemented method of clause 4, 5, 6, 7, 8, or 9, further comprising:

[0235] receiving audio data corresponding to the input data;

[0236] processing the audio data using automatic speech recognition (ASR) to determine input textual data;

[0237] processing the input textual data using natural language understanding (NLU) to determine frequency data indicative of an intent to receive the output once;

[0238] associating the frequency data with the trigger data; and

[0239] during a time period after receiving the audio data:

[0240] receiving event data,

[0241] determining, using the trigger data, that the event data triggers output with respect to the user profile,

[0242] determining output data using the priority,

[0243] sending the output data to a device, and

[0244] determining to delete the trigger data based on the frequency data.

[0245] 11. The computer-implemented method of clause 4, 5, 6, 7, 8, 9, or 10, further comprising:

[0246] receiving audio data corresponding to the input data;

[0247] processing the audio data using automatic speech recognition (ASR) to determine input textual data;

[0248] processing the input text data using natural language understanding (NLU) to determine first frequency data indicative of a number of times to generate the output;

[0249] associating the first frequency data with the trigger data; and

[0250] during a time period after receiving the audio data:

[0251] receiving event data,

[0252] determining, using the trigger data, that the event data triggers a first output with respect to the user profile,

[0253] determining second frequency data indicative of a remaining number of times to generate the output, and

[0254] associating the second frequency data with the trigger data.

[0255] 12. The computer-implemented method of clause 4, 5, 6, 7, 8, 9, 10, or 11, further comprising:

[0256] determining, using the priority, that the output is a notification;

[0257] determining that an output device includes a capability to output notification data;

[0258] determining that receipt of notification data is disabled at the output device;

[0259] generating output data representing a request to enable receipt of notification data at the output device; and

[0260] sending the output data to the output device.

[0261] 13. A system comprising:

[0262] at least one processor; and

[0263] at least one memory including instructions that, when executed by the at least one processor, cause the system to:

[0264] receive input data representing natural language, the input data being associated with a user profile;

[0265] process the input data to determine intent data, the intent data indicating a request to receive an output when an event occurs;

[0266] determine, using the input data, trigger data representing the event;

[0267] determine a priority corresponding to the trigger data; and

[0268] associating the trigger data and the priority with the user profile.

[0269] 14. The system of clause 13, wherein the instructions, when executed by the at least one processor, further cause the system to:

[0270] determine that the priority is high;

[0271] determine, using the priority, that the output is an announcement;

[0272] determining that an output capability of an output device includes the ability to output synthesized speech;

[0273] generating output data representing a confirmation that an announcement will be generated when the event occurs;

[0274] processing the output data using text-to-speech (TTS) processing to determine output audio data; and

[0275] sending the output audio data to the output device.

[0276] 15. The system of clause 13 or 14, wherein the instructions, when executed by the at least one processor, further cause the system to:

[0277] determine that the priority is high;

[0278] determining that a first output device is not capable of outputting synthesized speech;

[0279] determining, using the priority, to output a push notification to the first output device and a second output device associated with the user profile;

[0280] generating output data representing a confirmation that a notification will be generated when the event occurs; and

[0281] sending the output data to the first output device.

[0282] 16. The system of clause 13, 14, or 15, wherein the instructions, when executed by the at least one processor, further cause the system to:

[0283] determining a trigger type associated with the trigger data, the trigger type indicating a time at which the event will occur;

[0284] associating the trigger type and the time with the trigger data;

[0285] generating output data representing a confirmation that a reminder of the event occurring will be generated;

[0286] processing the output data using TTS processing to determine output audio data; and

[0287] sending the output audio data to a device.

[0288] 17. The system of clause 13, 14, 15, or 16, wherein the instructions, when executed by the at least one processor, further cause the system to:

[0289] determine that the priority is low;

[0290] determine, using the priority, that the output is a push notification;

[0291] generate output data representing a confirmation that a notification will be generated when the event occurs; and

[0292] send the output data to a device.

[0293] 18. The system of clause 13, 14, 15, 16, or 17, wherein the instructions, when executed by the at least one processor, further cause the system to:

[0294] receive audio data corresponding to the input data;

[0295] process the audio data using automatic speech recognition (ASR) to determine input textual data;

[0296] process the input textual data using natural language understanding (NLU) to determine frequency data indicative of an intent to receive the output once;

[0297] associate the frequency data with the trigger data; and

[0298] during a time period after receiving the audio data:

[0299] receive event data,

[0300] determine, using the trigger data, that the event data triggers output with respect to the user profile,

[0301] determine output data using the priority,

[0302] send the output data to a device, and

[0303] determine, based on the frequency data, to delete the trigger data and the priority.

[0304] 19. The system of clause 13, 14, 15, 16, 17, or 18, wherein the instructions, when executed by the at least one processor, further cause the system to:

[0305] receiving audio data corresponding to the input data;

[0306] processing the audio data using automatic speech recognition (ASR) to determine input textual data;

[0307] processing the input textual data using natural language understanding (NLU) to determine first frequency data indicative of a number of times to generate the output;

[0308] associating the first frequency data with the trigger data; and

[0309] during a time period after receiving the audio data:

[0310] receiving event data,

[0311] determining, using the trigger data, that the event data triggers a first output with respect to the user profile,

[0312] determining second frequency data indicative of a remaining number of times to generate the output, and

[0313] associating the second frequency data with the trigger data.

[0314] 20. The system of clauses 13, 14, 15, 16, 17, 18, or 19, wherein the instructions, when executed by the at least one processor, further cause the system to:

[0315] determine, using the priority, that the output is a notification;

[0316] determine that an output device includes a capability to output notification data;

[0317] determine that receipt of notification data is disabled at the output device;

[0318] generate output data representing a request to enable receipt of notification data at the output device; and

[0319] send the output data to the output device.

[0320] The concepts disclosed herein are applicable within a number of different apparatuses and computer systems, including, for example, general purpose computing systems, speech processing systems, and distributed computing environments.

[0321] The above aspects of the present disclosure are intended to be illustrative. The aspects were chosen for illustration of the principles and applications of the present disclosure, and are not intended to be exhaustive or to limit the disclosure. Many modifications and variations of the disclosed aspects will be apparent to those of ordinary skill in the art. Those of ordinary skill in the art in the computer and speech processing arts will recognize that the components and process steps described herein can be interchanged or combined with other components or steps, or combinations of components or steps, and still achieve the benefits and advantages of the present disclosure. Moreover, those of ordinary skill in the art will appreciate that the present disclosure can be practiced with some or all of the specific details and steps disclosed herein.

[0322] Aspects of the disclosed system can be implemented as a computer method or article of manufacture such as a memory device or non-transitory computer readable storage medium. The computer readable storage medium can be read by a computer and can include instructions for causing a computer or other device to perform processes described herein. The computer readable storage medium can be realized by a volatile computer memory, a non-volatile computer memory, a hard disk drive, a solid-state memory, a flash drive, a removable disk, and / or other media. Additionally, components of the system can be implemented as firmware or hardware, such as an Acoustic Front End (AFE) that includes, among other things, analog and / or digital filters (e.g., filters configured as firmware of a Digital Signal Processor (DSP)).

[0323] Conditional language used herein, such as, among others, "can," "could," "might," "may," "e.g.," and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and / or steps. Thus, such conditional language is not generally intended to imply that one or more embodiments are required to include a feature, element and / or step. The terms "comprising," "including," "having," and the like are synonymous and are used inclusively, in an open-ended fashion, and do not exclude additional elements, features, acts, operations, and so forth. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, in a list of elements, the phrase "one or more of the elements" means one element, some of the elements, or all of the elements. In addition, the terms "a" and "an" are defined as one or more unless explicitly stated otherwise, and the term "plurality" is defined as two or more unless explicitly stated otherwise.

[0324] Unless specifically stated otherwise, as apparent from the preceding discussions, it is appreciated that, throughout the specification, discussions utilizing terms such as "at least one of," "comprising," "including," or "comprising," "containing," or "including," "having," and / or "including" are intended to convey that the item(s) named after such a term is / are among the items constituting the item(s) of the subject matter. For example, reference to "at least one of a and b" can mean "a" or "b" or "a and b." Likewise, reference to "comprising a and b" can mean "a" or "b" or "a and b." Further, unless otherwise specified, the use of "or" is intended to represent an inclusive "or" in which the items connected by "or" are not mutually exclusive, and "or" can be used in the context of "and / or." For example, "A or B" or "A or B or C" can mean "A," "B," "C," or "A and B," "A and C," "B and C," or "A and B and C." Further, unless otherwise specified, the use of "a" or "b" or "one of a and b" can mean "a" or "b" or "a and b" unless otherwise specified.

[0325] As used in the present disclosure, unless specifically stated otherwise, the term "a" or "an" can include one or more items. Further, unless specifically stated otherwise, the phrase "based on" is intended to convey that something is "based, at least in part, on."< / tellmewhen> < / playmusic> < / playmusic> < / playmusic> < / tellmewhen> < / playmusic> < / tellmewhen> < / playmusic> < / tellmewhen> < / playmusic> < / tellmewhen> < / mute>

Claims

1. A computer-implemented method comprising: receiving user input data representing a natural language, the user input data being associated with a user profile; processing the user input data to determine intent data corresponding to the user input data, the intent data indicating a request to receive an output in the future when a triggering event occurs; determining trigger data using the user input data, the trigger data representing the triggering event that causes output data to be generated, the output data representing an action to be performed in the future when the triggering event occurs; determining a priority corresponding to the trigger data; and associating the trigger data and the priority with the user profile.

2. The computer-implemented method of claim 1, further comprising: determining a category corresponding to the triggering event; determining first priority data corresponding to the triggering event; determining second priority data represented in the user input data; and determining the priority using the first priority data and the second priority data.

3. The computer-implemented method of claim 1, further comprising: determining that the priority is high; determining that the output is an announcement using the priority; determining that an output capability of an output device includes the capability to output synthesized speech; generating output data representing a confirmation that an announcement will be generated when the triggering event occurs; processing the output data using text-to-speech (TTS) processing to determine output audio data; and sending the output audio data to the output device.

4. The computer-implemented method of claim 1, further comprising: determining that the priority is high; determining that a first output device is not capable of outputting synthesized speech; determining to output a push notification to the first output device and a second output device using the priority; generating output data representing a confirmation that a notification will be generated when the triggering event occurs; and sending the output data to the first output device.

5. The computer-implemented method of claim 1, further comprising: determining a trigger type associated with the trigger data, the trigger type indicating a time when the triggering event will occur; associating the trigger type and the time with the trigger data; generating output data representing a confirmation that a reminder of the triggering event occurring will be generated; processing the output data using TTS processing to determine output audio data; and sending the output audio data to a device.

6. The computer-implemented method of claim 1, further comprising: determining that the priority is low; determining that the output is a push notification using the priority; generating output data representing a confirmation that a notification will be generated when the triggering event occurs; and sending the output data to a device.

7. The computer-implemented method of any one of claims 1 to 6, further comprising: receiving audio data corresponding to the user input data; ​ ​ ​ ​ ​ ​ processing the audio data using automatic speech recognition (ASR) to determine input text data; processing the input text data using natural language understanding (NLU) to determine frequency data indicative of a frequency of receiving the output; associating the frequency data with the trigger data; and during a time period after receiving the audio data: receiving trigger event data, determining, using the trigger data, that the trigger event data triggers an output with respect to the user profile, determining, using the priority, output data, sending the output data to a device, and determining, based on the frequency data, to delete the trigger data.

8. The computer-implemented method of any of claims 1-6, further comprising: receiving audio data corresponding to the user input data; processing the audio data using automatic speech recognition (ASR) to determine input text data; processing the input text data using natural language understanding (NLU) to determine first frequency data indicative of a number of times to generate the output; associating the first frequency data with the trigger data; and during a time period after receiving the audio data: receiving trigger event data, determining, using the trigger data, that the trigger event data triggers a first output with respect to the user profile, determining second frequency data indicative of a remaining number of times to generate the output, and associating the second frequency data with the trigger data.

9. The computer-implemented method of any of claims 1-6, further comprising: determining, using the priority, that the output is a notification; determining that an output device includes a capability to output notification data; determining that receipt of notification data is disabled at the output device; generating output data representing a request to enable receipt of notification data at the output device; and sending the output data to the output device.

10. A system comprising: at least one processor; and at least one memory including instructions that, when executed by the at least one processor, cause the system to: receive user input data representing natural language, the user input data being associated with a user profile; process the user input data to determine intent data corresponding to the user input data, the intent data indicating a request to receive an output in the future when a trigger event occurs; determine, using the user input data, trigger data representing a trigger event that causes generation of output data, the output data representing an action to be performed in the future when the trigger event occurs; determine a priority corresponding to the trigger data; and associate the trigger data and the priority with the user profile.

11. The system of claim 10, wherein the instructions, when executed by the at least one processor, further cause the system to: determine that the priority is high; determine, using the priority, that the output is an announcement; determine that an output capability of an output device includes a capability to output synthesized speech; and determine, using the priority, that the output is a notification. ​ ​ generating output data representing a confirmation that a notification will be generated when the trigger event occurs; processing the output data using text-to-speech (TTS) processing to determine output audio data; and sending the output audio data to the output device.

12. The system of claim 10, wherein the instructions, when executed by the at least one processor, further cause the system to: determine that the priority is high; determine that a first output device is unable to output synthesized speech; determine, using the priority, to output a push notification to the first output device and a second output device associated with the user profile; generate output data representing a confirmation that a notification will be generated when the trigger event occurs; and send the output data to the first output device.

13. The system of claim 10, wherein the instructions, when executed by the at least one processor, further cause the system to: determine a trigger type associated with the trigger data, the trigger type indicating a time when the trigger event will occur; associate the trigger type and the time with the trigger data; generate output data representing a confirmation that a reminder of the occurrence of the trigger event will be generated; process the output data using TTS processing to determine output audio data; and send the output audio data to a device.

14. The system of claim 10, wherein the instructions, when executed by the at least one processor, further cause the system to: determine that the priority is low; determine, using the priority, that the output is a push notification; generate output data representing a confirmation that a notification will be generated when the trigger event occurs; and send the output data to a device.

15. The system of any of claims 10 to 14, wherein the instructions, when executed by the at least one processor, further cause the system to: receive audio data corresponding to the user input data; process the audio data using automatic speech recognition (ASR) to determine input text data; process the input text data using natural language understanding (NLU) to determine frequency data indicating an intent to receive the output once; associate the frequency data with the trigger data; and during a time period after receiving the audio data: receive trigger event data, determine, using the trigger data, that the trigger event data triggers output with respect to the user profile, determine, using the priority, output data, send the output data to a device, and determine, based on the frequency data, to delete the trigger data and the priority.

Images