Application Integration with Digital Assistants

Abstract

To provide a system and process of application integration with a digital assistant.SOLUTION: For example, the method includes: receiving an audio input including a natural language user input; and specifying an intent object in an intent object set. The intent object can be derived from the natural language user input. The method further includes; specifying a software application associated with the intent object in the intent object set; providing the intent object to the software application to cause the software application to execute a task associated with the intent object; receiving a result response indicating whether or not the execution of the task has been succeeded; and providing an output indicating whether or not the task has been executed.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present disclosure relates generally to interfacing with applications, and more particularly to application integration techniques with digital assistants. [Background technology]

[0002] Digital assistants can help users perform various functions on their devices. For example, digital assistants can set alarms, provide weather updates, and perform searches both locally and over the Internet, all while providing a natural language interface to users. However, existing digital assistants cannot effectively integrate with third-party applications, particularly those stored locally on the user device. Therefore, existing digital assistants cannot provide a natural language interface with such applications. Summary of the Invention

[0003] Exemplary methods are disclosed herein. One example method includes, at an electronic device having one or more processors and a touch-sensitive display, receiving an audio input including natural language user input, identifying an intent object from a set of intent objects, the intent object derived from the natural language user input, identifying a software application associated with the intent object from the set of intent objects, providing the intent object to the software application to cause the software application to perform a task associated with the intent object, receiving a result response from the software application indicating whether the task was successfully performed, and providing an output indicating whether the task was performed.

[0004] One example method includes, at one or more electronic devices each having one or more processors, receiving audio input including natural language user input, obtaining a string from the natural language user input, determining an intent object from a set of intent objects based on the string, determining a software application associated with the intent object from the set of intent objects, providing the intent object to the software application to cause the software application to perform a task associated with the intent object, receiving a result response from the software application indicating whether the task was successfully performed, and providing an output indicating whether the task was performed.

[0005] One example method includes, at an electronic device having one or more processors, receiving natural language user input; identifying a software application associated with an intent object from a set of intent objects; receiving a task flow associated with the software application from a second electronic device; providing the intent object to the software application to cause the software application to perform a task associated with the intent object; receiving a request from the software application; determining a query based on the request, the query determined according to the task flow; providing the query using a touch-sensitive display; after providing the query, receiving a second user input indicating a query response; and providing the query response to the software application.

[0006] An example method includes, at one or more electronic devices each having one or more processors, determining an intent object from a set of intent objects, identifying a software application associated with the intent object from the set of intent objects, determining a task flow associated with the software application, providing the intent object to the software application to cause the software application to perform a task associated with the intent object, receiving a request from the software application, determining a query based on the request, the query determined according to the task flow, providing the query using a touch-sensitive display, receiving a second user input indicating a query response after providing the query, and providing the query response to the software application.

[0007] An exemplary non-transitory computer-readable medium is disclosed herein. An exemplary non-transitory computer-readable storage medium stores one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to receive audio input including natural language user input, identify an intent object from a set of intent objects, the intent object derived from the natural language user input, identify a software application associated with the intent object from the set of intent objects, provide the intent object to the software application, cause the software application to perform a task associated with the intent object, receive a result response from the software application indicating whether execution of the task was successful, and provide an output indicating whether the task was performed.

[0008] An example non-transitory computer-readable storage medium stores one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to receive audio input including natural language user input, obtain a string from the natural language user input, determine an intent object from a set of intent objects based on the string, determine a software application associated with the intent object from the set of intent objects, provide the intent object to the software application, cause the software application to perform a task associated with the intent object, receive a result response from the software application indicating whether execution of the task was successful, and provide an output indicating whether the task was performed.

[0009] An example non-transitory computer-readable storage medium stores one or more programs comprising instructions that, when executed by one or more processors of one or more electronic devices, cause the one or more electronic devices to receive natural language user input, identify a software application associated with an intent object from a set of intent objects, receive a task flow associated with the software application from a second electronic device, provide the intent object to the software application to cause the software application to perform a task associated with the intent object, receive a request from the software application, determine a query based on the request, the query determined according to the task flow, provide the query using a touch-sensitive display, receive a second user input indicating a query response after providing the query, and provide the query response to the software application.

[0010] An example non-transitory computer-readable storage medium stores one or more programs comprising instructions that, when executed by one or more processors of one or more electronic devices, cause the one or more electronic devices to determine an intent object from a set of intent objects, identify a software application associated with an intent object from the set of intent objects, determine a task flow associated with the software application, provide the intent object to the software application to cause the software application to perform a task associated with the intent object, receive a request from the software application, determine a query based on the request, the query determined according to the task flow, provide the query using a touch-sensitive display, receive a second user input indicating a query response after providing the query, and provide the query response to the software application.

[0011] Exemplary electronic devices and systems are disclosed herein. An exemplary electronic device includes one or more processors, a memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for receiving audio input including natural language user input, identifying an Intent Object from a set of Intent Objects, the Intent Object derived from the natural language user input, identifying a software application associated with the Intent Object from the set of Intent Objects, providing the Intent Object to the software application to cause the software application to perform a task associated with the Intent Object, receiving a result response from the software application indicating whether the task was successfully performed, and providing an output indicating whether the task was performed.

[0012] An example system includes one or more processors of one or more electronic devices, one or more memories of the one or more electronic devices, and one or more programs stored in the one or more memories and configured to be executed by the one or more processors, the one or more programs including instructions to receive audio input including natural language user input, obtain a string from the natural language user input, determine an intent object from a set of intent objects based on the string, determine a software application associated with the intent object from the set of intent objects, provide the intent object to the software application, cause the software application to perform a task associated with the intent object, receive a result response from the software application indicating whether execution of the task was successful, and provide an output indicating whether the task was performed.

[0013] An example electronic device includes one or more processors, a memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions to receive natural language user input, identify a software application associated with an intent object from a set of intent objects, receive a task flow associated with the software application from a second electronic device, provide the intent object to the software application to cause the software application to perform a task associated with the intent object, receive a request from the software application, determine a query based on the request, the query determined according to the task flow, provide the query using a touch-sensitive display, receive a second user input indicating a query response after providing the query, and provide the query response to the software application.

[0014] An example system includes one or more processors of one or more electronic devices, one or more memories of the one or more electronic devices, and one or more programs stored in the one or more memories and configured to be executed by the one or more processors, the one or more programs including instructions to determine an intent object from a set of intent objects, identify a software application associated with an intent object from the set of intent objects, determine a task flow associated with the software application, provide the intent object to the software application to cause the software application to perform a task associated with the intent object, receive a request from the software application, determine a query based on the request, the query determined according to the task flow, provide the query using a touch-sensitive display, receive a second user input indicating a query response after providing the query, and provide the query response to the software application.

[0015] An example electronic device comprises means for receiving audio input including natural language user input; means for identifying an intent object from a set of intent objects, the intent object being derived from the natural language user input; means for identifying a software application associated with the intent object from the set of intent objects; means for providing the intent object to the software application to cause the software application to perform a task associated with the intent object; means for receiving a result response from the software application indicating whether the task was successfully performed; and means for providing an output indicating whether the task was performed.

[0016] An example system includes means for receiving audio input including natural language user input, means for obtaining a string from the natural language user input, means for determining an intent object from a set of intent objects based on the string, means for determining a software application associated with an intent object from the set of intent objects, means for providing the intent object to the software application to cause the software application to perform a task associated with the intent object, means for receiving a result response from the software application indicating whether the task was successfully performed, and means for providing an output indicating whether the task was performed.

[0017] An example electronic device comprises means for receiving natural language user input, means for identifying a software application associated with an intent object from a set of intent objects, means for receiving a task flow associated with the software application from a second electronic device, means for providing the intent object to the software application to cause the software application to perform a task associated with the intent object, means for receiving a request from the software application, means for determining a query based on the request, the query being determined according to the task flow, means for providing the query using a touch-sensitive display, means for receiving a second user input indicating a query response after providing the query, and means for providing the query response to the software application.

[0018] An example system includes means for determining an intent object from a set of intent objects, means for identifying a software application associated with an intent object from the set of intent objects, means for determining a task flow associated with the software application, means for providing an intent object to the software application to cause the software application to perform a task associated with the intent object, means for receiving a request from the software application, means for determining a query based on the request, the query being determined according to the task flow, means for providing the query using a touch-sensitive display, means for receiving a second user input indicating a query response after providing the query, and means for providing the query response to the software application. For a better understanding of the various embodiments described, reference should be made to the following detailed description in conjunction with the following drawings, in which like reference characters refer to corresponding parts throughout. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a block diagram illustrating a system and environment for implementing a digital assistant according to various embodiments. [Figure 2A] FIG. 1 is a block diagram illustrating a portable multifunction device that implements a client-side portion of a digital assistant in accordance with some embodiments. [Figure 2B] FIG. 2 is a block diagram illustrating example components for event processing in accordance with various embodiments. [Figure 3] FIG. 1 illustrates a portable multifunction device that implements a client-side portion of a digital assistant according to various embodiments. [Figure 4] FIG. 1 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface in accordance with various embodiments. [Figure 5A] 1A-1C illustrate exemplary user interfaces for application menus of portable multifunction devices in accordance with various embodiments. [Figure 5B] 1A-1C illustrate exemplary user interfaces for a multifunction device with a touch-sensitive surface separated from a display in accordance with various embodiments. [Figure 6A] FIG. 1 illustrates a personal electronic device according to various embodiments. [Figure 6B] FIG. 1 is a block diagram illustrating a personal electronic device according to various embodiments. [Figure 7A] FIG. 1 is a block diagram illustrating a digital assistant system or its server portion according to various embodiments. [Figure 7B] FIG. 7B illustrates the functionality of the digital assistant shown in FIG. 7A in accordance with various embodiments. [Figure 7C] FIG. 1 illustrates a portion of an ontology according to various embodiments. [Figure 8] FIG. 1 is a flow diagram illustrating a process for operating a digital assistant in accordance with some embodiments. [Figure 9] FIG. 1 is a flow diagram illustrating a process for operating a digital assistant in accordance with some embodiments. [Figure 10A] FIG. 1 illustrates an exemplary user interface of an electronic device in accordance with some embodiments. [Figure 10B] FIG. 1 illustrates an exemplary user interface of an electronic device in accordance with some embodiments. [Figure 10C] FIG. 1 illustrates an exemplary user interface of an electronic device in accordance with some embodiments. [Figure 10D] FIG. 1 illustrates an example data flow for a digital assistant system according to some embodiments. [Figure 10E] FIG. 1 illustrates an example data flow for a digital assistant system according to some embodiments. [Figure 11] FIG. 1 is a flow diagram illustrating a process for operating a digital assistant in accordance with some embodiments. [Figure 12] FIG. 1 is a flow diagram illustrating a process for operating a digital assistant in accordance with some embodiments. [Figure 13A] FIG. 1 illustrates an exemplary user interface of an electronic device in accordance with some embodiments. [Figure 13B] FIG. 1 illustrates an exemplary user interface of an electronic device in accordance with some embodiments. [Figure 14] FIG. 1 is a functional block diagram of an electronic device in accordance with some embodiments. [Figure 15] FIG. 1 is a functional block diagram of an electronic device in accordance with some embodiments. [Figure 16] FIG. 1 is a functional block diagram of an electronic device in accordance with some embodiments. [Figure 17] FIG. 1 is a functional block diagram of an electronic device in accordance with some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0020] In the following description of the disclosure and embodiments, reference is made to the accompanying drawings which show, by way of illustration, specific embodiments which may be practiced, although it should be understood that other embodiments and examples may be practiced without departing from the scope of the disclosure.

[0021] In the following description, terms such as "first" and "second" are used to describe various elements, but these elements should not be limited by these terms. These terms are used merely to distinguish between elements. For example, a first input can be referred to as a second input, and similarly, a second input can be referred to as a first input, without departing from the scope of the various embodiments described. The first input and the second input may both be outputs, and in some cases, may be separate and distinct inputs.

[0022] The terms used in the description of various embodiments herein are merely for the purpose of describing particular embodiments and are not intended to limit the present invention. When used in the description of various embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural unless expressly stated otherwise. Also, as used herein, the term "and / or" is understood to refer to and encompass any or all possible combinations of one or more of the associated listed items. Furthermore, as used herein, the terms "comprises," "including," "comprises," and / or "comprising" are understood to specify the presence of stated features, integers, steps, operations, elements, and / or components, but not to exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0023] The word "if" can be interpreted to mean "when" or "when," or "upon determining" or "upon detecting," depending on the context. Similarly, the phrase "when determined" or "when [described condition or event] is detected" can be interpreted to mean "upon determining" or "upon determining," or "upon detecting [described condition or event]," or "upon detecting [described condition or event]," depending on the context. 1. System and Environment

[0024] FIG. 1 is a block diagram of a system 100 according to various embodiments. In some examples, the system 100 may implement a digital assistant. The terms “digital assistant,” “virtual assistant,” “intelligent automated assistant,” or “automated digital assistant” may refer to any information processing system that interprets spoken and / or written natural language input to infer a user's intent and performs an action (e.g., a task) based on the inferred user's intent. For example, to operate based on the inferred user's intent, the system may perform one or more of: identifying a task flow with steps and parameters designed to implement the inferred user's intent; inputting specific requests from the inferred user's intent into the task flow; executing the task flow by invoking programs, methods, services, APIs, etc.; and generating an output response to the user in an audible (e.g., speech) and / or visual form.

[0025] Specifically, a digital assistant can accept user requests, at least partially in the form of natural language commands, requests, statements, speech, and / or inquiries. Typically, a user request can request an informational answer or task performance by the digital assistant. A satisfactory response to a user request can be providing the requested informational answer, performing the requested task, or a combination of both. For example, a user can ask a digital assistant a question such as, "Where am I now?" Based on the user's current location, the digital assistant can respond, "You're in Central Park near the Westgate." The user can also request the performance of a task, such as, "Invite my friends to my girlfriend's birthday party next week." In response, the digital assistant can fulfill the request by saying, "Yes, I'm back," and sending appropriate date invites on behalf of the user to each of the user's friends in the user's electronic address book. During the performance of a requested task, the digital assistant may interact with the user in a continuous conversation involving multiple information exchanges over an extended period of time. There are many other ways to interact with a digital assistant to request information or perform various tasks. In addition to providing verbal responses and performing programmed actions, digital assistants can also provide responses in other visible or audible forms, such as text, alerts, music, visuals, videos, etc.

[0026] 1 , in some embodiments, a digital assistant can be implemented according to a client-server model. The digital assistant can include a client-side portion 102 (hereinafter referred to as a “DA client 102”) that executes on a user device 104 and a server-side portion 106 (hereinafter referred to as a “DA server 106”) that executes on a server system 108. The DA client 102 can communicate with the DA server 106 over one or more networks 110. The DA client 102 can provide client-side functionality such as user-responsive input and output processing functions and communication with the DA server 106. The DA server 106 can provide server-side functionality for any number of DA clients 102 that reside on their respective user devices 104.

[0027] In some embodiments, the DA server 106 can include a client-facing I / O interface 112, one or more processing modules 114, data and models 116, and an I / O interface with an external service 118. The client-facing I / O interface 112 can facilitate client-facing input and output processing of the DA server 106. The one or more processing modules 114 can utilize the data and models 116 to process speech input and define user intent based on natural language input. Furthermore, the one or more processing modules 114 can perform tasks based on the inferred user intent. In some embodiments, the DA server 106 can communicate with an external service 120 over the network 110 to complete a task or obtain information. The I / O interface with the external service 118 can facilitate such communication.

[0028] User device 104 may be any suitable electronic device. For example, user device 104 may be a portable multifunction device (e.g., device 200, described below with reference to FIG. 2A ), a multifunction device (e.g., device 400, described below with reference to FIG. 4 ), or a personal electronic device (e.g., device 600, described below with reference to FIGS. 6A and 6B ). A portable multifunction device may be, for example, a mobile phone that includes other functions, such as PDA and / or music player functionality. Specific examples of portable multifunction devices include the iPhone®, iPod Touch®, and iPad® manufactured by Apple Inc. of Cupertino, California. Other examples of portable multifunction devices include, but are not limited to, laptops or tablet computers. Furthermore, in some embodiments, user device 104 may be a non-portable multifunction device. In particular, user device 104 may be a desktop computer, a game console, a television, or a television set-top box. In some embodiments, user device 104 may include a touch-sensitive surface (e.g., a touchscreen display and / or a touchpad). Additionally, the user device 104 may optionally include one or more other physical user interface devices, such as a physical keyboard, a mouse, and / or a joystick. Various embodiments of electronic devices, such as multifunction devices, are described in more detail below.

[0029] Examples of communication network 110 include local area networks (LANs) and wide area networks (WANs), such as the Internet, etc. Communication network 110 may be implemented using any known network protocol, including various wired or wireless protocols, such as Ethernet, Universal Serial Bus (USB), Firewire, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0030] The server system 108 may be implemented on one or more standalone data processing devices or a distributed network of computers. In some embodiments, the server system 108 may also employ various virtual devices and / or services of third-party service providers (e.g., third-party cloud service providers) to provide the underlying computing and / or infrastructure resources of the server system 108.

[0031] In some embodiments, the user device 104 can communicate with the DA server 106 via the second user device 122. The second user device 122 can be similar to or identical to the user device 104. For example, the second user device 122 can be similar to devices 200, 400, or 600 described below with reference to FIGS. 2A, 4, 6A, and 6B. The user device 104 can be configured to be communicatively coupled to the second user device 122 via a direct communication connection, such as Bluetooth, NFC, or BTLE, or a wired or wireless network, such as a local Wi-Fi network. In some embodiments, the second user device 122 can be configured to act as a proxy between the user device 104 and the DA server 106. For example, the DA client 102 of the user device 104 can be configured to send information (e.g., a user request received at the user device 104) to the DA server 106 via the second user device 122. The DA server 106 can process the information and transmit relevant data (eg, data content responsive to the user request) back to the user device 104 via the second user device 122 .

[0032] In some embodiments, the user device 104 can be configured to communicate a request for shortening the data to the second user device 122 to reduce the amount of information transmitted from the user device 104. The second user device 122 can be configured to determine and add supplemental information to the shortened request, and generate and transmit a full request to the DA server 106. This system architecture allows a user device 104 with limited communication capabilities and / or limited battery power (e.g., a wristwatch or similar compact electronic device) to conveniently access services provided by the DA server 106 by using the second user device 122 with greater communication capabilities and / or battery power (e.g., a mobile phone, laptop computer, tablet computer, etc.) instead of the DA server 106. While only two user devices 104 and 122 are shown in FIG. 1 , it should be understood that the system 100 can include any number and type of user devices configured in this proxy structure to communicate with the DA server system 106.

[0033] 1 can include both a client-side portion (e.g., DA client 102) and a server-side portion (e.g., DA server 106), but in some embodiments, the digital assistant's functionality can be implemented as a standalone application installed on a user device. Also, the division of functionality between the client and server portions of the digital assistant can vary in various implementations. For example, in some embodiments, the DA client can be a thin client that provides only user-facing input and output processing functionality and offloads all other digital assistant functionality to a back-end server. 2. Electronic Devices

[0034] Attention now turns to embodiments of electronic devices for implementing the client-side portion of a digital assistant. FIG. 2A is a block diagram illustrating a portable multifunction device 200 with a touch-sensitive display system 212 according to some embodiments. Touch-sensitive display 212 may conveniently be referred to as a "touch screen" or sometimes known or referred to as a "touch-sensitive display system." Device 200 includes memory 202 (optionally including one or more computer-readable storage media), a memory controller 222, one or more processing units (CPUs) 220, a peripherals interface 218, RF circuitry 208, audio circuitry 210, a speaker 211, a microphone 213, an input / output (I / O) subsystem 206, other input control devices 216, and an external port 224. Device 200 optionally includes one or more optical sensors 264. Device 200 optionally includes one or more contact intensity sensors 265 for detecting contact intensity on device 200 (e.g., a touch-sensitive surface such as touch-sensitive display system 212 of device 200). Device 200 optionally includes one or more tactile output generators 267 for generating tactile output on device 200 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 212 of device 200 or touchpad 455 of device 400). These components optionally communicate over one or more communication buses or signal lines 203.

[0035] As used in this specification and in the claims, the term “intensity” of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or a surrogate (proxy) for the force or pressure of a contact on the touch-sensitive surface. Contact intensity has a range of values that includes at least four distinct values and more typically includes hundreds of distinct values (e.g., at least 256). Contact intensity is optionally determined (or measured) using various techniques and various sensors or combinations of sensors. For example, one or more force sensors under or adjacent to the touch-sensitive surface are optionally used to measure force at various points on the touch-sensitive surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted average) to determine an estimated contact force. Similarly, a pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and / or change in the contact area detected on the touch-sensitive surface, and / or the resistance and / or change in the touch-sensitive surface in the vicinity of the contact, are optionally used as a surrogate for contact force or pressure on the touch-sensitive surface. In some embodiments, the surrogate measure of contact force or pressure is used to directly determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the surrogate measure). In some embodiments, the surrogate measure of contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using contact intensity as an attribute of user input allows user access to additional device functionality that may otherwise be inaccessible on small devices with limited area for displaying affordances (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via the touch-sensitive display, touch-sensitive surface, or physical / mechanical controls such as knobs or buttons).

[0036] As used herein and in the claims, the term “tactile output” refers to the physical displacement of a device relative to a previous position of the device, the physical displacement of one component (e.g., a touch-sensitive surface) relative to another component (e.g., a housing) of the device, or the displacement of a component relative to the center of gravity of the device that is detected by a user using the user's sense of touch. For example, in a situation where a device or a component of a device is in contact with a touch-sensitive surface of a user (e.g., a finger, palm, or other part of the user's hand), the tactile output produced by the physical displacement is interpreted as a tactile sensation corresponding to a perceived change in a physical property of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a “downclick” or “upclick” of a physical actuator button. In some cases, a tactile sensation such as a “downclick” or “upclick” can be felt even when there is no movement of a physical actuator button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement. As another example, movement of the touch-sensitive surface is optionally interpreted or perceived by a user as "roughness" of the touch-sensitive surface, even when there is no change in the smoothness of the touch-sensitive surface. While such interpretation of touch by a user is influenced by the user's individual perception, there are many perceptions of touch that are common to the majority of users. Thus, when a tactile output is described as corresponding to a particular perception of a user (e.g., "upclick," "downclick," "roughness"), unless otherwise specified, the tactile output generated corresponds to a physical displacement of the device or its components and can produce the described perception for a typical (average) user.

[0037] It will be appreciated that device 200 is merely one example of a portable multifunction device, and that device 200 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of components. The various components shown in Figure 2A may be implemented in hardware, including one or more signal processing and / or application specific integrated circuits, software, or a combination of both hardware and software.

[0038] Memory 202 may include one or more computer-readable storage media. The computer-readable storage media may be tangible or non-transitory. Memory 202 may include high-speed random access memory or may include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 222 may control access to memory 202 by other components of device 200.

[0039] In some examples, the non-transitory computer-readable storage medium of memory 202 may be used to store instructions (e.g., for execution of process 1100 described below) for use by or in conjunction with an instruction execution system, apparatus, or device, such as a computer-based system, processor storage system, or other system capable of retrieving and executing instructions from an instruction execution system, apparatus, or device. In other examples, the instructions (e.g., for execution of process 1100 described below) may be stored in a non-transitory computer-readable storage medium (not shown) of server system 108, or may be split between the non-transitory computer-readable medium of memory 202 and the non-transitory computer-readable storage medium of server system 108. In the context of this document, a "non-transitory computer-readable storage medium" may be any medium that can store or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0040] A peripheral interface 218 can be used to couple input / output peripherals of the device to the CPU 220 and memory 202. The one or more processors 220 run or execute various software programs and / or instruction sets stored in memory 202 to perform various functions and process data for the device 200. In some embodiments, the peripheral interface 218, CPU 220, and memory controller 222 can be implemented on a single chip, such as chip 204. In some other embodiments, they may be implemented on separate chips.

[0041] RF (radio frequency) circuitry 208 transmits and receives RF signals, also known as electromagnetic signals. RF circuitry 208 converts between electrical and electromagnetic signals and communicates with communication networks and other communication devices via electromagnetic signals. RF circuitry 208 optionally includes known circuitry for performing these functions, including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, etc. RF circuitry 208 optionally communicates with networks such as the Internet, also known as the World Wide Web (WWW), an intranet, and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs) and / or metropolitan area networks (MANs), and other devices via wireless communication. RF circuitry 208 optionally includes known circuitry for detecting near field communication (NFC) fields, such as by short-range communication radios.Wireless communication optionally uses any of a number of communication standards, protocols, and technologies, including Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Evolution Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Event Package (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

[0042] The audio circuit 210, speaker 211, and microphone 213 provide an audio interface between a user and device 200. Audio circuit 210 receives audio data from peripherals interface 218, converts the audio data into electrical signals, and transmits the electrical signals to speaker 211. Speaker 211 converts the electrical signals into sound waves that humans can hear. Audio circuit 210 also receives electrical signals converted from the sound waves by microphone 213. Audio circuit 210 converts the electrical signals into audio data and transmits the audio data to peripherals interface 218 for processing. The audio data may be retrieved from or transmitted to memory 202 and / or RF circuit 208 by peripherals interface 218. In some embodiments, audio circuit 210 also includes a headset jack (e.g., 312 in FIG. 3 ). The headset jack provides an interface between audio circuitry 210 and a detachable audio input / output peripheral such as an output-only headphone or a headset with both an output (e.g., single or double ear headphones) and an input (e.g., a microphone).

[0043] I / O subsystem 206 couples input / output peripherals on device 200, such as touchscreen 212 and other input control devices 216, to peripheral interface 218. I / O subsystem 206 optionally includes one or more input controllers 260 for display controller 256, light sensor controller 258, intensity sensor controller 259, haptic feedback controller 261, and other input or control devices. One or more input controllers 260 send and receive electrical signals to other input control devices 216. Other input control devices 216 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, input controller 260 optionally is coupled to (or may not be coupled to) any of a keyboard, infrared port, USB port, and pointer device such as a mouse. One or more buttons (e.g., 308 in FIG. 3 ) optionally include up / down buttons for volume control of speaker 211 and / or microphone 213. The one or more buttons optionally include a push button (eg, 306 in FIG. 3).

[0044] A quick press of the push button can unlock the touch screen 212 or initiate a process for using gestures on the touch screen to unlock the device, as described in U.S. Patent Application No. 11 / 322,549, entitled "Unlocking a Device by Making a Gesture on an Unlock Image," filed December 23, 2005, U.S. Patent No. 7,657,849, which is incorporated by reference in its entirety. A longer press of a push button (e.g., 306) can also power the device 200 on or off. The user can also customize the functionality of one or more buttons. The touch screen 212 can be used to implement virtual or soft buttons and one or more soft keyboards.

[0045] The touch-sensitive display 212 provides an input and output interface between the device and a user. The display controller 256 sends and receives electrical signals to and from the touchscreen 212. The touchscreen 212 provides visual output to the user. The visual output may include graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output may correspond to user interface objects.

[0046] Touchscreen 212 has a touch-sensitive surface, sensor, or set of sensors that receive input from a user based on haptic and / or tactile contact. Touchscreen 212 and display controller 256 (along with any associated modules and / or instruction sets in memory 202) detects contacts (and any movement or disruption of contact) on touchscreen 212 and translates the detected contacts into interactions with user interface objects (e.g., one or more softkeys, icons, web pages, or images) displayed on touchscreen 212. In one example embodiment, the point of contact between touchscreen 212 and the user corresponds to the user's finger.

[0047] Touchscreen 212 may use LCD (liquid crystal display), LPD (light emitting polymer display), or LED (light emitting diode) technology, although other display technologies may be used in other embodiments. Touchscreen 212 and display controller 256 can detect contact and any movement or disruption thereof and determine one or more points of contact with touchscreen 212 using any of a number of now known or later developed touch sensing technologies, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements. In one example embodiment, projected mutual capacitance sensing technology is used, such as that found in the iPhone® and iPod Touch® from Apple Inc. of Cupertino, California.

[0048] The touch-sensitive display of some embodiments of touchscreen 212 may be similar to the multi-touch-sensing touchpads described in the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman), and / or U.S. Patent Publication 2002 / 0015024 A1, each of which is incorporated by reference herein in its entirety. However, while touchscreen 212 displays visual output from device 200, touch-sensitive touchpads do not provide visual output.

[0049] The touch-sensitive display of some embodiments of touchscreen 212 may be any of the following: (1) U.S. patent application Ser. No. 11 / 381,313, entitled "Multipoint Touch Surface Controller," filed May 2, 2006; (2) U.S. patent application Ser. No. 10 / 840,862, entitled "Multipoint Touchscreen," filed May 6, 2004; (3) U.S. patent application Ser. No. 10 / 903,964, entitled "Gestures For Touch Sensitive Input Devices," filed July 30, 2004; (4) U.S. patent application Ser. No. 11 / 048,264, entitled "Gestures For Touch Sensitive Input Devices," filed January 31, 2005; and (5) U.S. patent application Ser. No. 11 / 038,590, entitled "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices." No. 11 / 228,758 entitled "Virtual Input Device Placement On A Touch Screen User Interface," filed September 16, 2005; (7) U.S. Patent Application No. 11 / 228,700 entitled "Operation Of A Computer With A Touch Screen Interface," filed September 16, 2005; (8) U.S. Patent Application No. 11 / 228,737 entitled "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard," filed September 16, 2005; and (9) U.S. Patent Application No. 11 / 367,749 entitled "Multi-Functional Hand-Held Device," filed March 3, 2006. All of these applications are incorporated herein by reference in their entirety.

[0050] Touchscreen 212 has a video resolution greater than 100 dpi (approximately 3937 dots / m). In some embodiments, the touchscreen has a video resolution of approximately 160 dpi (approximately 6299 dots / m). A user may contact touchscreen 212 with any suitable object or attachment, such as a stylus, finger, etc. In some embodiments, the user interface is designed primarily to work with finger-based contacts and gestures, which may be less precise than stylus-based input due to the larger contact area of a finger on the touchscreen. In some embodiments, the coarse finger-based input is translated into a precise pointer / cursor position or command to perform the action desired by the user.

[0051] In some embodiments, in addition to a touchscreen, device 200 may include a touchpad (not shown) for activating or deactivating certain functions. In some embodiments, a touchpad is a touch-sensitive area of a device that, unlike the touchscreen, does not display visual output. The touchpad may be a touch-sensitive surface that is separate from touchscreen 212 or may be an extension of the touch-sensitive surface formed by the touchscreen.

[0052] Device 200 also includes a power system 262 that powers the various components. Power system 262 may include a power management system, one or more power sources (e.g., battery, alternating current (AC)), a charging system, power fault detection circuitry, power converters or inverters, power status indicators (e.g., light emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power within a portable device.

[0053] Device 200 may include one or more light sensors 264. FIG. 2A shows a light sensor coupled to light sensor controller 258 in I / O subsystem 206. Light sensor 264 may include a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. Light sensor 264 receives light from the environment, projects it through one or more lenses, and converts the light into data representing an image. In conjunction with imaging module 243 (also called a camera module), light sensor 264 can capture still images or video. In some embodiments, the light sensor is located on the back of device 200, opposite touchscreen display 212 on the front of the device, allowing the touchscreen display to be used as a viewfinder for still and / or video image capture. In some embodiments, the light sensor is located on the front of the device, allowing an image of the user to be captured for a videoconference while the user observes other videoconference participants on the touchscreen display. In some embodiments, the position of the light sensor 264 is user-changeable (by rotating the lens and sensor within the device housing), allowing the light sensor 264 to be used in conjunction with a touchscreen display for both video conferencing and still and / or video image capture.

[0054] Device 200 also optionally includes one or more contact intensity sensors 265. FIG. 2A shows a contact intensity sensor coupled to intensity sensor controller 259 in I / O subsystem 206. Contact intensity sensor 265 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of a contact on a touch-sensitive surface). Contact intensity sensor 265 receives contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is coupled to or proximate to the touch-sensitive surface (e.g., touch-sensitive display system 212). In some embodiments, at least one contact intensity sensor is located on the back of device 200 opposite touchscreen display 212, which is located on the front of device 200.

[0055] Device 200 may include one or more proximity sensors 266. Figure 2A shows proximity sensor 266 coupled to peripheral interface 218. Alternatively, proximity sensor 266 can be coupled to input controller 260 within I / O subsystem 206. Proximity sensor 266 functions as described in U.S. patent application Ser. Nos. 11 / 241,839, entitled "Proximity Detector In Handheld Device," 11 / 240,788, entitled "Proximity Detector In Handheld Device," 11 / 620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output," 11 / 586,862, entitled "Automated Response To And Sensing Of User Activity In Portable Devices," and 11 / 638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals," all of which are incorporated by reference in their entireties. In some embodiments, the proximity sensor turns off and disables touchscreen 212 when the multifunction device is placed near the user's ear (eg, when the user makes a phone call).

[0056] Device 200 also optionally includes one or more tactile output generators 267. FIG. 2A shows a tactile output generator coupled to haptic feedback controller 261 in I / O subsystem 206. Tactile output generator 267 optionally includes one or more electroacoustic devices, such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components that convert electrical signals into tactile output on the device). Contact intensity sensor 265 receives tactile feedback generation instructions from haptic feedback module 233 and generates a tactile output on device 200 that can be sensed by a user of device 200. In some embodiments, at least one tactile output generator is coupled to or proximate to a touch-sensitive surface (e.g., touch-sensitive display system 212) and, optionally, generates a tactile output by moving the touch-sensitive surface vertically (e.g., in and out of the surface of device 200) or horizontally (e.g., back and forth in the same plane as the surface of device 200). In some embodiments, at least one tactile output generator sensor is located on the back of device 200 opposite touchscreen display 212, which is located on the front of device 200.

[0057] Device 200 may include one or more accelerometers 268. FIG. 2A shows accelerometer 268 coupled to peripherals interface 218. Alternatively, accelerometer 268 can be coupled to input controller 260 in I / O subsystem 206. Accelerometer 268 can function as described in U.S. Patent Application Publication No. 20050190059, entitled "Acceleration-Based Theft Detection System for Portable Electronic Devices," and U.S. Patent Application Publication No. 20060017692, entitled "Method and Apparatus for Operating a Portable Device Based on an Accelerometer," both of which are incorporated herein by reference in their entireties. In some embodiments, information is displayed on a touchscreen display in portrait or landscape orientation based on analysis of data received from one or more accelerometers. Device 200 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown), in addition to accelerometer 268, to obtain information regarding the location and orientation (e.g., portrait or landscape) of device 200.

[0058] In some embodiments, software components stored in memory 202 include an operating system 226, a communications module (or instruction set) 228, a touch / motion module (or instruction set) 230, a graphics module (or instruction set) 232, a text input module (or instruction set) 234, a global positioning system (GPS) module (or instruction set) 235, a digital assistant client module 229, and applications (or instruction sets) 236. Additionally, memory 202 can store data and models, such as user data and models 231. Additionally, in some embodiments, memory 202 (FIG. 2A) or 470 (FIG. 4) stores device / global internal state 257, as shown in FIGS. 2A and 4. The device / global internal state 257 includes one or more of: active application state, which indicates which applications, if any, are currently active; display state, which indicates which applications, views, or other information are occupying various areas of the touchscreen display 212; sensor state, which includes information obtained from the device's various sensors and input control devices 216; and position information regarding the device's position and / or orientation.

[0059] Operating system 226 (e.g., an embedded operating system such as Darwin, RTXC, Linux, UNIX, OS X, iOS, Windows, or VxWorks) includes various software components and / or drivers that control and manage general system tasks (e.g., memory management, storage control, power management, etc.) and facilitate communication between the various hardware and software components.

[0060] Communications module 228 facilitates communication with other devices on one or more external ports 224 and includes various software components that process data received by RF circuitry 208 and / or external port 224. External port 224 (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) is adapted to couple to other devices directly or indirectly over a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, similar to, and / or compatible with the 30-pin connector used on iPod (registered trademark of Apple Inc.) devices.

[0061] Contact / motion module 230 optionally detects contact with touchscreen 212 (together with display controller 256) and other touch-sensing devices (e.g., a touchpad or physical click wheel). Contact / motion module 230 includes various software components that perform various operations related to detecting contact, such as determining whether contact has occurred (e.g., detecting a finger-down event), determining contact strength (e.g., contact force or pressure, or a surrogate for contact force or pressure), determining whether there is contact movement and tracking the movement across the touch-sensitive surface (e.g., detecting a one or more finger drag event), and determining whether contact has ended (e.g., detecting a finger-up event or contact cessation). Contact / motion module 230 receives contact data from the touch-sensitive surface. Determining movement of a contact point represented by the set of contact data optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These operations are optionally applied to a single contact (e.g., one finger touch) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, contact / motion module 230 and display controller 256 detect contacts on the touchpad.

[0062] In some embodiments, contact / motion module 230 uses one or more sets of intensity thresholds to determine whether an action has been taken by a user (e.g., whether a user has “clicked” on an icon). In some embodiments, at least a subset of the intensity thresholds are determined according to software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of a particular physical actuator and can be adjusted without modifying the physical hardware of device 200). For example, the mouse “click” threshold of a trackpad or touchscreen display can be set to any of a large range of predetermined thresholds without modifying the trackpad or touchscreen display hardware. Additionally, some embodiments provide a software setting to a device user to adjust one or more sets of intensity thresholds (e.g., by adjusting individual and / or multiple intensity thresholds on the fly using a system-level click “intensity” parameter).

[0063] Contact / motion module 230 optionally detects gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different movements, timing, and / or detected contact intensities). Thus, gestures are optionally detected by detecting particular contact patterns. For example, detecting a finger-tap gesture includes detecting a finger-down event, followed by a finger-lift (lift-off) event at the same location (or substantially the same location) as the finger-down event (e.g., at the location of an icon). As another example, a finger-swipe gesture on the touch-sensitive surface includes detecting a finger-down event, followed by a finger-drag event, followed by a finger-lift (lift-off) event.

[0064] Graphics module 232 includes various known software components that draw and display graphics on touchscreen 212 or other display, including components that modify the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user interface objects, including softkeys), digital images, video, animation, etc.

[0065] In some embodiments, graphics module 232 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. Graphics module 232 receives one or more codes specifying the graphics to be displayed, along with coordinate data and other graphic characteristic data, as needed, from an application or the like, and then generates and outputs screen image data to display controller 256.

[0066] The haptic feedback module 233 includes various software components that generate instructions used by the tactile output generator 267 to generate tactile outputs at one or more locations on the device 200 in response to user interaction with the device 200 .

[0067] The text input module 234 may be a component of the graphics module 232 and provides a soft keyboard for entering text into various applications (e.g., contacts 237, email 240, IM 241, browser 247, and any other application requiring text input).

[0068] The GPS module 235 determines the location of the device and provides this information for use in various applications (e.g., to the phone 238 for use in location-based dialing, to the camera 243 as picture / video metadata, and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0069] Digital assistant client module 229 can include various client-side digital assistant instructions and provide client-side functionality of the digital assistant. For example, digital assistant client module 229 can receive voice input (e.g., speech input), text input, touch input, and / or gesture input via various user interfaces of portable multifunction device 200 (e.g., microphone 213, accelerometer 268, touch-sensitive display system 212, light sensor 229, other input control devices 216, etc.). Digital assistant client module 229 can also provide output in audio (e.g., speech output), visual, and / or tactile form via various output interfaces of portable multifunction device 200 (e.g., speaker 211, touch-sensitive display system 212, tactile output generator 267, etc.). For example, output can be provided as voice, sound, alert, text message, menu, graphics, video, animation, vibration, and / or a combination of two or more of the above. In operation, the digital assistant client module 229 can communicate with the DA server 106 using the RF circuitry 208.

[0070] User data and models 231 can include various data related to a user (e.g., user-specific vocabulary data, user preference data, user-specific name pronunciations, data from a user's electronic address book, to-do lists, shopping lists, etc.) and can provide client-side functionality for the digital assistant. Additionally, user data and models 231 can include various models (e.g., speech recognition models, statistical language models, natural language processing models, ontologies, task flow models, service models, etc.) that process user input and determine user intent.

[0071] In some examples, digital assistant client module 229 can utilize various sensors, subsystems, and peripherals of portable multifunction device 200 to gather additional information from the environment surrounding portable multifunction device 200 and establish a context associated with the user, the current user interaction, and the current user input. In some examples, digital assistant client module 229 can provide context information or a subset thereof to user input to DA server 106 to help infer the user's intent. In some examples, the digital assistant can also use the context information to determine how to prepare and send output to the user. Context information can also be referred to as context data.

[0072] In some examples, the context information associated with a user input may include sensor information such as lighting, ambient noise, ambient temperature, images or videos of the surrounding environment, etc. In some examples, the context information may also include the physical state of the device, such as device orientation, device location, device temperature, power level, speed, acceleration, movement patterns, cellular signal strength, etc. In some examples, information related to the software state of the DA server 106, such as the running processes of the portable multifunction device 200, installed programs, past and current network activity, background services, error logs, resource usage, etc., may also be provided to the DA server 106 as context information associated with the user input.

[0073] In some examples, digital assistant client module 229 can selectively provide information (e.g., user data 231) stored on portable multifunction device 200 in response to a request from DA server 106. In some examples, digital assistant client module 229 can also elicit additional input from the user via a natural language dialog or other user interface upon request by DA server 106. Digital assistant client module 229 can pass the additional input to DA server 106 to assist DA server 106 in inferring the intent expressed in the user request and / or in realizing the user's intent.

[0074] A more detailed description of the digital assistant is provided below with reference to Figures 7A-C. It should be appreciated that the digital assistant client module 229 can include any number of sub-modules of the digital assistant module 726 described below.

[0075] The application 236 may include the following modules (or instruction sets), or a subset or superset thereof: Contacts module 237 (also called an address book or contact list) ●Telephone module 238 ●Videoconferencing module 239 ●Email client module 240 ● Instant Messaging (IM) Module 241 ●Workout Support Module 242 Camera module 243 for still and video images ●Image Management Module 244 ●Video player module ●Music player module Browser module 247 ●Calendar module 248 Widget module 249 (which may include one or more of a weather widget 249-1, a stock widget 249-2, a calculator widget 249-3, an alarm clock widget 249-4, a dictionary widget 249-5, and other widgets acquired by the user and a user-created widget 249-6). A widget creator module 250 for creating user-created widgets 249-6. ●Search module 251 ●Video and music player module 252 that combines a video player module and a music player module ●Memo Module 253 Map module 254 and / or ●Online Video Module 255

[0076] Examples of other applications 236 that may be stored in memory 202 include other word processing applications, other image editing applications, drawing applications, presentation applications, Java-enabled applications, encryption, digital rights management, voice recognition, and voice duplication.

[0077] The contacts module 237, along with the touch screen 212, the display controller 256, the contact / motion module 230, the graphics module 232, and the text input module 234, may be used to manage an address book or contact list (e.g., stored in the memory 202 or in the application internal state 292 of the contacts module 237 in memory 470), including adding names to the address book, removing names from the address book, associating names with phone numbers, email addresses, physical addresses or other information, associating names with images, categorizing and sorting names, providing phone numbers or email addresses, initiating and / or facilitating communication by telephone 238, videoconferencing module 239, email 240, or IM 241, etc.

[0078] Telephone module 238, along with RF circuitry 208, audio circuitry 210, speaker 211, microphone 213, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, may also be used to enter character sequences corresponding to telephone numbers, access one or more telephone numbers in contact module 237, modify entered telephone numbers, dial each telephone number, conduct a conversation, and disconnect or hang up when the conversation is complete. As noted above, wireless communication may use any of a number of communication standards, protocols, and technologies.

[0079] Along with RF circuitry 208, audio circuitry 210, speaker 211, microphone 213, touch screen 212, display controller 256, light sensor 264, light sensor controller 258, touch / motion module 230, graphics module 232, text input module 234, contact module 237, and telephone module 238, videoconferencing module 239 contains executable instructions for initiating, conducting, and terminating a videoconference between a user and one or more other participants in accordance with user commands.

[0080] The email client module 240, together with the RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, contains executable instructions for creating, sending, receiving, and managing emails in response to user commands. Together with the image management module 244, the email client module 240 greatly facilitates the creation and sending of emails with still or video images captured by the camera module 243.

[0081] Instant messaging module 241, together with RF circuitry 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, includes executable instructions for entering sequences of characters corresponding to instant messages, modifying previously entered characters, sending each instant message (e.g., using Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for telephone-based instant messaging, or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and displaying received instant messages. In some embodiments, sent and / or received instant messages may include graphics, photos, audio files, video files, and / or other attachments, as supported by MMS and / or Extended Messaging Service (EMS). As used herein, "instant messaging" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0082] Along with the RF circuitry 208, touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, GPS module 235, map module 254, and music player module, workout support module 242 contains executable instructions for creating workouts (e.g., with time, distance, and / or calorie burn goals), communicating with workout sensors (sports devices), receiving workout sensor data, calibrating sensors used to monitor workouts, selecting and playing music for workouts, and displaying, storing, and transmitting workout data.

[0083] Camera module 243, along with touch screen 212, display controller 256, light sensor 264, light sensor controller 258, contact / motion module 230, graphics module 232, and image management module 244, contains executable instructions to capture still images or video (including video streams), store them in memory 202, modify characteristics of the still images or video, or delete the still images or video from memory 202.

[0084] Image management module 244, along with touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, and camera module 243, includes executable instructions for arranging, modifying (e.g., editing), or manipulating, labeling, deleting, presenting (e.g., in a digital slide show or album), and storing still and / or video images.

[0085] The browser module 247, along with the RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, contains executable instructions for browsing the Internet according to user commands, including retrieving, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0086] Calendar module 248, along with RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, email client module 240, and browser module 247, contains executable instructions for creating, displaying, modifying, and storing calendars and calendar-related data (calendar entries, to-do lists, etc.) according to user commands.

[0087] Widget module 249, along with RF circuitry 208, touchscreen 212, display controller 256, touch / motion module 230, graphics module 232, text input module 234, and browser module 247, are mini-applications that can be downloaded and used by a user (e.g., weather widget 249-1, stocks widget 249-2, calculator widget 249-3, alarm clock widget 249-4, and dictionary widget 249-5) or created by a user (e.g., user-created widget 249-6). In some embodiments, a widget includes an HTML (HyperText Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, a widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widget).

[0088] The widget creator module 250, together with the RF circuitry 208, touch screen 212, display controller 256, touch / motion module 230, graphics module 232, text input module 234, and browser module 247, can be used by a user to create a widget (e.g., to insert a user-specified portion of a web page into the widget).

[0089] The search module 251, along with the touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, includes executable instructions to search memory 202 for text, music, audio, images, video, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user commands.

[0090] Video and music player module 252, along with touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, audio circuitry 210, speaker 211, RF circuitry 208, and browser module 247, includes executable instructions that enable a user to download and play music and other audio files stored in one or more file formats, such as MP3 or AAC files, and to display, present, or play videos (e.g., on touchscreen 212 or on an external display connected via external port 224). In some embodiments, device 200 optionally includes the functionality of an MP3 player, such as an iPod (a registered trademark of Apple Inc.).

[0091] Along with touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, notes module 253 contains executable instructions for creating and managing notes, to-do lists, etc. according to user commands.

[0092] Map module 254, along with RF circuitry 208, touch screen 212, display controller 256, touch / motion module 230, graphics module 232, text input module 234, GPS module 235, and browser module 247, may be used to receive, display, modify, and store maps and map-related data (e.g., driving directions, data on stores and other desired points at or near a particular location, and other location-based data) in accordance with user instructions.

[0093] Online video module 255, along with touchscreen 212, display controller 256, contact / motion module 230, graphics module 232, audio circuitry 210, speaker 211, RF circuitry 208, text input module 234, email client module 240, and browser module 247, contains instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play (on the touchscreen or on an external display connected via external port 224), send emails with links to particular online videos, or manage online videos in one or more file formats, such as H.264. In some embodiments, instant messaging module 241 is used to send links to particular online videos rather than email client module 240. Additional description of online video applications can be found in U.S. Provisional Patent Application No. 60 / 936,562, entitled "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," filed June 20, 2007, and U.S. Patent Application No. 11 / 968,067, entitled "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," filed December 31, 2007, the contents of which are incorporated by reference in their entireties.

[0094] The above-identified modules and applications each correspond to executable instruction sets that perform one or more of the functions and methods described herein (e.g., computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) need not be implemented as separate software programs, procedures, or modules; thus, various embodiments may combine or rearrange various subsets of these modules. For example, a video player module may be combined with a music player module into a single module (e.g., video and music player module 252 of FIG. 2A). In some embodiments, memory 202 may store a subset of the above-identified modules and data structures. Additionally, memory 202 may store additional modules and data structures not described above.

[0095] In some embodiments, device 200 is a device that performs operation of a predetermined set of functions on the device exclusively via a touchscreen and / or touchpad. By using the touchscreen and / or touchpad as the primary input control device for operation of device 200, the number of physical input control devices (push buttons, dials, etc.) on device 200 can be reduced.

[0096] The predetermined set of functions performed exclusively via the touchscreen and / or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 200 to a main menu, home menu, or root menu from any user interface displayed on device 200. In such embodiments, the touchpad can be used to implement a "menu button." In some other embodiments, the menu button is a physical push button or other physical input control device instead of a touchpad.

[0097] 2B is a block diagram illustrating example components for event processing according to some embodiments. In some embodiments, memory 202 (FIG. 2A) or 470 (FIG. 4) includes event sorter 270 (e.g., in operating system 226) and each application 236-1 (e.g., any of applications 237-251, 255, 480-490 described above).

[0098] Event sorter 270 receives event information and determines application 236-1 and application view 291 for application 236-1 to send the event information. Event sorter 270 includes event monitor 271 and event dispatcher module 274. In some embodiments, application 236-1 includes application internal state 292 that indicates the current application view that is displayed on touch-sensitive display 212 when the application is active or running. In some embodiments, device / global internal state 257 is used by event sorter 270 to determine which application is currently active, and application internal state 292 is used by event sorter 270 to determine which application view 291 to send the event information.

[0099] In some embodiments, application internal state 292 includes additional information such as one or more of resume information used when application 236-1 resumes execution, user interface state information indicating information to be displayed or prepared for display by application 236-1, state cues that allow the user to revert to a previous state or view of application 236-1, and redo / undo cues for previous actions taken by the user.

[0100] Event monitor 271 receives event information from peripherals interface 218. The event information includes information about sub-events (e.g., a user touch on touch-sensitive display 212 as part of a multi-touch gesture). Peripherals interface 218 transmits information it receives from I / O subsystem 206 or sensors such as proximity sensor 266, accelerator 268, and / or microphone 213 (via audio circuitry 210). Information that peripherals interface 218 receives from I / O subsystem 206 includes information it receives from touch-sensitive display 212 or a touch-sensitive surface.

[0101] In some embodiments, event monitor 271 sends requests to peripherals interface 218 at predetermined intervals. In response, peripherals interface 218 sends event information. In other embodiments, peripherals interface 218 sends event information only when there is a significant event (e.g., an input is received above a predetermined noise threshold and / or for a predetermined duration).

[0102] In some embodiments, event sorter 270 also includes a hit view determination module 272 and / or an active event recognizer determination module 273 .

[0103] Hit view determination module 272 provides software procedures that determine where a sub-event occurred within one or more views when touch-sensitive display 212 displays more than one view. A view consists of controls and other elements visible to the user on the display.

[0104] Another aspect of a user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures are performed. The application view (for each application) in which touches are detected may correspond to a program level within the application's program or view hierarchy. For example, the lowest view for touch detection may be referred to as the hit view, and the set of events recognized as legal inputs may be determined, at least in part, based on the hit view of the initial touch that initiates a touch-based gesture.

[0105] The hit view determination module 272 receives sub-event-related information for a touch-based gesture. If an application has multiple views organized hierarchically, the hit view determination module 272 identifies the hit view as the lowest view in the hierarchy that should process the sub-event. The hit view is most often the lowest view in which the first sub-event occurs (e.g., the first event in a series of sub-events that make up an event or potential event). Once the hit view determination module 272 identifies a hit view, the hit view typically receives all sub-events related to the same touch or input source that was identified as the hit view.

[0106] The active event recognizer determination module 273 determines which one or more views in the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognizer determination module 273 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognizer determination module 273 determines that all views that contain the physical location of a sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of each sub-event. In another embodiment, even if all of the touch sub-events are limited to an area associated with a particular view, it is assumed that views higher in the hierarchy will still remain actively participating views.

[0107] Event dispatcher module 274 sends the event information to an event recognizer (e.g., event recognizer 280). In embodiments that include active event recognizer determination module 273, event dispatcher module 274 delivers the event information to the event recognizer determined by active event recognizer determination module 273. In some embodiments, event dispatcher module 274 stores the event information in an event queue, from which the event information is retrieved by each event receiver 282.

[0108] In some embodiments, operating system 226 includes event sorter 270, or event sorter 270 is included in application 236-1. In other embodiments, event sorter 270 is a separate module or part of another module, such as contact / motion module 230, stored in memory 202.

[0109] In some embodiments, application 236-1 includes multiple event handlers 290 and one or more application views 291, each containing instructions for processing touch events occurring within a respective view of the application's user interface. Each application view 291 of application 236-1 includes one or more event recognizers 280. Typically, each application view 291 includes multiple event recognizers 280. In other embodiments, one or more event recognizers 280 are part of a separate module of a user interface kit (not shown) or a higher-level object from which application 236-1 inherits methods or other properties. In some embodiments, each event handler 290 includes one or more of a data updater 276, an object updater 277, a GUI updater 278, and / or event data 279 received from event sorter 270. Event handler 290 may utilize or call data updater 276, object updater 277, or GUI updater 278 to update application internal state 292. Alternatively, one or more application views 291 include one or more event handlers 290. In some embodiments, each application view 291 includes one or more of data updater 276, object updater 277, and GUI updater 278.

[0110] Each event recognizer 280 receives event information (e.g., event data 279) from event sorter 270 and identifies an event from the event information. Event recognizer 280 includes an event receiver 282 and an event comparator 284. In some embodiments, event recognizer 280 further includes at least a subset of metadata 283 and event delivery instructions 288 (which may include sub-event delivery instructions).

[0111] Event receiver 282 receives event information from event sorter 270. The event information includes information about sub-events, such as touches or touch movement. Depending on the sub-event, the event information may also include additional information, such as the location of the sub-event. If the sub-event relates to touch movement, the event information may also include the speed and direction of the sub-event. In some embodiments, each event includes a rotation of the device from one orientation to another (e.g., from landscape to portrait or vice versa), and the event information includes corresponding information about the device's current orientation (also called device pose).

[0112] The event comparator 284 compares the event information with predefined event or sub-event definitions and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, the event comparator 284 includes an event definition 286. The event definition 286 includes definitions of events (e.g., predefined series of sub-events), such as Event 1 (287-1) and Event 2 (287-2). In some embodiments, sub-events of an event (287) include, for example, touch start, touch end, touch movement, touch cancellation, and multiple touches. In one example, the definition of Event 1 (287-1) is a double tap on a display object. The double tap is, for example, comprised of a first touch on the display object in a predetermined phase (touch start), a first lift-off in the predetermined phase (touch end), a second touch on the display object in the predetermined phase (touch start), and a second lift-off in the predetermined phase (touch end). In another example, the definition of Event 2 (287-2) is a drag on a display object. A drag consists of, for example, a touch (or contact) to a display object in a predetermined phase, a movement of the touch on the touch-sensitive display 212, and a lift-off of the touch (end of the touch). In some embodiments, the event further includes information for one or more associated event handlers 290.

[0113] In some embodiments, event definition 287 includes a definition of an event for each user interface object. Also, in some embodiments, event comparator 284 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view in which three user interface objects are displayed on touch-sensitive display 212, if a touch on touch-sensitive display 212 is detected, event comparator 284 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each display object is associated with a respective event handler 290, event comparator 284 uses the results of the hit test to determine which event handler 290 to activate. For example, event comparator 284 selects the event handler associated with the sub-event and the object that triggers the hit test.

[0114] In some embodiments, the definition of each event (287) further includes a delay action that delays delivery of the event information until it has been determined whether the set of sub-events matches the event type of the event recognizer.

[0115] If an event recognizer 280 determines that a series of sub-events does not match any event in the event definition 286, the event recognizer 280 enters an event-disabled, event-failed, or event-ended state and ignores subsequent sub-events for the touch-based gesture. In this situation, if there are other event recognizers still active for the hit view, those event recognizers continue to track and process sub-events for the ongoing touch-based gesture.

[0116] In some embodiments, each event recognizer 280 includes metadata 283 with configurable properties, flags, and / or lists that indicate how the event distribution system should handle sub-event distribution to actively participating event recognizers. In some embodiments, metadata 283 includes configurable properties, flags, and / or lists that indicate how event recognizers interact with each other or what interactions are possible. In some embodiments, metadata 283 includes configurable properties, flags, and / or lists that indicate whether to deliver sub-events to various levels in a view or programmatic hierarchy.

[0117] In some embodiments, each event recognizer 280 runs an event handler 290 associated with an event when one or more specific sub-events of that event are recognized. In some embodiments, each event recognizer 280 delivers event information associated with that event to the event handler 290. Running the event handler 290 is distinct from sending (and deferring) sub-events to each hit view. In some embodiments, the event recognizer 280 throws a flag associated with the recognized event, and the associated event handler 290 catches the flag and performs predefined processing.

[0118] In some embodiments, the event delivery instructions 288 include sub-event delivery instructions that deliver sub-event related event information without running an event handler. Instead, the sub-event delivery instructions deliver the event information to event handlers associated with a set of sub-events or actively participating views. Each event handler associated with a set of sub-events or actively participating views receives the event information and performs a predetermined process.

[0119] In some embodiments, data updater 276 generates and updates data for use by application 236-1. For example, data updater 276 updates phone numbers for use by contacts module 237 or saves video files for use by a video player module. In some embodiments, object updater 277 creates and updates objects for use by application 236-1. For example, object updater 277 creates new user interface objects or updates the positions of user interface objects. GUI updater 278 updates the GUI. For example, GUI updater 278 prepares and sends display information to graphics module 232 to display the display information on the touch-sensitive display.

[0120] In some embodiments, one or more event handlers 290 include or have access to data updaters 276, object updaters 277, and GUI updaters 278. In some embodiments, data updaters 276, object updaters 277, and GUI updaters 278 are included in a single module of application view 291 or each application 236-1. In other embodiments, each updater is included in two or more software modules.

[0121] The above description of event processing of user touches on a touch-sensitive display naturally also applies to different forms of user input, including non-touchscreen initiations, for operating multifunction device 200 with an input device. For example, mouse movements and mouse button presses, optionally combined with single or multiple keyboard presses or sustained presses, contact movements such as tapping, dragging, scrolling, etc. on a touchpad, pen stylus input, device movement, verbal commands, detected eye movements, biometric input, and / or any combination thereof, may be used as appropriate as inputs corresponding to sub-events that define the event being recognized.

[0122] FIG. 3 illustrates portable multifunction device 200 with touchscreen 212 according to some embodiments. The touchscreen optionally displays one or more graphics in user interface (UI) 300. In this embodiment, as with other embodiments described below, a user can select one or more graphics by making a gesture on the graphics, for example, with one or more fingers 302 (illustration not to scale) or one or more styluses 303 (illustration not to scale). In some embodiments, one or more graphics are selected when the user releases contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (left to right, right to left, up and / or down), and / or a rotation of a finger (right to left, left to right, up and / or down) in contact with device 200. Depending on the implementation and circumstances, inadvertent contact with a graphic does not select the graphic. For example, if the gesture corresponding to selection is a tap, a swipe gesture passing quickly over an application icon may not select the corresponding application.

[0123] Device 200 may further include one or more physical buttons, such as a "home" or menu button 304. As described above, menu button 304 may be used to navigate to any application 236 of a group of applications that may be running on device 200. Alternatively, in some embodiments, the menu button is implemented as a soft key within a GUI displayed on touchscreen 212.

[0124] In one embodiment, device 200 includes a touchscreen 212, a menu button 304, a pushbutton 306 for powering the device on / off and locking the device, one or more volume control buttons 308, a subscriber identity module (SIM) card slot 310, a headphone jack 312, and an external docking / charging port 224. Pushbutton 306 can be used to power the device on / off by pressing and holding the button down for a predetermined period of time, to lock the device by pressing and releasing the button before the predetermined period of time has elapsed, and / or to unlock the device or initiate an unlocking process. In another embodiment, device 200 also accepts verbal input via microphone 213 for enabling or disabling certain features. Device 200 also includes one or more contact intensity sensors 265 for detecting the intensity of a contact on touchscreen 212 and / or one or more tactile output generators 267 for generating a tactile output for a user of device 200, as needed.

[0125] FIG. 4 is a block diagram illustrating an example of a multifunction device with a display and a touch-sensitive surface, according to some embodiments. Device 400 need not be a portable device. In some embodiments, device 400 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as an educational toy), a gaming system, or a control device (e.g., a home or industrial controller). Device 400 typically includes one or more processing units (CPUs) 410, one or more network or other communication interfaces 460, memory 470, and one or more communication buses 420 for interconnecting these components. Communication bus 420 optionally includes circuitry (also called a chipset) for interconnecting system components and controlling communication between each system component. Device 400 includes an input / output (I / O) interface 430 with a display 440. Display 440 is typically a touchscreen display. I / O interface 430 may optionally further include a keyboard and / or mouse (or other pointing device) 450, a touchpad 455, a tactile output generator 457 (similar to one or more tactile output generators 267 described above with reference to FIG. 2A ) for generating tactile output on device 400, and sensors 459 (e.g., optical sensors, acceleration sensors, proximity sensors, touch-sensitive sensors, and / or contact intensity sensors similar to one or more contact intensity sensors 265 described above with reference to FIG. 2A ). Memory 470 optionally includes high-speed random access memory such as DRAM, SRAM, DDR, RAM, or other random access solid-state memory devices, and optionally includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 470 may also optionally include one or more storage devices remotely located from one or more CPUs 410.In some embodiments, memory 470 stores programs, modules, and data structures similar to, or a subset of, the programs, modules, and data structures stored in memory 202 of portable multifunction device 200 (FIG. 2A). Additionally, memory 470 optionally stores additional programs, modules, and data structures not present in memory 202 of portable multifunction device 200. For example, memory 470 of device 400 optionally stores drawing module 480, presentation module 482, word processing module 484, website creation module 486, disc authoring module 488, and / or spreadsheet module 490. However, these modules may not be stored in memory 202 of portable multifunction device 200 (FIG. 2A).

[0126] Each element identified in FIG. 4 may be stored in one or more of the memory devices mentioned above. Each module described in the previous section corresponds to an instruction set that performs the function described above. The modules or programs (e.g., instruction sets) described above need not be implemented as separate software programs, procedures, or modules; various embodiments may combine or rearrange various subsets of these modules. In some embodiments, memory 470 may store a subset of the modules and data structures described above, and further, memory 470 may store additional modules and data structures other than those described above.

[0127] Next, various embodiments of a user interface that can be implemented in, for example, portable multifunction device 200 will be described.

[0128] 5A is a diagram illustrating an example user interface for a menu of applications on portable multifunction device 200 in accordance with some embodiments. A similar user interface may be implemented on device 400. In some embodiments, user interface 500 includes the following elements, or a subset or superset thereof: One or more signal strength indicators 502 for one or more wireless communications, such as cellular signals or Wi-Fi signals ●Time 504 ●Bluetooth® indicator 505 ● Battery status indicator 506 ● Tray 508 containing icons for frequently used applications such as: An icon 516 for the phone module 238 labeled "Phone" that optionally includes an indicator 514 showing the number of missed calls or voicemail messages An icon 518 for the email client module 240 labeled "Mail" that includes an indicator 510 showing the number of unread emails ○ Icon 520 for Browser module 247 labeled "Browser" Icon 522 for the video and music player module 252 labeled "iPod," also known as the iPod (trademark of Apple Inc.) module 252 ●Icons for applications other than those listed above ○ Icon 524 for IM module 241 labeled "Message" ○ Icon 526 for Calendar module 248 labeled "Calendar" ○ Icon 528 for image management module 244 labeled "Photo" ○ Icon 530 for camera module 243 labeled "camera" ○ Icon 532 for the online video module 255 labeled "Online Video" ○ Icon 534 for stock widget 249-2 labeled "Stocks" ○ Icon 536 for map module 254 labeled "Map" ○ Icon 538 for weather widget 249-1 labeled "Weather" ○ Icon 540 for alarm clock widget 249-4 labeled "Clock" ○ Icon 542 for Workout Support Module 242 labeled "Workout Support" ○ Icon 544 for the note module 253 labeled "Note" An icon 546 for a settings application or module labeled "Settings," which provides access to settings for the device 200 and its various applications 236

[0129] Of course, the icon labels shown in FIG. 5A are merely exemplary. For example, icon 522 for video and music player module 252 may be labeled "Music" or "Music Player." Optionally, labels other than those described above may be used for the various application icons. In some embodiments, the label used for each application icon includes the name of the application corresponding to the application icon. In some embodiments, the label used for a particular application icon is different from the name of the application corresponding to the particular application icon.

[0130] 5B is a diagram illustrating an example of a user interface on a device (e.g., device 400, see FIG. 4) that includes a touch-sensitive surface 551 (e.g., tablet or touchpad 455, see FIG. 4) that is separate from display 550 (e.g., touchscreen display 212). Device 400 optionally further includes one or more contact intensity sensors (e.g., one or more sensors 457) for detecting the intensity of a contact on touch-sensitive surface 551 and / or one or more tactile output generators 459 for generating a tactile output for a user of device 400.

[0131] Although some of the examples below relate to input on touchscreen display 212 (a combination touch-sensitive surface and display), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, as shown in FIG. 5B . In some embodiments, the touch-sensitive surface (e.g., 551 in FIG. 5B ) has a major axis (e.g., 552 in FIG. 5B ) that aligns with a major axis (e.g., 553 in FIG. 5B ) on the display (e.g., 550). According to these embodiments, the device detects contact with touch-sensitive surface 551 (e.g., 560 and 562 in FIG. 5B ) at locations that correspond to respective locations on the display (e.g., 560 corresponds to 568 and 562 corresponds to 570 in FIG. 5B ). In this manner, in a multifunction device, if the touch-sensitive surface is separate from the display, user input (e.g., contacts 560, 562 and their movement) that the device detects on the touch-sensitive surface (e.g., 551 in FIG. 5B ) is used to manipulate a user interface on the device's display (e.g., 550 in FIG. 5B ), and it will be appreciated that similar methods may be used for other user interfaces disclosed herein, as appropriate.

[0132] Additionally, while the examples provided below primarily relate to finger input (e.g., finger contact, finger tap gesture, finger swipe gesture), it should be understood that in some embodiments, one or more of the finger inputs can be substituted with input from other input devices (e.g., mouse or stylus input). For example, a swipe gesture can be replaced with a mouse click (e.g., instead of a contact) followed by cursor movement along the path of the swipe (e.g., instead of a contact movement). As another example, a tap gesture can be replaced with a mouse click with the cursor positioned at the tap gesture location (e.g., instead of a contact detection followed by an end of contact detection). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice or a mouse and finger input can be used simultaneously, as needed.

[0133] FIG. 6A illustrates an example of a personal electronic device 600. Device 600 includes a main body 602. In some embodiments, device 600 can include some or all of the features described for devices 200 and 400 (e.g., FIGS. 2A-4B). In some embodiments, device 600 includes a touch-sensitive display screen 604 (hereinafter, touchscreen 604). Alternatively, in addition to or instead of touchscreen 604, device 600 includes a display and a touch-sensitive surface. Similar to devices 200 and 400, in some embodiments, touchscreen 604 (or the touch-sensitive surface) may include one or more contact intensity sensors for detecting the intensity of a received contact (e.g., a touch). The one or more intensity sensors in touchscreen 604 (or the touch-sensitive surface) can provide output data representing the touch intensity, and the user interface of device 600 can respond to each touch based on the touch intensity. This means that touches of different intensities can invoke different user interface actions on device 600.

[0134] Touch intensity detection and processing techniques are disclosed, for example, in the following related applications: International Patent Application No. PCT / US2013 / 040061, filed May 8, 2013, entitled "Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application," and International Patent Application No. PCT / US2013 / 069483, filed November 11, 2013, entitled "Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships," the entire contents of which are incorporated herein by reference.

[0135] In some embodiments, device 600 includes one or more input mechanisms 606 and 608. When input mechanisms 606 and 608 are included, the input mechanisms may be physical devices. Examples of physical input mechanisms include push buttons and rotary mechanisms. In some embodiments, device 600 includes one or more attachment mechanisms. When included, device 600 may be attached to, for example, a hat, glasses, earrings, a necklace, a shirt, a jacket, a bracelet, a watch band, a chain, pants, a belt, shoes, a wallet, a backpack, etc. Such attachment mechanisms allow device 600 to be worn by a user.

[0136] FIG. 6B illustrates an example of a personal electronic device 600. In some embodiments, the device 600 can include some or all of the components described above with reference to FIGS. 2A, 2B, and 4. The device 600 includes a bus 612 operably connecting an I / O unit 614 to one or more computer processors 616 and memory 618. The I / O unit 614 can be connected to a display 604, which can include a touch-sensing element 622 and, optionally, a touch intensity-sensing element 624. The I / O unit 614 can also be connected to a communication unit 630 that receives application and operating system data via Wi-Fi, Bluetooth, near-field communication (NFC), cellular, and / or other wireless communication technologies. The device 600 can include an input mechanism 606 and / or an input mechanism 608. Examples of the input mechanism 606 can be a rotatable input device or a depressible and rotatable input device. In some examples, the input mechanism 608 can be a button.

[0137] In some examples, the input mechanism 608 may be a microphone. The personal electronic device 600 may include various sensors, such as a GPS sensor 632, an accelerometer 634, an orientation sensor 640 (e.g., a compass), a gyroscope 636, a motion sensor 638, and / or any combination thereof. Additionally, each of the sensors and instruments that may be included may be operably connected to the I / O section 614.

[0138] The memory 618 of the personal electronic device 600 may be a computer-readable, non-transitory storage medium storing computer-executable instructions. The instructions, when executed by one or more computer processors 616, may cause each computer processor to perform the techniques described below, including processes 800-900 (FIGS. 8-9). The computer-executable instructions may be stored in and / or transmitted to any computer-readable, non-transitory storage medium used by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a processor-including system, or other system capable of retrieving and executing instructions from the instruction execution system, apparatus, or device. The personal electronic device 600 is not limited to the components and configurations shown in FIG. 6B and may include various non-illustrated or additional components.

[0139] As used herein, "affordance" refers to a graphical user interface object that may be displayed on the display screen of device 200, 400, and / or 600 (FIGS. 2, 4, and 6). For example, images (e.g., icons), buttons, and text (e.g., links) may each constitute an affordance.

[0140] As used herein, a “focus selector” refers to an input element that indicates where in a user interface a user is currently operating. In some implementations involving a cursor or other position marker, the cursor acts as a “focus selector” such that when input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 455 of FIG. 4 or touch-sensitive surface 551 of FIG. 5B ) while the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some embodiments involving a touchscreen display (e.g., touch-sensitive display system 212 of FIG. 2A or touchscreen 212 of FIG. 5A ) that allows direct manipulation of user interface elements on the touchscreen display, the detected contact on the touchscreen acts as a “focus selector” such that when input (e.g., a press input by contact) is detected on the touchscreen display at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations, focus moves from one area of a user interface to another area of the user interface without corresponding movement of a contact or cursor on the touchscreen display (e.g., using the tab or arrow keys to move focus from one button to another). In these implementations, the focus selector moves along with the movement of focus across different areas of the user interface. Regardless of the particular form the focus selector takes, it is typically a user interface element (or contact on a touchscreen display) that the user controls to communicate the user's intended user interface operation (e.g., by indicating to the device which user interface element the user intends to operate).For example, the position of a focus selector (e.g., cursor, touch, or selection box) on each button while detecting a pressure input on a touch-sensitive surface (e.g., a touchpad or touchscreen) indicates (with respect to other user interface elements displayed on the device's display) that the user intends to activate that button.

[0141] As used herein and in the claims, the "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on a plurality of intensity samples. The characteristic intensity is optionally based on a predetermined number of intensity samples or intensity samples collected within a predetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) relative to a predetermined event (e.g., after detecting the contact, before detecting liftoff of the contact, before or after detecting the start of contact movement, before detecting the end of the contact, before or after detecting an increase in contact intensity, and / or before or after detecting a decrease in contact intensity). The characteristic intensity of a contact is optionally based on one or more of the maximum contact intensity, the median contact intensity, the mean contact intensity, the top 10 percentile contact intensity, half the maximum contact intensity, 90% maximum contact intensity, etc. In some embodiments, the duration of the contact is used to determine the characteristic intensity (e.g., when the characteristic intensity is the average intensity over time). In some embodiments, the characteristic intensity is compared to one or more intensity thresholds to determine whether the user has performed an action. For example, the one or more intensity thresholds may include a first intensity threshold and a second intensity threshold. In this example, a contact having a characteristic intensity that does not exceed the first threshold is a first action, a contact having a characteristic intensity that exceeds the first threshold but not a second threshold is a second action, and a contact having a characteristic intensity that exceeds the second threshold is a third action. In some embodiments, comparing the characteristic intensity to one or more thresholds determines whether to perform one or more actions (e.g., perform each action or refrain from performing each action), rather than determining whether to perform either the first or second action.

[0142] In some embodiments, a portion of the gesture is identified to determine the characteristic intensity. For example, a touch-sensitive surface may receive successive swipe contacts that transition from a start position to an end position with increasing contact intensity at the end position. In this example, the characteristic intensity of the contact at the end position may be based, for example, on only a portion of the successive swipe contacts (e.g., the portion of the swipe contact at the end position) rather than the entirety of the successive swipe contacts. In some embodiments, a smoothing algorithm may be applied to the intensity of the swipe contacts before determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of an unweighted sliding-average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some circumstances, such a smoothing algorithm removes narrow spikes or dips in the swipe contact intensity for purposes of determining the characteristic intensity.

[0143] The intensity of contact on the touch-sensitive surface can also be characterized by comparison to one or more intensity thresholds, such as a contact-detection intensity threshold, a soft intensity threshold, a hard intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the soft intensity threshold corresponds to the intensity at which the device performs an action typically associated with clicking a physical mouse or trackpad button. In some embodiments, the hard intensity threshold corresponds to the intensity at which the device performs an action different from the action typically associated with clicking a physical mouse or trackpad button. In some embodiments, when a contact is detected with a characteristic intensity below the soft intensity threshold (e.g., above a nominal contact-detection intensity threshold at which the contact is undetectable), the device moves the focus selector in accordance with the movement of the contact on the touch-sensitive surface without performing an action associated with the soft or hard intensity thresholds. Unless otherwise specified, the intensity thresholds described above are generally consistent across the illustrated user interfaces.

[0144] When the characteristic intensity of the contact increases from an intensity below the light pressure intensity threshold to an intensity between the light pressure intensity and the hard pressure intensity thresholds, it may be referred to as a "light press" input. When the characteristic intensity of the contact increases from an intensity below the hard pressure intensity threshold to an intensity above the hard pressure intensity threshold, it may be referred to as a "hard press" input. When the characteristic intensity of the contact increases from an intensity below the contact detection intensity threshold to an intensity between the contact detection intensity and the light pressure intensity threshold, it may be referred to as a detection of a contact on the touch surface. When the characteristic intensity of the contact decreases from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold, it may be referred to as a detection of a lift-off of the contact from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In other embodiments, the contact detection intensity threshold is greater than zero.

[0145] In some embodiments disclosed herein, one or more actions are performed upon detecting a gesture including a pressure input or upon detecting a pressure input made by a contact (or multiple contacts). Here, the pressure inputs are detected based, at least in part, on detecting an increase in intensity of the contact (or multiple contacts) beyond a pressure input intensity threshold. In some embodiments, the actions are performed upon detecting an increase in the contact intensity beyond the pressure input intensity threshold (e.g., a "downstroke" of the pressure input). In some embodiments, the pressure inputs include an increase in the contact intensity beyond the pressure input intensity threshold and a subsequent decrease below the pressure input intensity threshold (e.g., an "upstroke" of the pressure input), and the actions are performed upon detecting the latter decrease below the pressure input intensity threshold (e.g., an "upstroke" of the pressure input).

[0146] In some embodiments, the device uses intensity hysteresis to prevent accidental input, sometimes referred to as "jitter." Specifically, the device defines or selects a hysteresis intensity threshold that has a predetermined relationship to the pressure input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units below the pressure input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or any reasonable percentage of the pressure input intensity threshold). Thus, in some embodiments, the pressure input includes an increase in contact intensity above the pressure input intensity threshold, followed by a decrease in contact intensity below a hysteresis intensity threshold corresponding to the pressure input intensity threshold, and upon detecting a decrease in contact intensity below the latter hysteresis intensity threshold (e.g., an "upstroke" of the pressure input), a respective action is performed. Similarly, in some embodiments, a pressure input is detected only when the device detects an increase in contact intensity from an intensity below the hysteresis intensity threshold to an intensity above the pressure input intensity threshold, optionally followed by a decrease in contact intensity back to the hysteresis intensity, and actions are then performed in response to the detection of the pressure input (e.g., an increase or decrease in contact intensity, as the situation demands).

[0147] For ease of explanation, actions performed in response to pressure inputs or gestures including pressure inputs associated with the pressure input intensity thresholds described above may optionally be triggered upon detection of any of the following: an increase in contact intensity above the pressure input intensity threshold, an increase in contact intensity from below a hysteresis intensity threshold to above the pressure input intensity threshold, a decrease in contact intensity below the pressure input intensity threshold, and / or a decrease in contact intensity below a hysteresis intensity threshold corresponding to the pressure input intensity threshold. Furthermore, although in some examples actions are performed upon detection of a decrease in contact intensity below the pressure input intensity threshold, in this case, the action is optionally performed in response to detection of a decrease in contact intensity below a hysteresis intensity threshold that matches or is lower than the pressure input intensity threshold. 3. Digital Assistant System

[0148] FIG. 7A is a block diagram of a digital assistant system 700 based on various examples. In some examples, digital assistant system 700 may be implemented on a stand-alone computer system. In some examples, digital assistant system 700 may be distributed across multiple computers. In some examples, some of the modules and functionality of the digital assistant may be divided into a server portion and a client portion, as shown in FIG. 1, where the client portion resides on one or more user devices (such as devices 104, 122, 200, 400, or 600) and communicates with the server portion (such as server system 108) over one or more networks. In some examples, digital assistant system 700 may embody server system 108 (and / or digital assistant server 106) shown in FIG. 1. It should be noted that digital assistant system 700 is merely one example of a digital assistant system, and that digital assistant system 700 may have more or fewer components than those shown, may combine two or more components, or may have different configurations or arrangements of these components. The various components shown in FIG. 7A may be embodied in hardware, software instructions executed by one or more processors, firmware with one or more signal processing and / or application specific integrated circuits, or a combination thereof.

[0149] Digital assistant system 700 may include memory 702, one or more processors 704, an input / output (I / O) interface 706, and a network communication interface 708. These components may be able to communicate with each other via one or more communication buses or signal lines 710.

[0150] In some examples, memory 702 may include non-transitory computer-readable media, such as high-speed random access memory (such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices) and / or non-volatile computer-readable storage media.

[0151] In some examples, I / O interface 706 may couple input / output devices 706, such as a display, keyboard, touchscreen, and microphone, of digital assistant system 700 to user interface module 722. I / O interface 706 may cooperate with user interface module 722 to receive user input (such as voice input, keyboard input, or touch input) and process it appropriately. For example, in some examples where the digital assistant is implemented on a standalone user device, digital assistant system 700 may have any of the components and I / O communication interfaces described with respect to device 200, 400, or 600 of FIGS. 2A, 4, 6A-B, respectively. In some examples, digital assistant system 700 may represent the server portion of a digital assistant implementation and may interact with a user through a client-side portion residing on the user device (such as device 104, 200, 400, or 600).

[0152] In some examples, the network communication interface 708 may include one or more wired communication ports 712 and / or wireless transceiver circuitry 714. The wired communication ports may transmit and receive communication signals via one or more wired interfaces, such as Ethernet, Universal Serial Bus (USB), or FIREWIRE. The wireless circuitry 714 may transmit and receive RF and / or optical signals to and from communication networks and other communication devices. The wireless devices may use any of a number of communication standards, protocols, and technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or other suitable communication protocols. The network communication interface 708 may enable communication between the digital assistant system 700 and networks, such as the Internet, intranets, and / or wireless networks, such as cellular networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs), and other devices.

[0153] In some examples, memory 702 or the computer-readable storage medium of memory 702 may store a number of programs, modules, instructions, and data structures, including all or part of an operating system 718, a communications module 720, a user interface module 722, one or more applications 724, and a digital assistant module 726. In particular, memory 702 or the computer-readable storage medium of memory 702 may store a number of instructions that perform processes 800, 900, described below. One or more processors 704 may execute these programs, modules, and instructions and read / write to a number of data structures.

[0154] An operating system 718 (such as Darwin®, RTXC, LINUX®, UNIX®, iOS®, OS X®, WINDOWS®, or an embedded operating system such as VxWorks®) may have various software components and / or drivers for control and management of general system tasks (such as memory management, storage control, power management, etc.) and for facilitating communication between various hardware, firmware, and software components.

[0155] A communications module 720 may facilitate communication between the digital assistant system 700 and other devices via the network communications interface 708. For example, the communications module 720 may be capable of communicating with the RF circuitry 208 of electronic devices such as devices 200, 400, and 600 shown in FIGS. 2A, 4, and 6A-B, respectively. The communications module 720 may include various components for processing data received by the wireless circuitry 714 and / or the wired communications port 712.

[0156] A user interface module 722 may receive commands and / or input from a user (e.g., from a keyboard, touchscreen, pointing device, controller, and / or microphone) via the I / O interface 706. The user interface module 722 may generate output (e.g., voice, sound, video, text, icons, vibration, haptic feedback, light, etc.) and communicate this to the user via the I / O interface 706 (e.g., through a display, audio channel, speaker, touchpad, etc.).

[0157] Applications 724 may include programs and / or modules configured to be executed by one or more processors 704. For example, if the digital assistant system is implemented on a standalone user device, applications 724 may include user applications such as games, calendar applications, navigation applications, or email applications. If the digital assistant system 700 is implemented on a server, applications 724 may include, for example, resource management applications, diagnostic applications, or scheduling applications.

[0158] Memory 702 may store a digital assistant module 726 (or a server portion of a digital assistant). In some examples, the digital assistant module 726 may include the following submodules: an input / output processing module 728, a speech-to-text (STT) processing module 730, a natural language processing module 732, a dialog flow processing module 734, a task flow processing module 736, a service processing module 738, and a speech synthesis module 740, or a subset or superset thereof. Each of these modules may include the following systems or data and models of the digital assistant module 726: an ontology 760, a vocabulary index 744, user data 748, a task flow model 754, a service model 756, and an ASR system, or a subset or superset thereof.

[0159] In some examples, using processing modules, data, and models implemented in digital assistant module 726, the digital assistant may perform at least some of the following: converting voice input to text; identifying a user's intent expressed in natural language input received from the user; actively eliciting and obtaining information necessary to fully infer the user's intent (e.g., by disambiguating words, games, intents, etc.); determining a task flow for fulfilling the inferred intent; and executing the task flow for fulfilling the inferred intent.

[0160] In some examples shown in FIG. 7B , I / O processing module 728 may interact with a user via I / O device 716 of FIG. 7A or with a user device (such as device 104, 200, 400, or 600) via network communication interface 708 of FIG. 7A to obtain user input (such as voice input) and provide responses (such as voice output) to the user input. Concurrently with or shortly after receiving the user input, I / O processing module 728 may optionally obtain contextual information regarding the user input from the user device. The contextual information may include user-specific data, vocabulary, and / or preferences related to the user input. In some examples, the contextual information may further include information regarding the software and hardware state of the user device at the time the user request is received and / or the user's ambient environment at the time the user request is received. In some examples, I / O processing module 728 may send follow-up questions to the user regarding the user request and receive answers thereto from the user. A user request is received by the I / O processing module 728, and if the user request may include voice input, the I / O processing module 728 may send the voice input to the STT processing module 730 (or voice recognizer) for speech-to-text conversion.

[0161] The STT processing module 730 may include one or more ASR systems. The one or more ASR systems may process the speech input received via the I / O processing module 728 to generate recognition results. Each ASR system may include a front-end speech preprocessor. The front-end speech preprocessor may extract representative features from the speech input. For example, the front-end speech preprocessor may perform a Fourier transform on the speech input to extract spectral features characterizing the speech input as a sequence of representative multidimensional vectors. Furthermore, each ASR system may include one or more speech recognition models (e.g., acoustic models and / or language models) and implement one or more speech recognition engines. Examples of speech recognition models may include hidden Markov models, Gaussian mixture models, deep neural network models, n-gram language models, and other statistical models. Examples of speech recognition engines may include dynamic time warping-based engines and weighted finite state transducer (WFST)-based engines. One or more speech recognition models and one or more speech recognition engines may be used to process the extracted representative features of the front-end speech pre-processor to generate intermediate recognition results (such as phonemes, phoneme strings, and subwords) and ultimately generate text recognition results (such as words, word strings, or token strings). In some examples, the speech input may be processed at least partially by a third-party service or on the user's device (such as device 104, 200, 400, or 600) to generate the recognition results. Once the STT processing module 730 generates a recognition result including a text string (such as a word, word string, or token string), the recognition result may be passed to the natural language processing module 732 for intent inference.

[0162] Further details regarding speech-to-text processing are described in U.S. Utility Application No. 13 / 236,942, filed September 20, 2011, entitled "Consolidating Speech Recognition Results," the entire disclosure of which is incorporated herein by reference.

[0163] In some examples, the STT processing module 730 may have, or access via the phonetic alphabet conversion module 731, a vocabulary of recognizable words. Each vocabulary word may be associated with one or more candidate pronunciations of the word represented in the speech recognition phonetic alphabet. In particular, the vocabulary of recognizable words may include words associated with multiple candidate pronunciations. For example, if the vocabulary includes a candidate pronunciation

[0164]

number

[0165] In some examples, the candidate pronunciations may be ranked based on the degree of commonality of the candidate pronunciations.

[0166]

number

[0167]

number

[0168]

number

[0169]

number

[0170]

number

[0171]

number

[0172]

number

[0173] When a speech input is received, the STT processing module 730 may be used to determine the phonemes that correspond to the speech input (e.g., using an acoustic model) and then attempt to determine words that match the phonemes (e.g., using a language model). For example, the STT processing module 730 may first determine a sequence of phonemes that correspond to a portion of the speech input.

[0174]

number

[0175] In some examples, the STT processing module 730 may use approximate matching techniques to identify words in an utterance. Thus, for example, the STT processing module 730 may identify the phoneme sequence that corresponds to the word "tomato."

[0176]

number

[0177] The digital assistant's natural language processing module 732 ("natural language processor") may receive the string of words or tokens ("token string") generated by the STT processing module 730 and attempt to associate the token string with one or more "actionable intents" recognized by the digital assistant. An "actionable intent" may represent a task that can be performed by the digital assistant and may have an associated task flow embodied as a task flow model 754. The associated task flow may be a series of programmed actions and steps that the digital assistant takes to perform the task. The scope of the digital assistant's capabilities may depend on the number and types of task flows embodied and stored as task flow models 754, or in other words, the number and types of "actionable intents" that the digital assistant recognizes. The efficiency of the digital assistant may depend on the assistant's ability to infer the correct "actionable intent" from a user request expressed in natural language.

[0178] In some examples, in addition to the string of words or tokens obtained from the STT processing module 730, the natural language processing module 732 may also receive contextual information related to the user request, for example, from the I / O processing module 728. The natural language processing module 732 may optionally use the contextual information to clarify, supplement, and / or further define the information contained in the string of tokens received from the STT processing module 730. The contextual information may include, for example, user preferences, hardware and / or software state of the user device, sensor information collected before, during, or immediately after the user request, previous interactions (e.g., dialogs) between the digital assistant and the user, etc. As described herein, the contextual information may be dynamic and change with time, location, dialog content, and other factors.

[0179] In some examples, natural language processing may be based on, for example, ontology 760. Ontology 760 may be a hierarchical structure including multiple nodes, each representing an "actionable intention" or "attribute" that is related to one or more of multiple "actionable intentions" or multiple other "attributes." As described above, an "actionable intention" may represent a task that the digital assistant can perform, i.e., that is "actionable" or can act upon. An "attribute" may represent a parameter associated with a subordinate aspect of an actionable intention or another attribute. The connection between an actionable intention node and an attribute node in ontology 760 may define how the parameter represented by the attribute node relates to the task represented by the actionable intention node.

[0180] In some examples, ontology 760 may be formed from multiple actionable intention nodes and multiple attribute nodes. Within ontology 760, each actionable intention node may be linked to one or more attribute nodes directly or through one or more intermediate attribute nodes. Similarly, each attribute node may be linked to one or more actionable intention nodes directly or through one or more intermediate attribute nodes. For example, as shown in FIG. 7C , ontology 760 may include a “restaurant reservation” node (i.e., an actionable intention node). The attribute nodes “restaurant,” “date / time” (of the reservation), and “number of people” may each be directly linked to an actionable intention node (i.e., the “restaurant reservation” node).

[0181] Additionally, the attribute nodes “cuisine,” “price range,” “phone number,” and “location” may be subordinate nodes of the attribute node “restaurant,” and each may link to the “restaurant reservation” node (i.e., an actionable intention node) via the intermediate attribute node “restaurant.” As another example, as shown in FIG. 7C , ontology 760 may further include a “set reminder” node (i.e., another actionable intention node). The attribute nodes “date / time” (for setting a reminder) and “target” (of the reminder) may each be linked to the “set reminder” node. Because the attribute “date / time” may be related to both the task of making a restaurant reservation and the task of setting a reminder, in ontology 760, the attribute node “date / time” may be linked to both the “restaurant reservation” node and the “set reminder” node.

[0182] An actionable intent node, together with its linked concept nodes, may be described as a "domain." In this description, each domain may be associated with each actionable intent and refers to a group of nodes (and their relationships) related to a particular actionable intent. For example, the ontology 760 shown in FIG. 7C may include an example of a restaurant reservation domain 762 and an example of a reminder domain 764 within the ontology 760. The restaurant reservation domain may include the actionable intent node "reservation of restaurant," the attribute nodes "restaurant," "date / time," and "number of people," and the subordinate attribute nodes "cuisine," "price range," "phone number," and "location." The reminder domain 764 may include the actionable intent node "reminder setting" and the attribute nodes "target" and "date / time." In some examples, the ontology 760 may be formed from multiple domains. Each domain may share one or more attribute nodes with one or more other domains. For example, a "date / time" attribute node may be associated with a number of different domains (eg, a scheduling domain, a travel booking domain, a movie ticket domain, etc.) in addition to a restaurant reservation domain 762 and a reminder domain 764.

[0183] 7C illustrates two example domains in ontology 760, other domains may include, for example, "find a movie," "make a phone call," "get directions," "schedule a meeting," "send a message," and "answer a question," "read a list," "provide navigation instructions," "provide task instructions," etc. The "send a message" domain may be associated with a "send a message" actionable intent node and may further include attribute nodes such as "recipient," "message type," and "message body." The attribute node "recipient" may be further defined by subordinate attribute nodes such as "recipient name" and "message destination."

[0184] In some examples, ontology 760 may include all domains (and hence actionable intents) that a digital assistant can understand and act upon. In some examples, ontology 760 may be modified by adding or removing domains or entire nodes, or by changing the relationships between nodes in ontology 760.

[0185] In some examples, nodes associated with interrelated actionable intents may be clustered under a "superdomain" in ontology 760. For example, a "travel" superdomain may include a cluster of travel-related attribute nodes and actionable intent nodes. Actionable intent nodes related to travel may include "book an airline flight," "book a hotel," "rent a car," "get directions," "find points of interest," etc. Actionable intent nodes under the same superdomain (such as the "travel" superdomain) may have many attribute nodes in common. For example, the actionable intent nodes for "book an airline flight," "book a hotel," "rent a car," "get directions," and "find points of interest" may share one or more of the attribute nodes "origin," "destination," "departure date / time," "arrival date / time," and "number of people."

[0186] In some examples, each node in ontology 760 may be associated with a set of words and / or phrases related to the attribute or actionable intent represented by the node. Each set of words and / or phrases associated with each node may be what is called a "vocabulary" associated with the node. Each set of words and / or phrases associated with each node may be stored in vocabulary index 744 in association with the attribute or actionable intent represented by the node. For example, referring again to FIG. 7B , the vocabulary associated with the attribute node "restaurant" may include words such as "food," "drink," "dish," "hungry," "eat," "pizza," "fast food," and "meal." As another example, the vocabulary associated with the actionable intent node "make a phone call" may include words and phrases such as "call," "phone," "dial," "receive," "call this number," and "call to." The vocabulary index 744 may optionally include words and phrases in multiple different languages.

[0187] The natural language processing module 732 may receive a token string (e.g., a text string) from the STT processing module 730 and determine which nodes are associated with words in the token string. In some examples, if a word or phrase in the token string is found to be associated (via the vocabulary index 744) with one or more nodes in the ontology 760, the word or phrase may "trigger" or "activate" those nodes. The natural language processing module 732 may select one of multiple possible intents as the task the user intends the digital assistant to perform based on the amount and / or relative importance of the activated nodes. In some examples, the domain with the most "triggered" nodes may be selected. In some examples, the domain with the highest confidence value (e.g., based on the relative importance of its various triggered nodes) may be selected. In some examples, the domain may be selected based on a combination of the number and importance of the triggered nodes. In some examples, additional factors, such as whether the digital assistant has previously correctly interpreted a similar request from the user, may also be considered in selecting a node.

[0188] User data 748 may include user-specific information such as the user's specific vocabulary, user preferences, user addresses, the user's default and secondary languages, the user's contact list, and other short-term or long-term information for each user. In some examples, natural language processing module 732 may use the user-specific information to supplement information included in the user input to further define the user's intent. For example, in response to a user request to "invite my friends to my birthday party," natural language processing module 732 may access user data 748 to determine who the "friends" are and when and where the "birthday party" is being held, rather than requiring the user to explicitly include such information in the request.

[0189] Further details of searching ontologies based on token strings are described in U.S. Utility Application No. 12 / 341,743, filed December 22, 2008, entitled "Method and Apparatus for Searching Using An Active Ontology," the entire disclosure of which is incorporated herein by reference.

[0190] In some examples, once the natural language processing module 732 identifies an actionable intent (or domain) based on the user request, the natural language processing module 732 may generate a structured query that represents the identified actionable intent. In some examples, the structured query may include parameters of one or more nodes in the domain for the actionable intent, where at least some of the parameters incorporate unique information and requirements specified in the user request. For example, a user may say, "Make a dinner reservation at a sushi restaurant for 7 o'clock." In this case, the natural language processing module 732 may be able to correctly recognize that the actionable intent is "reservation for restaurant" based on the user input. According to the ontology, a structured query for the "reservation for restaurant" domain may include parameters such as {cuisine}, {time}, {date}, and {number of people}. In some examples, based on the speech input and text derived from the speech input using the STT processing module 730, the natural language processing module 732 may generate a partially structured query for the restaurant reservation domain, where the partially structured query may include the parameter {cuisine="sushi"} and the parameter {time="7 PM"}. However, in this example, the information included in the user's utterance is insufficient to complete a structured query related to the domain. Therefore, other required parameters, such as {number of people} and {date}, may not be specified in the structured query based on the currently available information. In some examples, the natural language processing module 732 may incorporate received context information into some parameters of the structured query. For example, in some examples, if the user requests sushi restaurants "near me," the natural language processing module 732 may incorporate GPS coordinates from the user device into the {location} parameter in the structured query.

[0191] In some examples, natural language processing module 732 may provide the generated structured query (including any completed parameters) to task flow processing module 736 ("task flow processor"). Task flow processing module 736 may be configured to receive the structured query from natural language processing module 732, complete the structured query if necessary, and perform the actions necessary to "complete" the user's final request. In some examples, the various steps necessary to complete these tasks may be provided in task flow model 754. In some examples, task flow model 754 may include steps for obtaining further information from the user and task flows for performing actions associated with the actionable intent.

[0192] As described above, to complete a structured query, the task flow processing module 736 may need to initiate further dialogue with the user to obtain more information and / or disambiguate potentially ambiguous utterances. If such dialogue is necessary, the task flow processing module 736 may invoke the dialog flow processing module 734 to dialogue with the user. In some examples, the dialog flow processing module 734 may determine how (and / or when) to request more information from the user and may receive and process user responses. Questions may be sent to and replies received from the user via the I / O processing module 728. In some examples, the dialog flow processing module 734 may provide dialog output to the user via audio and / or visual output and receive input from the user via verbal or physical responses (e.g., clicks). In the above example, when task flow processing module 736 invokes dialog flow processing module 734 to determine the "number of people" and "date" information for a structured query associated with the domain "restaurant reservation," dialog flow processing module 734 may generate questions for the user, such as "how many people?" and "what day?" Upon receiving a response from the user, dialog flow processing module 734 may incorporate the missing information into the structured query or forward the information to task flow processing module 736 to complete the missing information from the structured query.

[0193] Once the taskflow processing module 736 completes the structured query for the actionable intent, the taskflow processing module 736 may proceed to execute the final task associated with the actionable intent. Thus, the taskflow processing module 736 may execute the steps and instructions in the taskflow model according to specific parameters included in the structured query. For example, a taskflow model for an actionable intent "make a restaurant reservation" may include steps and instructions for contacting a restaurant and actually requesting a reservation for a specific number of people at a specific time. For example, using a structured query such as {restaurant reservation, restaurant=ABC Cafe, date=3 / 12 / 2012, time=7 PM, number of people=5}, the taskflow processing module 736 may execute the following steps: (1) log on to ABC Cafe's server or the restaurant reservation system OPENTABLE®, (2) enter the date, time, and number of people into a form on the website, (3) submit the form, and (4) add a calendar entry for the reservation to the user's calendar.

[0194] In some examples, task flow processing module 736 may use the assistance of service processing module 738 ("service processing module") to complete a task requested in the user input or return information requested in the user input. For example, service processing module 738 may make a phone call, set a calendar entry, invoke a map search, or invoke or interact with a third-party service (e.g., a restaurant reservation portal, a social networking website, a banking portal, etc.) on behalf of task flow processing module 736. In some examples, the protocols and application programming interfaces (APIs) required by each service may be specified by each service model in multiple service models 756. Service processing module 738 may access the appropriate service model for a service and generate a request for the service according to the protocols and APIs required by the service according to the service model.

[0195] For example, if a restaurant allows an online reservation service, the restaurant may submit a service model to specify the necessary parameters for making a reservation and an API for transmitting the values of the necessary parameters to the online reservation service. When requested by the task flow processing module 736, the service processing module 738 may establish a network connection with the online reservation service using the web address stored in the service model and transmit the necessary parameters for the reservation (such as time, date, and number of people) to the online reservation interface in a format compatible with the online reservation service's API.

[0196] In some examples, the natural language processing module 732, the dialog flow processing module 734, and the task flow processing module 736 may be collectively used iteratively to infer and define a user's intent, obtain information to further clarify and refine the user's intent, and ultimately generate a response (i.e., output to the user or completion of a task) to fulfill the user's intent. The generated response may be a dialog response to the voice input that at least partially fulfills the user's intent. Furthermore, in some examples, the generated response may be output as a voice output. In these examples, the generated response may be sent to a speech synthesis module 740 (e.g., a voice synthesizer), where it may be processed to synthesize the dialog response into speech. In still other examples, the generated response may be data content related to fulfilling a user request in the voice input.

[0197] A speech synthesis module 740 may be configured to synthesize speech output for the user. The speech synthesis module 740 may synthesize speech output based on text provided by the digital assistant. For example, a generated dialog response may be in the form of a text string. The speech synthesis module 740 may convert the text string into audible speech output. The speech synthesis module 740 may use any suitable speech synthesis technique to generate speech output from text, including, but not limited to, concatenative synthesis, unit selection synthesis, diphone synthesis, domain-specific synthesis, formant synthesis, articulatory synthesis, synthesis based on a hidden Markov model (HMM), and sinusoidal synthesis. In some examples, the speech synthesis module 740 may be configured to synthesize individual words based on phoneme strings corresponding to those words. For example, a phoneme string may be associated with a word in the generated dialog response. The phoneme string may be stored in metadata associated with the word. The speech synthesis model 740 may be configured to directly process the phoneme string in the metadata to synthesize the word into speech.

[0198] In some examples, instead of (or in addition to) using the speech synthesis module 740, speech synthesis may be performed on a remote device (such as the server system 108), and the synthesized speech may be sent to the user device for output to the user. For example, this may occur in an implementation in which output to the digital assistant is generated on the server system. Also, because server systems typically have more processing power or resources than user devices, it may be possible to obtain higher quality speech output than would actually be used in client-side synthesis.

[0199] Further details regarding digital assistants are provided in U.S. Utility Application No. 12 / 987,982, filed January 10, 2011, entitled "Intelligent Automated Assistant," and U.S. Utility Application No. 13 / 251,088, filed September 30, 2011, entitled "Generating and Processing Task Items That Represent Tasks to Perform," the entire disclosures of which are incorporated herein by reference. 4. Processing for the operation of the digital assistant

[0200] FIG. 8 is a flow diagram of a process 800 for operating a digital assistant according to some examples. Process 800 is performed, for example, using one or more electronic devices (such as devices 104, 108, 200, 400, or 600) implementing a digital assistant. In some examples, process 800 is performed using a client-server system (such as system 100), and blocks of process 800 may be divided in any manner between a server (such as DA server 106) and a client service. In other examples, process 800 may be divided between a server and multiple client devices (such as a mobile phone and a smartwatch). Thus, while portions of process 800 are described herein as being performed by a particular device in a client-server system, it will be understood that process 800 is not so limited. In other examples, process 800 is performed using only a single client device (such as user device 104) or multiple client devices. Some blocks are optionally combined, the order of some blocks is optionally changed, and some blocks are optionally omitted in process 800. In some examples, additional steps may be performed in combination with process 800.

[0201] In block 805, natural language user input is received by a user device, such as user device 104 of FIG. 1 . The natural language user input may be, for example, a voice input or a voice input included in a text input. In some examples, the natural language user input may be a request to the user device and / or another device to perform a task. For example, in the example "Call a car to 1200 Main Street," the natural language user input may include a request to the user device to reserve a car using a ride booking service. In some examples, the natural language user input may further specify one or more parameters of the requested task. For example, "1200 Main Street" specifies a pickup location for the car reservation. In the example "Order my usual at Domino's," the natural language user input may include a request to the user device to order food at the pizza chain Domino's. "My usual" may further contextually specify which food to order.

[0202] At block 810, an intent and, optionally, one or more parameters associated with the intent are identified. The intent and parameters may be derived, for example, from natural language user input. In some examples, a text string is obtained from the natural language user input, and the intent and parameters are derived from the text string. The text string may be obtained using any known speech-to-text technique.

[0203] As described above, an intent may correspond to a task requested by a user. Thus, identifying (e.g., determining) an intent may include identifying a task specified in the natural language user input and / or inferring an intent corresponding to the requested task based on the language and / or context of the natural language user input. An intent may correspond to any type of task performed by a user device, and in particular, may correspond to a task performed by one or more applications of the user device, which are described in more detail below.

[0204] The intent and / or parameters may be identified based on the type of device, as a first type of device may perform a different task than a second type of device. For example, a first type of device may perform a task associated with the intent to reserve a car, and a second type of device may perform a task associated with the intent to provide the status of the car reservation after the reservation has been made.

[0205] In some examples, an intent is associated with (e.g., included in) one or more domains (e.g., intent categories, sets of intents). Each domain may include a particular class of intents, allowing for intuitive grouping of intents. For example, a ride booking domain may include intents to reserve a car, cancel a car reservation, and / or any other intents directed to tasks commonly associated with ride booking. In another example, an air travel domain may include intents to check in for a flight, cancel a flight, change a flight, retrieve flight information, and / or any other intents directed to tasks commonly related to air travel. In another example, a navigation domain may include intents to get directions, retrieve traffic information, and / or any other intents directed to tasks commonly related to navigation. In yet another example, a financial transaction domain may include intents to send money, receive payment, and / or any other intents directed to tasks commonly related to financial transactions.

[0206] Identifying parameters may include identifying portions of the natural language input that specify how a task corresponding to the intent is to be performed. Parameters may specify, for example, a location (such as an address or point of interest), a time, a date, a contact, a type, text (e.g., to be inserted into an email or message), an amount (such as a distance or amount), and possibly the name of a software application that will perform the task. Parameters may also specify other conditions of the task as exemplified herein.

[0207] The parameters may be identified, for example, using one or more detectors. Each of the multiple detectors may be configured to parse the natural language user input (e.g., a textual representation of the natural language user input) to identify one or more data types. For example, a first detector may be configured to identify the user's contact information, and a second detector may be configured to identify an address. Other detectors may identify data types including, but not limited to, phone numbers, names, people of interest, points of interest, URLs, times, flight numbers, baggage tracking numbers, and dates.

[0208] In some examples, words of the custom vocabulary may be identified in the natural language user input, for example, as parameters. In one example, one or more detectors may be configured to identify user-specific terms. User-specific terms may include any terms associated with and / or specified by a user of the electronic device, such as contact names, addresses, phone numbers, etc. In another example, one or more detectors may be configured to identify custom vocabulary for one or more applications, respectively. The custom vocabulary for an application may include the name of the application (e.g., Uber®, Lyft®, Instagram®, Flickr®, WeChat®, WhatsApp®, LINE®, Viber®, etc.) and / or other terms uniquely associated with the application (e.g., UberX, DM, Lyftline, ZipCar®, etc.). In some examples, the custom vocabulary may be included in vocabulary index 744 (FIG. 7B).

[0209] In some examples, one or more applications may register with an application registration service. The service may be hosted by or accessible by the server 108 and / or the user device 104. Registering in this manner may include specifying one or more custom vocabulary terms associated with the application and, optionally, one or more language models for the custom vocabulary terms. A language model may, for example, provide one or more pronunciations for each of multiple custom vocabulary terms. The language models used in this manner may be used to assist in identifying uses of such custom terms during subsequent analysis of natural language user input.

[0210] In some examples, one or more parameters may be inferred from natural language user input. In the example "Get a ride to the stadium," one parameter associated with the intent may be inferred to be the user's current location. In another example, "Refund John for his meal," one parameter associated with the intent may be inferred to be a monetary amount.

[0211] In some examples, parameters associated with an intent are identified after the intent is identified based on natural language user input. Additionally, in some examples, parameters not associated with the intent are not identified. For example, an intent corresponding to guidance (e.g., driving directions) may be associated with parameters specifying one or more locations (e.g., a starting location and / or a destination) and / or a mode of transportation. Consider the example of "Give me real-time driving directions to 1200 Main Street." In this example, the identified intent corresponds to the task of providing directions, "driving" is a parameter specifying a mode of transportation, and "1200 Main Street" is a parameter specifying a location. The "real-time" portion of the user input is not a parameter associated with the identified intent. Thus, although there may be an intent for which "real-time" is a valid parameter, "real-time" is not identified as a parameter during operation.

[0212] In other examples, one or more parameters may be first identified, and then the intent may be identified based on the identified one or more parameters. In still other examples, the intent and the parameters associated with the intent may be identified simultaneously.

[0213] In some examples, the intent and parameters of the natural language user input may be identified by a user device, such as user device 104 of Figure 1. In other examples, the user device provides the natural language user input (or a representation thereof) to a server, such as server 108 of Figure 1, which identifies (e.g., determines) the intent and parameters of the natural language user input as described above. The server then provides (e.g., transmits) the identified intent and parameters to the user device.

[0214] Optionally, after the intent and any parameters are identified, the user device confirms the identified intent and / or parameters to a user of the user device, and in some cases, confirms the inferred parameters to a user of the user device. Confirming in this manner may include prompting the user to confirm the identified intent and all identified parameters associated with the intent in response to the natural language query. For example, in response to a user input of "arrange a ride to the airport," the user device may provide the natural language query of "Do you want me to arrange a ride from my current location to the airport?" The natural language query provided by the user device may be provided to the user as text using a touch-sensitive display of the user device and / or as audio using an audio output component (such as speaker 211 of FIG. 2 ) of the user device. The user may respond to the natural language query, for example, by providing natural language user input to the user device.

[0215] Optionally, the user device confirms individual parameters. In some examples, this may include prompting the user to confirm one or more parameters. For example, in response to a user input of "Get me a ride to the station," the user device may provide the natural language query of "Are you talking about Penn Station?" As another example, in response to a user input of "Pay John $5," the user device may provide the natural language query of "Are you talking about John Smith?" The natural language query provided by the user device may be provided to the user as text using a touch-sensitive display of the user device and / or as audio using an audio output component (such as speaker 211 in FIG. 2 ) of the user device. The user may respond to the natural language query, for example, by providing natural language user input to the user device.

[0216] In some examples, one or more parameters are contextual. Thus, the user device may determine (e.g., analyze) one or more parameters based on context information. The context information may be context information of the user device (or any data stored thereon) and / or context information of a user of the user device. For example, the natural language user input may be "arrange a ride to my house." Because "my house" is a contextual parameter and does not specify an actual location, the user device may determine the location of the user device and specify the determined location as the parameter (i.e., in place of "my house"). As another example, the natural language user input may be "call him back." Because "to him" is a contextual parameter and does not specify a specific contact, the user device may determine the contact intended by "to him" and specify the contact as the parameter (i.e., in place of "to him").

[0217] In some examples, the identified intents and parameters may be embodied as Intent objects. Each such embodied Intent object is an object (e.g., a data structure, a programming object, etc.) that corresponds to a respective intent. Each Intent object may have one or more fields (e.g., instance variables), each corresponding to one or more parameters. For example, an Intent object corresponding to the intent to book a ride may be represented by the following pseudocode: public final class RideBookingIntent{ public final static String INTENT_TYPE=”RideBookinglntent”; public final parameter <location>pickupLocation; public final parameter <location>dropOffLocation; public final parameter <datastring>vehicleType; public final parameter <integer>partySize; public final parameter <datetime>pickupTime; private RideBookingIntent(Location p,Location d,dataString v,Integer p,dateTime pt){ pickupLocation=p; dropOffLocation=d;v vehicleType=v; partySize=p; pickupTime=pt;}} It will be appreciated by those skilled in the art that the above pseudocode is exemplary and that an Intent object may be embodied in other ways.

[0218] By embodying an intent as an Intent object, the intent may be language-independent. As described above, the intent may be derived from natural language user input. Thus, the same intent may be derived from natural language input provided in any number of speech languages. For example, an English natural language user input of "Send an Uber to 1200 Park Avenue" and a German natural language user input of "Senden Sie eine Uber auf 1200 Park Avenue" would each result in the same intent being identified (and result in the same Intent Object).

[0219] At block 815, software applications associated with the intent may be identified (e.g., selected). Generally, this may involve identifying one or more software applications configured to perform a task corresponding to the intent.

[0220] In some examples, identifying software applications may include determining one or more domains corresponding to the intent and identifying applications corresponding to the domains. In some examples, one or more machine learning mechanisms (such as neural networks) are used to identify one or more candidate domains. For example, a relevance score may be assigned to each of the multiple candidate domains, and the multiple candidate domains may be ranked using the relevance scores. The relevance scores may be determined by providing text of the natural language user input and / or context of the natural language user input to a neural network. The one or more highest-ranked candidate domains may be determined as the one or more domains associated with the intent. In some cases, the relevance scores of two or more of the highest-ranked candidate domains may be within a threshold range. The user device may distinguish between these candidate domains using an ontology, such as ontology 760.

[0221] As described above, one or more software applications may be assigned with an application registration service. Registering in this manner may include specifying which of the domains (e.g., ride booking domain, air travel domain, navigation domain, etc.) the software application corresponds to. An application corresponding to a domain may target each of the intents identified by the domain, or may target only a subset of the intents identified by the domain. In some examples, applications may be assigned individual intents, and identifying the application may include identifying the application that corresponds to the identified intent.

[0222] In some examples, applications are identified according to identified parameters. For example, based on a user input such as "Get me a black car to the airport," parameters specifying an intent to reserve a car using a ride booking service and the type of car (i.e., "black car") may be identified. While several available applications may be configured to reserve cars generally, only applications configured to reserve "black cars" may be identified. In another example, based on a user input such as "Message Sam 'Hi!'," several applications may be configured to send messages, but only applications with contact information for contacting Sam may be identified.

[0223] In some examples, only applications installed on and / or accessible to the user device may be identified. For example, several available applications may be configured to perform a task, but only applications accessible to the user device may be identified. Accessible applications include applications resident and / or installed on the user device, and also include applications remotely accessible by the user device, such as on one or more other devices.

[0224] Thus, in at least some embodiments, the identified application is an application that is configured to perform the task according to the identified parameters and that is accessible by the user device. In some examples, multiple applications may meet this criteria, but the user device may desire to identify fewer applications or only one application. Thus, applications may be further identified based on previous use of the application by the user device. In some examples, for a given intent, the application last used to perform the task corresponding to the intent may be identified. Considering a user input of "call Rob," the intent (i.e., to call) and the parameter "i.e., 'Rob'" may be identified. In this example, the application last used to make a call is identified. In other examples, the application most frequently used to perform the task corresponding to the intent is identified. In the same example, the application most frequently used to make a call is identified. In some examples, applications are further selected according to one or more parameters. For example, the application last used to call Rob's contact or the application most frequently used to call Rob's contact is identified. In some examples, a default application may be specified for one or more particular tasks and / or parameters, e.g., by a user or a digital assistant. A user may, for example, specify that a first application be used when calling a first contact and a second application be used when calling a second contact.

[0225] As described above, the natural language user input may include custom vocabulary that can be specified as one or more parameters. In some examples, such custom vocabulary includes an application name, and thus, an application may be identified based on the presence of the custom vocabulary in the input. For example, the natural language user input may be "Call Rob on Skype®." In response, the software application Skype® may be the identified software application. In another example, the natural language user input may be "Play The Beatles on Spotify®." In response, the software application Spotify® may be the identified software application.

[0226] A custom vocabulary may further include terms uniquely associated with an application. Accordingly, such terms may be identified as parameters and, optionally, used to identify the application. In the example "Connect me to UberX®," Uber® is identified as the software application because "UberX®" is a term in the custom vocabulary for the software application Uber®. In the example "Tweet that I hope the Sharks win," Twitter® is identified as the software application because "Tweet" is a term in the custom vocabulary for the software application Twitter®.

[0227] The custom vocabulary may further include terms that are uniquely associated with a user. Such terms may then be specified as parameters and, optionally, used to identify an application. In the "call my boss" example, "boss" is a user-specific term in the custom vocabulary and may therefore be specified as a parameter to identify, for example, the user's contact list.

[0228] In some examples, a user device may receive custom vocabulary from another device. Thus, the user device may be configured to identify vocabulary not otherwise known to the device. In some examples, the user device may retrieve the custom vocabulary periodically and / or in response to establishing a connection with another device. In other examples, the user device may retrieve custom vocabulary from another device in response to determining that the user has provided a term that it does not recognize.

[0229] In some examples, any application configured to perform a task according to the specified parameters may be inaccessible to the user device. As a result, the user device may access (e.g., download and / or install) an application configured to perform a task according to the specified parameters. In some examples, the user device may identify multiple software applications and present the list of software applications to the user. The user may select one or more of the multiple applications, and the user device may access the one or more selected applications.

[0230] The intent and parameters are provided to the identified software application at block 820. In some examples, the intent and parameters are provided to the software application as an Intent object.

[0231] In some examples, the intents and parameters may be provided to the software application based on the state of the user device. As one example, the intents and parameters may be selectively provided to the user device based on whether the user device is in a locked state. In some examples, the application may be permitted to receive particular intents and parameters when the device is in a locked state. In other examples, the application may be permitted to receive particular intents and parameters when the user device is not in a locked state. Whether the application is capable of receiving particular intents when the user device is in a given state may be specified by the software application, for example, during a registration process using an application registration device.

[0232] At block 825, the user device may receive one or more responses from the software application. In some examples, the user device receives a response for each parameter provided to the software application. Each response may indicate, for example, whether a parameter is valid or whether further user input is required. If the response indicates that a parameter is valid, no further action is taken with respect to the parameter.

[0233] If the response provided by the software application does not indicate that the parameters are valid, the response may indicate that parameter clarification is required. For example, the parameters may be inappropriate (i.e., invalid), and the software application may request further input from the user. As an example, consider a user input of "Call a blue car at 1200 Main Street." The user is requesting a blue car, but a ride-booking application (e.g., Uber®, Lyft®, etc.) may not allow the blue car to be selected (e.g., the application determines that blue cars are not a supported parameter or that blue cars are currently unavailable). Thus, if the parameters are inappropriate (e.g., the user specifies an invalid car type), the application may request that an appropriate (e.g., valid) value for the parameter (e.g., car type) be provided. For example, referring to FIG. 10A , based on a response from the software application indicating that the parameters are inappropriate, the user device may provide a natural language query 1002 prompting the user to select a valid parameter. In this example, the user device provides a natural language query asking the user to select a valid car type. A natural language query 1002 may be provided to the user as text using a touch-sensitive display of the user device and / or as speech using an audio output component of the user device. As shown, in some examples, the user device may present (e.g., display) one or more candidate parameters 1004 to the user for selection. In this example, the candidate parameters 1004 may include "budget," "black car," "SUV," and "share." In some examples, the candidate parameters provided in this manner may be provided by a software application.The user device may select one of the candidate parameters by providing touch input to the user device and / or by providing natural language user input, and in response, the user device may provide the selected candidate parameter to the software application.

[0234] If the response provided by the software application does not indicate that the parameters are valid, the response may indicate that parameter disambiguation is required. For example, for a user input of "Call Tom," a software application configured to place a call may be provided with parameters for contacting Tom. In determining contact information (e.g., a phone number) for contacting Tom, if the software application determines that there are multiple contacts with the name "Tom," the software application may request an indication of which "Tom" is intended.

[0235] The software application may request the user device to disambiguate a parameter based on user input. As part of the request, the software application optionally includes in its response a disambiguation list containing multiple candidate parameters. If the software application requests disambiguation based on user input, the device may provide a natural language query asking the user to select a candidate parameter. Further, the candidate parameters in the disambiguation list may be displayed for the user to select. The user may select one of the multiple candidate parameters, for example, by providing touch input and / or natural language user input, and the user device may provide the selected candidate parameter to the software application.

[0236] The software application may request that the user device disambiguate the parameter without user input. Thus, the user device may automatically disambiguate the parameter based on the context of the user device, the software application, and / or the parameter. For example, the user device may select "Tom," the contact the user last contacted.

[0237] In some cases, a software application may not be able to access the information necessary to parse and / or disambiguate parameters. The software application may not be able to access a contact list and therefore may not be able to determine whether multiple entries or even a single entry exists for a name. As a result, the software application may request that the user device disambiguate any parameters that the software application is unable to parse. Additionally or alternatively, the software application may request that the user device provide the software application with candidate parameters so that the software application can perform the disambiguation. Referring to the example above, the software application may request that the device disambiguate "Tom" or provide a list of all of "Tom's" contacts so that the software application can disambiguate "Tom" itself.

[0238] Performing a task may require one or more specific types of parameters to be specified by the user. Furthermore, in some examples, the natural language user input may omit one or more required parameters. Thus, the software application may optionally provide one or more responses indicating which, if any, required parameters have not been specified. Consider the user input "Call a car at 1200 Main Street." While the user input typically requests a car, a ride-booking application identified based on the user input may require a specific car model to be selected. Thus, if a parameter is missing (e.g., the user has not specified a car model), the application may require that a recognizable value for the parameter (e.g., a car model) be provided. Referring again to FIG. 10A , the user device may provide a natural language query 1002 prompting the user to specify valid parameters. The user may then select candidate parameters, for example, from a list of candidate parameters, and the selected parameters may be provided to the software application as described above.

[0239] Once the software application indicates that each parameter is valid and that no further information is required, the user device may confirm the intent with the software application at block 830. In particular, the user device may request notification that the software application can perform the task corresponding to the intent, given the intent and the parameters associated with the intent.

[0240] When the software application provides a notification indicating that the software application is able to perform the task, optionally, the user device confirms intent to the user. For example, the user device may provide a natural language query such as, "I can call uberX® to your location. Would you like to sign up?" The user may confirm or deny intent via touch input or natural language input. In some examples, the notification provided by the software application may include information to provide to the user. The information may, for example, enable the user to make a more detailed decision when prompted for confirmation. For example, the user device may provide a natural language query such as, "I can call uberX® to your location in 9 minutes. Would you like to sign up?"

[0241] The user device then causes (e.g., commands) the software application to perform a task corresponding to the intent.

[0242] In other examples, when a software application provides a notification indicating that the software application is able to perform the task, the user device causes the software application to perform the task without user confirmation. For example, if a user provides natural language user input such as "call Lyft® to the airport," and the identified application, Lyft®, indicates that it can reserve a car for the user, the software application automatically performs the task without user confirmation. In some examples, whether a task is performed without user confirmation is based on the type of user device. A first type of device, such as a mobile phone, may require user confirmation for one or more tasks, while a second type of device, such as a smartwatch, may perform one or more tasks without first receiving user confirmation.

[0243] At block 835, the user device receives a result response from the software application indicating whether the software application successfully performed the task.

[0244] A result response indicating a failure to perform a task may further indicate one or more reasons for the failure. In some examples, the user device may provide an output, such as a natural language output or a text output, to the user indicating the one or more reasons for the failure.

[0245] A result response indicating successful execution of a task may include one or more response items. Each response item may be a result determined (e.g., received, generated, etc.) by a software application during execution of a task. For example, response items corresponding to reserving a car in a ride-booking application may include the car model, license plate number, driver name, current car location, pickup location, destination, estimated travel time, estimated cost and route, type of service (e.g., Uberpool® vs. uberX®), etc. As another example, response items corresponding to starting a workout session in a fitness application may include confirmation that the session has started, the duration of the workout, the type of activity, and one or more goals.

[0246] In some examples, one or more of the plurality of response items may be provided to the user. For example, referring to FIG. 10B , one or more of the plurality of response items may be provided to the user as natural language output 1012, as text input and / or audio output. The response items may also be provided visually. For example, a map 1014 of an estimated route may be provided to the user. It will be appreciated that the response items may be provided to the user in any desired manner.

[0247] In some examples, a software application may specify how one or more response items are provided to a user. That is, the software application may determine which response items are displayed and / or pronounced to the user, and the digital assistant may provide each response item accordingly.

[0248] In some examples, a software application may specify how response items are provided using the digital assistant's UI extension. For example, a user device may provide a set of view controller parameters (such as fields that can be provided to a view controller for display) to the software application, and the software application may respond by providing a set of view controller parameter values. The set of view controller parameters may indicate which response items are displayed in various fields of the view controller and / or how the response items are displayed in each field. In this way, the digital assistant may control (e.g., limit) how the response items are displayed by the software application.

[0249] In other examples, the digital assistant may determine how the response item is provided. In yet other examples, a software application is invoked to allow the user to interact directly with the software application. In some examples, invoking the application in this manner may terminate the session with the digital assistant.

[0250] In some examples, upon successful execution of a task, a user device (e.g., a digital assistant on the user device) may provide one or more notifications to the user indicating the status of the action associated with the task. As an example, once a car is reserved in response to user input, the user device may provide the location, estimated arrival time, and / or route of the reserved car. The notification may further indicate whether an error occurred in connection with the action. For example, the user may be notified if the reserved car becomes unavailable. In some examples, such provided notifications may be provided periodically and / or in real time.

[0251] In some examples, upon successfully completing a task, the user device may adjust an action associated with the task. Adjusting the action associated with the task may include adjusting parameters associated with the intent (e.g., by removing, adding, and / or modifying) and providing the intent with the adjusted parameters to a software application. For example, the software application may reserve a car and indicate to the user that the car reservation was successful. The car reservation may then be adjusted by specifying a new destination for the ride. In other examples, adjusting the action includes providing a different intent to the software application. The intent may be, for example, an intent for a different domain.

[0252] In some examples, actions are adjusted in response to a selection of an affordance displayed by the device. As one example, upon successful completion of a task, the device may display one or more user-selectable affordances to acknowledge successful completion of the task and / or indicate how the user desires to adjust actions associated with the task. In some examples, affordances may be displayed based on context. As one example, an affordance for booking a ride may allow a user to change the destination address or cancel the ride ( FIG. 13A ). As another example, an affordance for making a dinner reservation may allow a user to change the reservation time or number of people. As yet another example, an affordance for route guidance may allow a user to change the destination address or transportation method. As yet another example, an affordance for making an electronic payment may allow a user to cancel the payment or change the payment amount.

[0253] In some cases, a software application may fail to perform a task. In response, the device may display one or more affordances that the user can select to indicate the task failure and / or to request the task be performed. As one example, selection of an affordance may cause the user device to perform the same task that previously failed. This may include providing the same intent to the software application. The intent may have the same or different parameters and / or parameter values. As another example, selection of an affordance may cause the device to perform a different task, such as a task associated with a different domain.

[0254] As an example, during operation, a software application may fail to complete a payment in response to a user request ( FIG. 13B ). In response, the device may display a first affordance, and in response to selecting the first affordance, the user device may provide an intent to the software application. The intent may be the same intent previously provided to the software application (i.e., an intent to make a payment). The user device may further display a second affordance, and in response to selecting the second affordance, the user device may provide the intent to a different software application. The user device may further display a third affordance, and in response to displaying the third affordance, the user device may provide the intent to send a message to another software application. For example, a user may wish to communicate a failed payment.

[0255] In some examples, the software application's permission is verified, e.g., before the intent and parameters are provided to the software application. For example, a user device may determine whether the software application has permission to access data associated with a particular intent. The determination may be made based on permissions configured on the user device. In the "call a black car to where I am right now" example, "location" may be a contextual parameter that requires contextual information (e.g., location data) from the user device. Thus, before analyzing the user device's location and providing the location as a parameter to the software application, the user device may first determine whether the software application is authorized to access the information. If the software application is authorized to access the information, operations are performed as described above. If the software application is not authorized to access the information, the intent and parameters are not provided to the software application and the task is not performed. In some cases, software application authorization may occur across multiple devices. For example, a first device having a software application may grant permission to a second device of the user having the same software application.

[0256] In some examples, the natural language input may include multiple task requests. Thus, multiple intents and / or multiple applications may be identified based on the natural language input. Parameters associated with each of the multiple intents may also optionally be identified. For example, the natural language input "Order a ride to the airport and get the status of my flight" may include both an intent to book a car and an intent to retrieve the status of the user's flight. In some examples, tasks corresponding to each of these intents may be performed sequentially or simultaneously.

[0257] In some instances, a natural language input may include multiple related task requests. For example, in some instances, a requested task of a natural language user input may depend on the completion of another requested task of the natural language input. In the example, "Email me directions to the airport," two tasks are requested: the first task is to provide directions, and the second task is to send the email. The parameter "me" specifies a specific contact for the intent to send the email, and the parameter "airport" specifies a destination for the intent to provide directions. Because emailing directions requires first providing directions, the second task (sending the email) is dependent on the first task (providing directions). Therefore, the task of retrieving directions is performed first.

[0258] In some examples, intents may be exchanged between applications. For example, one application may provide an intent object to another application to cause the application to perform a task. In the example above, both intents (to provide directions and to send an email) may be provided to the map application, which may provide the requested directions. For example, the second intent, or the intent to send an email, may be provided as a parameter to the map application. The map application may then provide the requested directions according to the first intent, or the intent to provide directions. The map application may then provide the second intent, for example, as an intent object including the directions as a parameter, to the email application. In response, the email application may provide the requested directions via email.

[0259] As another example, while watching a sports application (such as an ESPN application), a user may provide user input such as "Get me a ride to go to the Sharks game." In response, the sports application, which has information about the game, may forward the intent (e.g., to book a car) and parameters (e.g., the address of the game) to a ride booking application. In some cases, the intent and parameters may be provided as an intent object.

[0260] In yet another example, a user may provide user input while using a ride-booking application, such as "Pay my brother $5." In response, the ride-booking application may forward the intent (e.g., to pay) and parameters ($5) to a payment application (e.g., PayPal, Venmo, an online payment service, etc.). As noted above, in some examples, the intent and parameters may be provided as an intent object.

[0261] In some examples, intents may be provided between multiple devices. As described above, multiple devices of different types may be configured to perform different tasks. In some cases, natural language user input may be provided to a first device that is not configured to perform a task corresponding to the intent of the natural language user input. The first device may provide the intent to a second device so that a task associated with the intent can be performed. As an example, a first device (e.g., a mobile phone) may be configured to reserve a car, and a second device (e.g., a smartwatch) may be configured to provide the status of the reserved car. When a user uses the second device to request a car reservation (e.g., "Order an Uber to the airport"), the second device may derive an intent from the user input and determine that the second device cannot perform the derived intent. The second device may then provide the derived intent to the first device so that the task can be performed by the first device.

[0262] As described above, the device may provide natural language output to the user during operation. For example, a natural language query may be provided to the user requesting parameter confirmation. In some examples, the natural language output may be provided to the user according to a task flow associated with one or more software applications. The task flow may specify, for example, the manner and type of natural language output to be provided. For example, the user may provide natural language input (such as, "Order a Lyft® to my house"). After the software application receives the intent and any specified parameters, it may indicate that disambiguation of one or more parameters is required and / or that additional parameters (such as the user's location, "home" address, vehicle type, etc.) are required to perform the task associated with the intent. Thus, the device may provide the user with one or more task-specified natural language queries so that all required parameters are provided to the software application.

[0263] In some examples, one or more task flows are stored on a device. Thus, the device may provide natural language output (e.g., natural language queries) to a user without retrieving individual queries of the task flows from an external device during operation. Storing task flows in this manner may improve operational efficiency. In some examples, task flows may be retrieved and / or stored on a user device if the corresponding software application is stored on the device.

[0264] FIG. 9 illustrates a flow diagram of a process for operating a digital assistant according to some embodiments. Process 900 may be used to implement at least a portion of process 800 of FIG. 8, including, for example, but not limited to, blocks 815 and / or 820 of FIG. 8. Process 900 is performed using one or more electronic devices (e.g., devices 104, 108, 200, 400, or 600) that implement a digital assistant. In some examples, process 900 is performed using a client-server system (e.g., system 100), and blocks of process 900 may be divided in any manner between a server (e.g., DA server 106) and a client device. Thus, while portions of process 900 are described herein as being performed by a particular device in a client-server system, it will be understood that process 900 is not so limited. In other examples, process 900 is performed using only one client device (e.g., user device 104). Some blocks are optionally combined, the order of some blocks is optionally changed, and some blocks are optionally omitted in process 900. In some examples, additional steps may be performed in combination with process 900.

[0265] At block 905, natural language user input is received by a user device, such as user device 104 of Figure 1. As described above, the natural language user input may include a request to the user device and / or another device to perform a task, and may further specify one or more parameters of the requested task.

[0266] At block 910, an intent and, optionally, one or more parameters associated with the intent are identified. The intent and parameters may be derived from natural language user input. As previously described, the intent may correspond to any type of task performed by the user device, and in particular, may correspond to a task performed by one or more applications on the user device. The parameters associated with the intent may identify portions of the natural language input that specify how the task corresponding to the intent is to be performed. In the example of "Get me a ride to the airport," the intent corresponds to the task of booking a car, and "airport" is a parameter that specifies the destination. Since the user's intent is to book a car, the user's location may also be a parameter (e.g., an inferred parameter).

[0267] At block 915, it is determined whether a task corresponding to the intent can be performed. In some examples, the determination may include determining whether an application configured to perform the task according to the parameters is accessible to the user device. In this example, it is determined whether an application configured to reserve a car at the user device's location is accessible to the user device. As described above, accessible applications include applications stored locally on the user device and applications that can be accessed remotely by the user device.

[0268] Following a determination that the task corresponding to the intent is fulfillable, the intent and parameters are provided to a software application at block 920. For example, if it is determined at block 915 that the user device has access to a software application configured to perform the requested task according to any specified parameters, the intent and parameters are provided to the application to perform the task. In this example, this includes determining that a software application configured to reserve a car at the user's location is accessible to the user device. For example, the ride booking application Lyft® may be installed on the user device and used to reserve a car according to the examples described herein.

[0269] Following a determination that the task corresponding to the intent is unfulfillable, a list of one or more software applications is provided at block 925. The list of one or more software applications may include, for example, one or more software applications configured to perform the task associated with the intent according to any specified parameters. In some examples, one or more software applications in the list may be identified based on one or more domains associated with the intent (as described above, applications may be registered in one or more domains). Referring to FIG. 10C , once the list of software applications (e.g., ride booking applications) has been determined, the user device may provide the list to the user. As shown, providing the list may include providing a natural language input 1022 requesting the user to select an application from a list of one or more application software. In some examples, the list is generated by the user device. In other examples, the list of software applications may be generated by a server and provided to the user device, which then provides the list to the user as described above.

[0270] At block 930, the user device receives user input indicating a selection of one or more software applications from the list of one or more software applications. The user input may be touch input on a touch-sensitive display of the user device and / or may be natural language user input.

[0271] The intent and parameters are provided to the software application selected by the user at block 935. In some examples, providing the intent and parameters includes downloading and / or installing the software application so that the user device can access the software application locally. In other examples, this includes remotely accessing the selected application.

[0272] As described above, depending on the intent and parameters, the software application may provide one or more responses. After the parameters are verified, the user device may confirm the intent with the software application and cause the software application to perform a task. The user device may then receive a result response and, optionally, provide one or more response items of the result response to the user.

[0273] Providing natural language output and / or natural language queries to a user of a user device is described herein. In some examples, the manner in which the natural language output and queries are provided to the user depends on the type or state of the user device. For example, if the user device is a mobile phone, the user device may provide queries using both text and audio. On the other hand, if the user device is a speaker, the user device may provide queries using only audio. As another example, if the user device is a mobile phone without headphones paired, the user device may provide queries using text and / or relatively short natural language queries. If headphones are paired to the user device, the user device may provide queries using only relatively long natural language queries.

[0274] 10D-F illustrate example data flows for a digital assistant according to some embodiments. In some examples, the data flows of FIGS. 10D-F may be implemented using one or more of processes 800 and 900. FIG. 10D illustrates an example data flow for a digital assistant system 1030 according to some embodiments. In particular, FIG. 10D illustrates data flows for an application registration process and data flows for task performance. Data flows 1031-1041 may be associated with the application registration process (illustrated as dashed lines in FIG. 10D), and data flows 1043-1051 may be associated with task performance (illustrated as solid lines in FIG. 10D).

[0275] In general, the data flow associated with the application registration process involves registering an application with an application registration service (such as a verification service) that enables the application and its corresponding custom vocabulary to be accessed and / or used by the digital assistant to perform tasks.

[0276] In operation, in data flow 1031, an application is submitted to application review module 1032. A language model corresponding to the application and the intent of the application may also be submitted. The language model may include a custom vocabulary for the application. Then, in data flow 1033, application review module 1032 may provide the application, the custom vocabulary, and / or the intent of the application to validation service 1034. The validation service 1034 may determine whether to validate the application based, for example, on whether the application is operable on a digital assistant. This may include, for example, verifying that any intent of the application corresponds to one or more domains of the application. For example, the validation service may reject a messaging application associated with the intent to reserve a car because the domain and intent are an inappropriate combination. The validation service 1034 may provide a validation response, in data flow 1035, indicating whether the application is valid.

[0277] If the validation service 1034 indicates that the application is valid, the application review module 1032 provides the (validated) application to the application store 1036. In general, the application may be downloaded and / or accessed by the user device 1040 via the DA server 1038 at the application store 1036, as shown by data flow 1039. In some examples, the user device 1040 may be the user device 104 of FIG. 1, and the DA server 1038 may be the DA server 106 of FIG. 1. This may result, for example, in the application list (e.g., info.plist) of the user device 1040 being updated and / or synchronized with the DA server 1038. In data flow 1041, the validation service may provide the application's custom vocabulary (e.g., runtime vocabulary) to the DA server 1038 to assist in parsing natural language input, as described above.

[0278] In general, the data flow associated with performing a task involves providing an intent and, optionally, one or more parameters to an application for the execution of a task corresponding to the intent.

[0279] In operation, in data flow 1043, the user device 1040 may provide natural language input to the DA server 1038. In some examples, the natural language input may be provided by the digital assistant 1042 of the user device 1040. Based on the natural language input, the DA server 1038 may identify one or more parameters associated with the intent and one or more tasks requested in the natural language input. Additionally, the DA server 1038 may identify an application for performing the task by the intent. In some examples, the name (or other form of identification) of the identified application 1044 may be a parameter of the intent. Then, in data flow 1045, the DA server 1038 provides the intent, the parameters, and the identification of the identified application 1044 to the user device 1040 (e.g., the digital assistant 1042 of the user device 1040). In some examples, the intent and parameters may be provided to the user device 1040 as an intent object.

[0280] In response, the digital assistant determines whether the identified application 1044 is authorized to access information associated with the identified parameter. For example, if the parameter is the location of the user device 1040, the digital assistant queries the data permission unit 1046 to determine whether the application 1044 is authorized to access location data.

[0281] If the application is authorized to access the data for each of the parameters, the user device (e.g., the user device's digital assistant 1042) provides the intent to the application. As shown, the application may reside on the user device 1040. In other examples, the application may reside on one or more other devices, and the intent may be sent to the application over one or more networks. As described above, the application 1044 may then request input from a user of the user device 1040 if one or more parameters are missing, incorrect, and / or unclear. In some examples, the query for user input may be provided as a natural language query generated by the DA server 1038. Thus, in data flow 1051, the user device 1040 may request and subsequently receive one or more natural language queries. After all parameters have been parsed, the application 1044 may perform the task associated with the intent and provide a result response indicating whether the task was successfully performed.

[0282] 10D may be embodied using, for example, one or more electronic devices (such as devices 104, 108, 200, 400, or 600) that embody a digital assistant. In particular, the data flow provided between DA server 1038 and digital assistant 1042 of user device 1040 is illustrated as relying on a client-server architecture. In other examples, DA server 1038 may be embodied as a process and / or service on user device 1040. Thus, in some examples, data flows exchanged between DA server 1038 and digital assistant 1042 may be exchanged only on user device 1040.

[0283] FIG. 10E illustrates an example data flow for a digital assistant system 1060 according to some embodiments. In particular, FIG. 10E illustrates an example data flow for an application registration process, which may be used to implement the application registration process shown in FIG. 10D. Furthermore, some elements in FIG. 10E correspond to elements in FIG. 10D and are given the same reference numerals. For brevity, these functions and operations will not be described again.

[0284] In data flow 1065, the verified vocabulary is provided from the verification service 1034 to the global application vocabulary store 1060. In general, the global application vocabulary store may store language models and / or vocabularies for any number and / or versions of software applications. In data flows 1061 and 1063, a speech training module 1062 and a natural language training module 1064 are trained to recognize and natural language process the application-specific vocabulary provided with the verified application. Based on the data, the global application vocabulary store may generate and / or train one or more language models that enable the digital assistant to recognize and process utterances that include the application-specific vocabulary.

[0285] During operation of a user device, such as user device 1040 of FIG. 10D , a runtime-specific global application vocabulary store may receive vocabulary and / or language models for one or more applications of the user device from global application vocabulary store 1060. The vocabulary may be specific to a user ID of a user of the user device and / or specific to an application and / or operating system version of the user device. Based on the vocabulary, one or more terms of the natural language input may be identified, for example, as parameters.

[0286] 11 is a flow diagram of a process 1100 for operating a digital assistant according to some embodiments. Process 1100 is performed, for example, using one or more electronic devices (such as devices 104, 108, 200, 400, or 600) embodying a digital assistant. In some examples, process 1100 is performed using a client-server system (such as system 100) and may be distributed in any manner between a server (such as DA server 106) and a client service. In other ways, process 1100 may be distributed between a server and multiple client devices (such as a mobile phone and a smartwatch). Thus, although portions of process 1100 are described herein as being performed by a particular device in a client-server system, it will be understood that process 1100 is not so limited. In other examples, process 1100 is performed using only one client device (such as user device 104) or multiple client devices. Some blocks are optionally combined, the order of some blocks is optionally changed, and some blocks are optionally omitted in process 1100. In some examples, additional steps may be performed in combination with process 1100.

[0287] At block 1105, an electronic device receives audio input including natural language user input.

[0288] At block 1110, the electronic device identifies an intent object from the set of intent objects. The intent object is derived from the natural language user input. In some examples, identifying the object from the set of intent objects includes identifying the object from the set of intent objects based on a type of the electronic device.

[0289] At block 1115, the electronic device identifies a software application associated with an intent object of the set of intent objects. In some examples, identifying the software application associated with an intent object of the set of intent objects includes identifying a first parameter associated with the intent object.

[0290] In some examples, the electronic device receives a plurality of terms from another electronic device, receiving the natural language user input includes receiving the natural language user input including a term of the plurality of terms, and identifying a software application associated with an intent object of the set of intent objects includes identifying the software application based on the term of the plurality of terms.

[0291] At block 1120, the electronic device provides the intent object to the software application. Providing the intent object causes the software application to perform a task associated with the intent object. In some examples, providing the intent object to the software application to cause the software application to perform the task associated with the intent object includes providing a first parameter to the software application. In some examples, providing the intent object to the software application includes, after identifying the software application, determining whether user input requesting cancellation or modification of the task has been received within a certain period of time, and providing the intent object to the software application in accordance with a determination that user input requesting cancellation or modification of the task has not been received within a certain period of time. In some examples, providing the intent object to the software application includes providing the intent object to the software application without user confirmation of the intent object.

[0292] In some examples, the electronic device identifies multiple candidate parameters associated with the intent and provides the multiple candidate parameters to the software application.

[0293] In some examples, the electronic device receives a disambiguation request from a software application corresponding to a parameter, identifies a candidate parameter from a set of candidate parameters associated with the parameter, and provides the selected candidate parameter to the software application.

[0294] In some examples, the natural language user input is the first natural language user input, and the electronic device receives a request from the software application for a second parameter associated with the intent object, provides a natural language query based on the request, receives the second natural language user input, identifies a second parameter derived from the second natural language user input, and provides the second parameter to the software application.

[0295] In some examples, the electronic device receives a request from the software application for parameters associated with the intent object, uses the electronic device to identify a natural language query associated with the software application and the parameters based on the request, provides the natural language query, receives a fourth user input, identifies parameters based on the fourth user input, and provides the parameters to the software application.

[0296] A result response indicating whether the task was successfully executed is received by the electronic device at block 1125. The result response may be received from the software application.

[0297] At block 1130, the electronic device provides an output indicating whether the task was performed. In some examples, the output indicates that the software application successfully performed the task. In some examples, the output indicates that the software application failed to perform the task. In some examples, providing the output indicating whether the task was performed includes receiving a set of view controller parameter values from the software application and providing the output according to the set of view controller parameter values.

[0298] In some examples, after providing the output, the electronic device receives a second input and adjusts an action associated with the task in response to the second input. In some examples, adjusting the action associated with the task includes adjusting an Intent Object and providing the adjusted Intent Object to the software application. In some examples, adjusting the Intent Object includes adjusting a parameter of the Intent Object. In some examples, receiving the second input includes detecting a selection of an affordance, and adjusting the action associated with the task includes adjusting the action associated with the task in response to the selection of the affordance.

[0299] In some examples, after providing the output, the electronic device receives a third input, and in response to the third input, provides another intent object to the software application to cause the software application to perform a task associated with the other intent object. In some examples, receiving the third input includes detecting a selection of an affordance, and providing another intent object to the software application to cause the software application to perform a task associated with the other intent object includes providing the other intent object to the software application in response to the selection of the affordance. In some examples, the intent object and the other intent object are the same type of intent object, and a parameter associated with the intent object and a parameter associated with the other intent object have the same value. In some examples, the intent object and the other intent object are the same intent object.

[0300] 12 is a flow diagram of a process 1200 for operating a digital assistant according to some embodiments. Process 1200 is performed, for example, using one or more electronic devices (such as devices 104, 108, 200, 400, or 600) implementing a digital assistant. In some examples, process 1200 is performed using a client-server system (such as system 100), and blocks of process 1200 may be distributed in any manner between a server (such as DA server 106) and client devices. In other examples, process 1200 is distributed between a server and multiple client devices (such as a mobile phone and a smartwatch). Thus, while portions of process 1200 are described herein as being performed by a particular device in a client-server system, it will be understood that process 1200 is not so limited. In other examples, process 1200 is performed using only one client device (such as user device 104) or multiple client devices. Some blocks are optionally combined, the order of some blocks is optionally changed, and some blocks are optionally omitted in process 1200. In some examples, additional steps may be performed in combination with process 1200.

[0301] At block 1205, the electronic device identifies an intent object from a set of intent objects, the intent object being derived from the natural language user input.

[0302] At block 1210, the electronic device identifies a software application associated with an intent object from a set of intent objects.

[0303] At block 1215, the electronic device receives a task flow associated with the software application from the second electronic device. In some examples, receiving the task flow associated with the software application includes receiving the task flow from another electronic device and storing the task flow on the electronic device.

[0304] At block 1220, the electronic device provides the intent object to the software application, causing the software application to perform the task associated with the intent object.

[0305] At block 1225, the electronic device receives a request from the software application.

[0306] At block 1230, the electronic device determines a query based on the request. The query is determined based on the task flow. In some examples, determining the query based on the request includes retrieving the query from a task flow stored on the electronic device.

[0307] At block 1235, the electronic device provides a query using the touch-sensitive display.

[0308] At block 1240, the electronic device receives a second user input indicating a query response. In some examples, the second user input is received after providing the query. In some examples, the query response indicates parameters associated with the intent object.

[0309] The electronic device provides the query response to the software application at block 1245. In some examples, providing the query response to the software application includes adjusting an Intent Object and providing the adjusted Intent Object to the software application.

[0310] In some examples, the electronic device receives a result response from the software application indicating whether the task was successfully performed and provides an output indicating whether the task was performed.

[0311] According to some embodiments, a functional block diagram of an electronic device 1400 configured in accordance with the principles of various described embodiments, including those described with reference to FIG. 11, is shown in FIG. 14. The functional block diagram of the device is optionally embodied in hardware, software, or a combination of hardware and software to carry out the principles of various described embodiments. Those skilled in the art will understand that the functional blocks described in FIG. 14 may be optionally combined or divided into multiple sub-blocks to embody the principles of various described embodiments. Thus, the description herein optionally supports any anticipated combination or division or further definition of the functional blocks described herein.

[0312] 14 , electronic device 1400 includes a touch-sensitive display unit 1402 and a processing unit 1408 optionally coupled to touch-sensitive display unit 1402. In some embodiments, processing unit 1408 includes a receiving unit 1410, an identifying unit 1412, a providing unit 1414, and, optionally, an adjusting unit 1416, a detecting unit 1418, and a determining unit 1420.

[0313] In some examples, the processing unit 1408 is configured to receive (e.g., using the receiving unit 1410) an audio input including natural language user input; identify (e.g., using the identifying unit 1412) an intent object from a set of intent objects that is derived from the natural language user input; identify (e.g., using the identifying unit 1412) a software application associated with an intent object from the set of intent objects; provide (e.g., using the providing unit 1414) the intent object to the software application to cause the software application to perform a task associated with the intent object; receive (e.g., using the receiving unit 1410) a result response from the software application indicating whether the task was successfully performed; and provide (e.g., using the providing unit 1414) an output indicating whether the task was performed.

[0314] In some examples, identifying a software application associated with an intent object of the set of intent objects includes identifying a first parameter associated with the intent object (e.g., using the identifying unit 1412), and providing the intent object to the software application to cause the software application to perform a task associated with the intent object includes providing the first parameter to the software application (e.g., using the providing unit 1414).

[0315] In some examples, the natural language user input is the first natural language input, and the processing unit 1408 is further configured to receive (e.g., using the receiving unit 1410) a request from the software application for a second parameter associated with the intent object; provide (e.g., using the providing unit 1414) a natural language query based on the request; receive (e.g., using the receiving unit 1410) the second natural language user input; identify (e.g., using the identifying unit 1412) the second parameter, the second parameter derived from the second natural language user input; and provide the second parameter to the software application.

[0316] In some instances, the output indicates that the software application successfully performed the task.

[0317] In some examples, the processing unit 1408, after providing the output, is further configured to receive a second input (e.g., using the receiving unit 1410); and adjust an action associated with the task (e.g., using the adjusting unit 1416) in response to the second input.

[0318] In some examples, adjusting the action associated with the task includes adjusting an intent object (e.g., using the adjusting unit 1416) and providing the adjusted intent object to the software application (e.g., using the providing unit 1414).

[0319] In some examples, adjusting the intent object includes adjusting parameters of the intent object (eg, using the adjustment unit 1416).

[0320] In some examples, receiving the second input includes detecting a selection of an affordance (e.g., with the detection unit 1418), and adjusting the action associated with the task includes adjusting the action associated with the task in response to the selection of the affordance (e.g., with the adjustment unit 1416).

[0321] In some instances, the output indicates that the software application failed to perform a task.

[0322] In some examples, the processing unit 1408, after providing the output, is further configured to receive a third input (e.g., using the receiving unit 1410); and, in response to the third input, provide another intent object (e.g., using the providing unit 1414) to the software application, causing the software application to perform a task associated with the another intent object.

[0323] In some examples, receiving the third input includes detecting a selection of the affordance (e.g., using the detection unit 1418), and providing another intent object to the software application to cause the software application to perform a task associated with the another intent object includes providing the other intent object to the software application (e.g., using the providing unit 1414) in response to the selection of the affordance.

[0324] In some instances, the Intent object and the other Intent object are of the same type of Intent object, and a parameter associated with the Intent object and a parameter associated with the other Intent object have the same value.

[0325] In some instances, the Intent object and the other Intent object are the same Intent object.

[0326] In some examples, providing an output indicating whether the task has been performed includes receiving a set of view controller parameter values from the software application (e.g., using the receiving unit 1410); and providing an output according to the set of view controller parameter values (e.g., using the providing unit 1414).

[0327] In some examples, the processing unit 1408 is further configured to receive a plurality of terms from another electronic device (e.g., using the receiving unit 1410), wherein receiving the natural language user input includes receiving a natural language user input including a term of the plurality of terms (e.g., using the receiving unit 1410), and identifying a software application associated with an intent object from the set of intent objects includes identifying the software application based on the term from the set of intent objects (e.g., using the identifying unit 1412).

[0328] In some examples, identifying an intent object from the set of intent objects includes identifying the intent object from the set of intent objects based on a type of electronic device (e.g., using the identifying unit 1412).

[0329] In some examples, the processing unit 1408 is further configured to identify a plurality of candidate parameters associated with the intent (e.g., using the identifying unit 1412); and provide the plurality of candidate parameters to a software application (e.g., using the providing unit 1414).

[0330] In some examples, the processing unit 1408 is further configured to receive a disambiguation request corresponding to a parameter from a software application (e.g., using the receiving unit 1410), identify a candidate parameter from a set of candidate parameters associated with the parameter (e.g., using the identifying unit 1412), and provide the selected candidate parameter to the software application (e.g., using the providing unit 1414).

[0331] In some examples, providing an intent object to a software application to cause the software application to perform a task associated with the intent object includes, after identifying the software application, determining (e.g., using the determining unit 1420) whether user input requesting cancellation or modification of the task has been received within a certain time period; and providing (e.g., using the providing unit 1414) the intent object to the software application in accordance with a determination that user input requesting cancellation or modification of the task has been received within the time period.

[0332] In some examples, providing an intent object to the software application pursuant to a determination that user input requesting cancellation or modification of the task has not been received within the time period includes providing the intent object to the software application (e.g., using the providing unit 1414) without user confirmation of the intent object.

[0333] In some examples, the processing unit 1408 is further configured to: receive (e.g., using the receiving unit 1410) a request from the software application for parameters associated with the intent object; identify (e.g., using the identifying unit 1412) using the electronic device a natural language query based on the request, the natural language query being associated with the software application and the parameters; provide (e.g., using the providing unit 1414) the natural language query; receive (e.g., using the receiving unit 1410) a fourth user input; identify (e.g., using the identifying unit 1412) parameters based on the fourth user input; and provide (e.g., using the providing unit 1414) the parameters to the software application.

[0334] The operations described above with respect to Figure 11 are optionally implemented by components illustrated in Figures 1, 2A, 4, 6A-B, 7A, and 14. For example, receiving operations 1105 and 1125, identifying operations 1110 and 1115, and providing operations 1120 and 1130 are optionally implemented by processor 120.

[0335] Those skilled in the art will understand that the functional blocks described in FIG. 14 may be optionally combined or divided into sub-blocks to embody the principles of the various described embodiments. Accordingly, the description herein optionally supports any anticipated combinations, divisions, or further definitions of the functional blocks described herein. For example, processing unit 1408 may have an associated “controller” unit operatively coupled to processing unit 1408 to enable its operation. It will be understood that this controller unit is not separately illustrated in FIG. 14 but is understandable by those skilled in the art designing a device having a processing unit 1408, such as device 1400. As another example, one or more units, such as receiving unit 1410, may be hardware units external to processing unit 1408 in some implementations. Accordingly, the description herein optionally supports any combinations, divisions, and / or further definitions of the functional blocks described herein.

[0336] According to some embodiments, FIG. 15 illustrates a functional block diagram of an electronic device 1500 configured according to the principles of various described embodiments, including those described with reference to FIG. 11. The functional blocks of the device are optionally embodied in hardware, software, or a combination of hardware and software to realize the principles of the various described embodiments. Those skilled in the art will understand that the functional blocks described in FIG. 15 may be optionally combined or divided into sub-blocks to realize the principles of the various described embodiments. Thus, the description herein optionally supports any anticipated combination or division or further definition of the functional blocks described herein.

[0337] 15 , one or more devices 1500 include one or more processing units 1508 and one or more touch-sensitive displays 1502. In some embodiments, the one or more processing units 1508 include a receiving unit 1510, an obtaining unit 1512, a determining unit 1514, a providing unit 1516, and, optionally, an executing unit 1518.

[0338] In some examples, the one or more processing units 1508 are configured to: receive (e.g., using the receiving unit 1510) an audio input including natural language user input; acquire (e.g., using the acquiring unit 1512) a text string from the natural language user input; determine (e.g., using the determining unit 1514) an intent object from a set of intent objects based on the text string; determine (e.g., using the determining unit 1514) a software application associated with an intent object from the set of intent objects; provide (e.g., using the providing unit 1516) the intent object to the software application to cause the software application to perform a task associated with the intent object; receive (e.g., using the receiving unit 1510) a result response from the software application indicating whether the task was successfully performed; and provide (e.g., using the providing unit 1516) an output indicating whether the task was performed.

[0339] In some examples, determining an intent object of the set of intent objects based on the text string includes performing natural language processing on the text string (e.g., using the execution unit 1518) to determine the intent of the natural language user input and identifying an intent object associated with the intent (e.g., using the identification unit 1520).

[0340] In some examples, providing an output indicating whether the task was performed includes providing a natural language output (eg, using the providing unit 1516).

[0341] In some examples, determining a software application associated with an intent object of the set of intent objects includes determining parameters associated with the intent object (e.g., using the determining unit 1514), and providing the intent object to the software application to cause the software application to perform a task associated with the intent object includes providing the parameters to the software application (e.g., using the providing unit 1516).

[0342] In some examples, determining an intent object from the set of intent objects includes determining the intent object from the set of intent objects based on a type of the electronic device (e.g., using the determining unit 1514).

[0343] The above description of Figure 11 may optionally be implemented by the components described in Figures 1, 2A, 4, 6A-B, 7A, and 15. For example, receiving operations 1105 and 1125, identifying operations 1110 and 1115, and providing operations 1120 and 1130 may optionally be implemented by processor 120. It will be apparent to those skilled in the art how other processes can be implemented based on the components described in Figures 1, 2A, 4, 6A-B, 7A, and 15.

[0344] Those skilled in the art will understand that the functional blocks described in FIG. 15 may be optionally combined or divided into sub-blocks to implement various described embodiments. Accordingly, the description herein optionally supports any anticipated combinations, divisions, or further definitions of the functional blocks described herein. For example, one or more processing units 1508 may have an associated “controller” unit operatively coupled to at least one of the one or more processing units 1508 to enable operation. It will be understood that this controller unit is not separately illustrated in FIG. 15 but is understandable by those skilled in the art designing devices having processing units 1508, such as device 1500. As another example, one or more units, such as receiving unit 1510, may be hardware units external to processing unit 1208 in some implementations. Accordingly, the description herein optionally supports any combinations, divisions, and / or further definitions of the functional blocks described herein.

[0345] According to some embodiments, FIG. 16 illustrates a functional block diagram of an electronic device 1600 configured according to the principles of various described embodiments, including those described with reference to FIG. 12. The functional blocks of the device are optionally embodied in hardware, software, or a combination of hardware and software to realize the principles of the various described embodiments. Those skilled in the art will understand that the functional blocks described in FIG. 16 may be optionally combined or divided into sub-blocks to realize the principles of the various described embodiments. Thus, the description herein optionally supports any anticipated combination or division or further definition of the functional blocks described herein.

[0346] 16 , one or more devices 1600 include a touch-sensitive display unit 1602 and, optionally, a processing unit 1608 coupled to the touch-sensitive display unit. In some embodiments, the one or more processing units 1608 include an identifying unit 1610, a receiving unit 1612, a providing unit 1614, a determining unit 1616, and, optionally, an adjusting unit 1618 and a retrieving unit 1620.

[0347] In some examples, the processing unit 1608 is configured to: identify an intent object from a set of intent objects (e.g., using the identifying unit 1610), where the intent object is derived from natural language user input; identify a software application associated with the intent object from the set of intent objects (e.g., using the identifying unit 1610); receive a task flow associated with the software application from the second electronic device (e.g., using the receiving unit 1612); provide the intent object to the software application (e.g., using the providing unit 1614) to cause the software application to perform a task associated with the intent object; receive a request from the software application; determine a query based on the request (e.g., using the determining unit 1616), where the query is determined according to the task flow; provide the query using the touch-sensitive display 1602 (e.g., using the providing unit 1614); after providing the query, receive a second user input (e.g., using the receiving unit 1612) indicating a query response; and provide the query response to the software application (e.g., using the providing unit 1614).

[0348] In some examples, the processing unit 1608 is further configured to receive (e.g., using the receiving unit 1612) a result response from the software application indicating whether the task was successfully executed, and to provide (e.g., using the providing unit 1614) an output indicating whether the task was executed.

[0349] In some examples, providing the query response to the software application includes adjusting the intent object (eg, using the adjustment unit 1618) and providing the adjusted intent object to the software application.

[0350] In some examples, receiving a task flow associated with the software application includes receiving the task flow from another electronic device (e.g., using the receiving unit 1612) and storing the task flow on the electronic device.

[0351] In some examples, determining a query based on the request includes retrieving the query from a task flow stored on the electronic device (eg, using retrieval unit 1620).

[0352] In some examples, the query response indicates parameters associated with the intent object.

[0353] The operations described above with respect to Figure 12 are optionally implemented by the components illustrated in Figures 1, 2A, 4, 6A-B, 7A, and 16. For example, identifying operations 1205 and 1210; receiving operations 1215, 1225, and 1240; providing operations 1220, 1235, and 1245; and determining operation 1230 are optionally implemented by processor 120. It will be apparent to those skilled in the art how other processes can be implemented based on Figures 1, 2A, 4, 6A-B, 7A, and 16.

[0354] Those skilled in the art will understand that the functional blocks described in FIG. 16 may be optionally combined or divided into sub-blocks to embody the principles of the various described embodiments. Accordingly, the description herein optionally supports any anticipated combinations, divisions, or further definitions of the functional blocks described herein. For example, one or more processing units 1608 may have an associated “controller” unit operatively coupled to the processing unit 1608 to enable its operation. It will be understood that this controller unit is not separately illustrated in FIG. 16 but is understandable by those skilled in the art designing a device having a processing unit 1608, such as device 1600. As another example, one or more units, such as specific unit 1610, may be hardware units external to the processing unit 1608 in some implementations. Accordingly, the description herein optionally supports any combinations, divisions, and / or further definitions of the functional blocks described herein.

[0355] According to some embodiments, FIG. 17 illustrates a functional block diagram of an electronic device 1400 configured in accordance with the principles of various described embodiments, including those described with reference to FIG. 12. The functional blocks of the device are optionally embodied in hardware, software, or a combination of hardware and software to realize the principles of the various described embodiments. Those skilled in the art will understand that the functional blocks described in FIG. 17 may be optionally combined or divided into sub-blocks to realize the principles of the various described embodiments. Thus, the description herein optionally supports any anticipated combination or division or further definition of the functional blocks described herein.

[0356] 17, one or more devices 1700 include a touch-sensitive display 1402 and one or more processing units 1708. In some embodiments, the one or more processing units 1702 include a determining unit 1710, an identifying unit 1712, a providing unit 1714, a receiving unit 1716, and, optionally, an obtaining unit 1718 and a storage unit 1720.

[0357] In some examples, the one or more processing units 1708 are configured to: determine an intent object from the set of intent objects (e.g., using the determining unit 1710); identify a software application associated with the intent object from the set of intent objects (e.g., using the identifying unit 1712); determine a task flow associated with the software application (e.g., using the determining unit 1710); provide the intent object to the software application (e.g., using the providing unit 1714) to cause the software application to perform a task associated with the intent object; receive a request from the software application (e.g., using the receiving unit 1716); determine a query based on the request (e.g., using the determining unit 1710), the query being determined based on the task flow; provide the query using the touch-sensitive display (e.g., using the providing unit 1714); after providing the query, receive a second user input indicating a query response (e.g., using the receiving unit 1716); and provide the query response to the software application (e.g., using the sharing unit 1714).

[0358] In some examples, the one or more processing units 1708 are further configured to receive an audio input including a natural language user input (e.g., using the receiving unit 1716) and to obtain a text string from the natural language user input (e.g., using the obtaining unit 1718), and determining an intent object of the set of intent objects includes determining an intent object of the set of intent objects based on the text string (e.g., using the determining unit 1710).

[0359] In some examples, the one or more processing units 1708 are further configured to receive (e.g., using the receiving unit 1716) a result response indicating whether the task was successfully executed, and to provide (e.g., using the providing unit 1714) an output indicating whether the task was executed.

[0360] In some examples, determining a task flow associated with the software application includes determining the task flow at a first electronic device (e.g., using the determining unit 1710), providing the task flow to a second electronic device (e.g., using the providing unit 1714), and storing the task flow on the first electronic device (e.g., using the storage unit 1720).

[0361] In some examples, determining a query based on the request includes determining the query at the first electronic device (eg, using the determining unit 1710).

[0362] The operations described above with respect to Figure 12 are optionally implemented by the components illustrated in Figures 1, 2A, 4, 6A-B, 7A, and 17. For example, identifying operations 1205 and 1210; receiving operations 1215, 1225, and 1240; providing operations 1220, 1235, and 1245; and determining operation 1230 are optionally implemented by processor 120. It will be apparent to those skilled in the art how other processes can be implemented based on the components illustrated in Figures 1, 2A, 4, 6A-B, 7A, and 17.

[0363] Those skilled in the art will understand that the functional blocks described in FIG. 14 may be optionally combined or divided into sub-blocks to embody the principles of the various described embodiments. Accordingly, the description herein optionally supports any anticipated combinations, divisions, or further definitions of the functional blocks described herein. For example, processing unit 1708 may have an associated “controller” unit operatively coupled to processing unit 1708 to enable its operation. It will be understood that this controller unit is not separately illustrated in FIG. 17 but is understandable by those skilled in the art designing a device having processing unit 1708, such as device 1700. As another example, one or more units, such as specific unit 1710, may be hardware units external to processing unit 1708 in some implementations. Accordingly, the description herein optionally supports any combinations, divisions, and / or further definitions of the functional blocks described herein.

[0364] Although the foregoing description has been set forth in connection with specific embodiments for purposes of explanation, these illustrative descriptions are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Numerous modifications and variations are possible in light of the above teachings. The embodiments have been chosen and described in order to best explain the principles of the present technology and its practical application. Thus, those skilled in the art may utilize the present technology and its various embodiments to the fullest extent by making various modifications as appropriate to the particular use contemplated.

[0365] Although the disclosure and examples of the present invention have been described in detail with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art, and such changes and modifications are to be understood as being within the scope of the disclosure and examples as defined by the claims.< / datetime> < / integer> < / datastring> < / location> < / location>

Claims

1. In one or more electronic devices each having one or more processors, receiving an audio input comprising natural language user input, the audio input comprising a first parameter, the first parameter comprising a name of a first software application; determining an intent object from a set of intent objects based on the natural language user input; determining the first software application associated with the intent object in the set of intent objects based on the first parameter and one or more software applications registered with an application registration service; providing the intent object to the first software application to cause the first software application to perform a task associated with the intent object; A method comprising:

2. determining an intent object from a set of intent objects based on the natural language user input; performing natural language processing on the text string to determine the intent of the natural language user input; Identifying an Intent Object associated with the intent; and wherein the method comprises: The method of claim 1 , further comprising obtaining the text string from the natural language user input.

3. receiving a result response from the first software application indicating whether execution of the task was successful; providing an output indicating whether the task was performed; and Further comprising: The method of claim 1 or 2, wherein providing an output indicating whether the task has been performed comprises providing a natural language output.

4. providing the intent object to the first software application to cause the first software application to perform a task associated with the intent object includes providing the first parameter to the first software application.

4. The method according to any one of claims 1 to 3.

5. Determining an intent object from the intent object set The method of claim 1 , further comprising determining the intent objects of the set of intent objects based on one or more types of the one or more electronic devices.

6. The method of claim 3 , wherein the manner in which the output is provided is determined based on one or more types of the one or more electronic devices.

7. The method of claim 3 , wherein the manner in which the output is provided is determined based on one or more states of the one or more electronic devices.

8. after providing the intent object to the first software application, validating the intent object in the first software application, wherein validating the intent object in the first software application includes requesting notification that the first software application can successfully perform a task associated with the intent object; causing the first software application to perform the task after verifying the intent object in the first software application; The method of claim 1 , further comprising:

9. 9. The method of claim 1, wherein determining the first software application associated with the intent object in the intent object set is based on previous use of the first software application.

10. 10. The method of claim 1, wherein the audio input includes a second parameter, the second parameter includes a custom vocabulary, and determining the first software application is based on the second parameter.

11. The method of claim 10 , wherein the custom vocabulary includes terms uniquely associated with a user.

12. before providing the intent object to the first software application; determining whether the first software application has permission to access data associated with the intent object; in response to a determination that the software application is not authorized to access the data associated with the intent object, not providing the intent object to the first software application; preventing the first software application from performing the task; 12. The method of claim 1, further comprising:

13. determining the first software application; determining one or more domains corresponding to the intent object; determining the first software application based on the determined one or more domains; 13. The method of any one of claims 1 to 12, comprising:

14. 14. A computer-readable storage medium comprising one or more programs executed by one or more processors of one or more electronic devices, the one or more programs containing instructions that, when executed by the one or more processors, cause the one or more electronic devices to perform the method of any one of claims 1 to 13.

15. a first electronic device of the one or more electronic devices, one or more processors; Memory and one or more programs stored in a memory, the one or more programs comprising instructions for causing the one or more processors to perform the method of any one of claims 1 to 13; a first electronic device comprising:

Images