App and digital assistant integration

Abstract

The present disclosure relates to the integration of applications with digital assistants. The present invention provides systems and methods for integrating applications with digital assistants. According to an example, a method includes receiving an audio input including a natural language user input and identifying an intent object in a set of intent objects. The intent object can be derived from the natural language user input. The method also includes identifying a software application associated with the intent object in the set of intent objects, providing the intent object to the software application so that the software application performs a task associated with the intent object, receiving a result response indicating whether the task was successfully performed, and providing an output indicating whether the task was performed.

Description

[0001] Related Applications Citations

[0002] This application is a divisional application of the invention patent application with Chinese national application number 201810019395.8, application date January 9, 2018, and invention name “Application and Digital Assistant Integration”. Technical Field

[0003] The present disclosure generally relates to interacting with applications, and more specifically to techniques for integrating applications with digital assistants. Background Art

[0004] Digital assistants can help users perform various functions on user devices. For example, digital assistants can set alarms, provide weather updates, and perform searches locally and on the Internet while providing a natural language interface to the user. However, existing digital assistants cannot effectively integrate with applications such as applications stored locally on the user device, especially third-party applications. Therefore, existing digital assistants cannot provide a natural language interface with such applications. Summary of the invention

[0005] An example method is disclosed herein. The example method includes: at an electronic device having a touch-sensitive display and one or more processors, receiving an audio input including a natural language user input, identifying an intent object from a set of intent objects, wherein the intent object is derived from the natural language user input, identifying a software application associated with the intent object from the set of intent objects, providing the intent object to the software application so that the software application performs a task associated with the intent object, receiving a result response from the software application indicating whether the task was successfully performed, and providing an output indicating whether the task was performed.

[0006] An example method includes: at one or more electronic devices having one or more processors, receiving audio input including natural language user input, obtaining a text string from the natural language user input; determining an intent object in a set of intent objects based on the text string, determining a software application associated with the intent object in the set of intent objects, providing the intent object to the software application so that the software application performs a task associated with the intent object, receiving a result response from the software application indicating whether the task was successfully executed, and providing an output indicating whether the task was executed.

[0007] An example method includes, at an electronic device having one or more processors, receiving natural language user input, identifying a software application associated with an intent object in the set of intent objects, receiving a task flow associated with the software application from a second electronic device, providing the intent object to the software application so that the software application performs a task associated with the intent object, receiving a request from the software application, determining a query based on the request, wherein the query is determined according to the task flow, providing the query using a touch-sensitive display, receiving a second user input indicating a query response after providing the query, and providing the query response to the software application.

[0008] An example method includes, at one or more electronic devices having one or more processors, determining an intent object from a set of intent objects, identifying a software application associated with the intent object from the set of intent objects, determining a task flow associated with the software application, providing the intent object to the software application so that the software application performs a task associated with the intent object, receiving a request from the software application, determining a query based on the request, wherein the query is determined according to the task flow, providing the query using a touch-sensitive display, receiving a second user input indicating a query response after providing the query, and providing the query response to the software application.

[0009] The present invention discloses an example non-transitory computer-readable medium. The example non-transitory computer-readable storage medium stores one or more programs. The one or more programs include instructions that, when executed by one or more processors, cause an electronic device to receive an audio input including a natural language user input, identify an intent object in a set of intent objects, wherein the intent object is derived from the natural language user input, identify a software application associated with the intent object in the set of intent objects, provide the intent object to the software application so that the software application performs a task associated with the intent object, receive a result response from the software application indicating whether the task was successfully performed, and provide an output indicating whether the task was performed.

[0010] An example non-transitory computer-readable storage medium stores one or more programs. The one or more programs include instructions that, when executed by one or more processors, cause an electronic device to receive an audio input including a natural language user input, obtain a text string from the natural language user input, determine an intent object in a set of intent objects based on the text string, determine a software application associated with the intent object in the set of intent objects, provide the intent object to the software application so that the software application performs a task associated with the intent object, receive a result response from the software application indicating whether the task was successfully performed, and provide an output indicating whether the task was performed.

[0011] An example non-transitory computer-readable storage medium stores one or more programs. The one or more programs include instructions that, when executed by one or more processors, cause one or more electronic devices to receive natural language user input, identify a software application associated with an intent object in the set of intent objects, receive a task flow associated with the software application from a second electronic device, provide the intent object to the software application so that the software application performs a task associated with the intent object, receive a request from the software application, determine a query based on the request, wherein the query is determined according to the task flow, provide the query using a touch-sensitive display, receive a second user input indicating a query response after providing the query, and provide the query response to the software application.

[0012] An example non-transitory computer-readable storage medium stores one or more programs. The one or more programs include instructions that, when executed by one or more processors, cause one or more electronic devices to determine an intent object from a set of intent objects, identify a software application associated with the intent object from the set of intent objects, determine a task flow associated with the software application, provide the intent object to the software application so that the software application performs a task associated with the intent object, receive a request from the software application, determine a query based on the request, wherein the query is determined according to the task flow, provide the query using a touch-sensitive display, receive a second user input indicating a query response after providing the query, and provide the query response to the software application.

[0013] The present invention discloses an example electronic device and system. The example electronic device includes one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include instructions for the following operations: receiving audio input including natural language user input, identifying an intent object in a set of intent objects, wherein the intent object is derived from the natural language user input, identifying a software application associated with the intent object in the set of intent objects, providing the intent object to the software application so that the software application performs a task associated with the intent object, receiving a result response from the software application indicating whether the task was successfully performed, and providing an output indicating whether the task was performed.

[0014] An example system includes one or more processors of one or more electronic devices; one or more memories of one or more electronic devices; and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: receiving audio input including natural language user input, obtaining a text string from the natural language user input; determining an intent object in a set of intent objects based on the text string, determining a software application associated with the intent object in the set of intent objects, providing the intent object to the software application so that the software application performs a task associated with the intent object, receiving a result response from the software application indicating whether the task was successfully performed, and providing an output indicating whether the task was performed.

[0015] An example electronic device includes one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: receiving natural language user input, identifying a software application associated with an intent object in the group of intent objects, receiving a task flow associated with the software application from a second electronic device, providing the intent object to the software application so that the software application performs a task associated with the intent object, receiving a request from the software application, determining a query based on the request, wherein the query is determined according to the task flow, providing the query using a touch-sensitive display, receiving a second user input indicating a query response after providing the query, and providing the query response to the software application.

[0016] An example system includes one or more processors of one or more electronic devices; one or more memories of one or more electronic devices; and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, the one or more programs including instructions for: determining an intent object from a set of intent objects, identifying a software application associated with the intent object from the set of intent objects, determining a task flow associated with the software application, providing the intent object to the software application so that the software application performs a task associated with the intent object, receiving a request from the software application, determining a query based on the request, wherein the query is determined according to the task flow, providing the query using a touch-sensitive display, receiving a second user input indicating a query response after providing the query, and providing the query response to the software application.

[0017] An example electronic device includes a device for receiving audio input including natural language user input, a device for identifying an intent object from a set of intent objects, wherein the intent object is derived from the natural language user input, a device for identifying a software application associated with the intent object from the set of intent objects, a device for providing the intent object to the software application so that the software application performs a task associated with the intent object, a device for receiving a result response from the software application indicating whether the task was successfully performed, and a device for providing an output indicating whether the task was performed.

[0018] An example system includes a device for receiving audio input including natural language user input, a device for obtaining a text string from the natural language user input; a device for determining an intent object in a set of intent objects based on the text string, a device for determining a software application associated with the intent object in the set of intent objects, a device for providing the intent object to the software application so that the software application performs a task associated with the intent object, a device for receiving a result response from the software application indicating whether the task was successfully performed, and a device for providing an output indicating whether the task was performed.

[0019] An example electronic device includes: a device for receiving natural language user input, a device for identifying a software application associated with an intent object in the group of intent objects, a device for receiving a task flow associated with the software application from a second electronic device, a device for providing the intent object to the software application so that the software application performs a task associated with the intent object, a device for receiving a request from the software application, a device for determining a query based on the request, wherein the query is determined according to the task flow, a device for providing the query using a touch-sensitive display, a device for receiving a second user input indicating a query response after providing the query, and a device for providing the query response to the software application.

[0020] An example system includes: a device for determining an intent object from a set of intent objects, a device for identifying a software application associated with the intent object from the set of intent objects, a device for determining a task flow associated with the software application, a device for providing the intent object to the software application so that the software application performs a task associated with the intent object, a device for receiving a request from the software application, a device for determining a query based on the request, wherein the query is determined according to the task flow, a device for providing the query using a touch-sensitive display, a device for receiving a second user input indicating a query response after providing the query, and a device for providing the query response to the software application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, in which like reference numerals designate corresponding parts throughout the several views.

[0022] Figure 1 is a block diagram illustrating a system and environment for implementing a digital assistant according to various examples.

[0023] Figure 2A A block diagram of a portable multifunction device that implements the client-side portion of a digital assistant according to some embodiments is shown.

[0024] Figure 2B is a block diagram illustrating exemplary components for event processing according to various examples.

[0025] Figure 3 A portable multifunction device implementing the client-side portion of a digital assistant according to various examples is shown.

[0026] Figure 4 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to various examples.

[0027] Figure 5A An exemplary user interface for a menu of applications on a portable multifunction device is shown according to various examples.

[0028] Figure 5B An exemplary user interface for a multifunction device having a touch-sensitive surface separate from the display is shown according to various examples.

[0029] Fig. 6A A personal electronic device according to various examples is shown.

[0030] Figure 6B is a block diagram illustrating a personal electronic device according to various examples.

[0031] Fig. 7A is a block diagram illustrating a digital assistant system or a server portion thereof according to various examples.

[0032] Figure 7B Various examples are shown Fig. 7A Functions of the digital assistant shown.

[0033] Figure 7C A portion of an ontology according to various examples is shown.

[0034] Figure 8 A flowchart of a process for operating a digital assistant according to some embodiments is shown.

[0035] Fig. 9A flowchart of a process for operating a digital assistant according to some embodiments is shown.

[0036] FIG. 10A to FIG. 10C An exemplary user interface for an electronic device according to some embodiments is shown.

[0037] FIG. 10D to FIG. 10E An exemplary data flow for a digital assistant according to some embodiments is shown.

[0038] Fig.11 A flowchart of a process for operating a digital assistant according to some embodiments is shown.

[0039] Fig.12 A flowchart of a process for operating a digital assistant according to some embodiments is shown.

[0040] FIG. 13A to FIG. 13B An exemplary user interface for an electronic device according to some embodiments is shown.

[0041] Figures 14 to 17 A functional block diagram of an electronic device according to some embodiments is shown. DETAILED DESCRIPTION

[0042] In the following description of the present disclosure and embodiments, reference will be made to the accompanying drawings, in which are shown by way of illustration specific embodiments that can be implemented. It should be understood that other embodiments and examples can be practiced and changes can be made without departing from the scope of the present disclosure.

[0043] Although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the various described examples, a first input may be referred to as a second input, and similarly, a second input may be referred to as a first input. The first input and the second input may both be outputs, and in some cases may be independent and different inputs.

[0044] The terms used in the description of various examples described herein are only for describing specific examples and are not intended to be limiting. As used in the description of various examples described and in the appended claims, the singular forms "a", "an" and "the" are intended to also include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the items listed in association. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements, and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0045] Depending on the context, the term "if" may be interpreted to mean "when" or "upon" or "in response to determining that" or "in response to detecting that..." Similarly, depending on the context, the phrases "if it is determined that" or "if [a stated condition or event] is detected" may be interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]."

[0046] 1. System and Environment

[0047] Figure 1 A block diagram of a system 100 according to various examples is shown. In some examples, the system 100 may implement a digital assistant. The terms "digital assistant," "virtual assistant," "intelligent automated assistant," or "automatic digital assistant" may refer to any information processing system that interprets natural language input in speech form and / or text form to infer user intent and performs actions (e.g., tasks) based on the inferred user intent. For example, to act on an inferred user intent, the system may perform one or more of the following actions: identify a task flow with steps and parameters designed to accomplish the inferred user intent, input specific requirements from the inferred user intent into the task flow; execute the task flow by calling a program, method, service, API, etc.; and generate an output response to the user in an audible (e.g., voice) and / or visual form.

[0048] Specifically, the digital assistant may be able to accept user requests in the form of natural language commands, requests, statements, narrations and / or inquiries, at least in part. Typically, the user request may seek an informative answer from the digital assistant, or seek a digital assistant to perform a task. A satisfactory response to the user request may be to provide the requested informative answer, perform the requested task, or a combination of the two. For example, the user may ask the digital assistant a question, such as "where am I now?" Based on the user's current location, the digital assistant may answer "you are near the west gate of Central Park." The user may also request to perform a task, such as "please invite my friends to my girlfriend's birthday party next week." In response, the digital assistant may confirm the request by saying "OK, right away," and then send a suitable calendar invitation to each of the user's friends listed in the user's electronic address book on behalf of the user. During the execution of the requested task, the digital assistant may sometimes interact with the user in a continuous dialogue involving multiple information exchanges over a long period of time. There are many other methods of interacting with the digital assistant to request information or perform various tasks. In addition to providing verbal responses and taking programmed actions, digital assistants can also provide responses in other visual or audio forms, such as text, alerts, music, video, animation, etc.

[0049] like Figure 1 As shown in , in some examples, the digital assistant can be implemented according to a client-server model. The digital assistant may include a client-side portion 102 (hereinafter referred to as "DA client 102") executed on a user device 104, and a server-side portion 106 (hereinafter referred to as "DA server 106") executed on a server system 108. The DA client 102 can communicate with the DA server 106 via one or more networks HO. The DA client 102 can provide client-side functions, such as user-oriented input and output processing, and communicate with the DA server 106. The DA server 106 can provide server-side functions for any number of DA clients 102, each residing on a corresponding user device 104.

[0050] In some examples, the DA server 106 may include a client-facing I / O interface 112, one or more processing modules 114, data and models 116, and an I / O interface 118 to external services. The client-facing I / O interface 112 may facilitate client-facing input and output processing for the DA server 106. One or more processing modules 114 may utilize the data and models 116 to process voice input and determine the user's intent based on the natural language input. In addition, one or more processing modules 114 perform tasks based on the inferred user intent. In some examples, the DA server 106 may communicate with external services 120 via one or more networks 110 to complete tasks or collect information. The I / O interface 118 to external services may facilitate such communications.

[0051] The user device 104 may be any suitable electronic device. For example, the user device may be a portable multi-function device (such as the Figure 2A Device 200 described), multi-function device (such as the following reference Figure 4 Device 400 described herein), or a personal electronic device (such as the one described below with reference to Fig. 6A -B depicts device 600). The portable multifunction device may be, for example, a mobile phone that also includes other functions such as a PDA and / or a music player function. Specific examples of portable multifunction devices may include the Apple iPod from Apple Inc. (Cupertino, California). Devices, iPod Equipment, and Device. Other examples of portable multifunction devices may include, but are not limited to, laptops or tablet computers. In addition, in some examples, user device 104 may be a non-portable multifunction device. Specifically, user device 104 may be a desktop computer, a game console, a television, or a television set-top box. In some examples, user device 104 may include a touch-sensitive surface (e.g., a touch screen display and / or a touchpad). In addition, user device 104 may optionally include one or more other physical user interface devices, such as a physical keyboard, a mouse, and / or a joystick. Various examples of electronic devices such as multifunction devices are described in more detail below.

[0052] Examples of one or more communication networks 110 may include local area networks (LANs) and wide area networks (WANs), such as the Internet. One or more communication networks 110 may be implemented using any known network protocols, including various wired or wireless protocols, such as, for example, Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0053] The server system 108 may be implemented on one or more stand-alone data processing devices or distributed networks of computers. In some examples, the server system 108 may also employ various virtual devices and / or services of third-party service providers (e.g., third-party cloud service providers) to provide the potential computing resources and / or infrastructure resources of the server system 108.

[0054] In some examples, user device 104 can communicate with DA server 106 via second user device 122. Second user device 122 can be similar or identical to user device 104. For example, second user device 122 can be similar to the user device described below with reference to Figure 2A , Figure 4 and FIG. 6A to FIG. 6B Device 200, 400 or 600 described. The user device 104 can be configured to be communicatively coupled to the second user device 122 via a direct communication connection such as Bluetooth, NFC, BTLE, etc. or via a wired or wireless network such as a local Wi-Fi network. In some examples, the second user device 122 can be configured to act as a proxy between the user device 104 and the DA server 106. For example, the DA client 102 of the user device 104 can be configured to transmit information (e.g., a user request received at the user device 104) to the DA server 106 via the second user device 122. The DA server 106 can process the information and return relevant data (e.g., data content in response to the user request) to the user device 104 via the second user device 122.

[0055] In some examples, the user device 104 may be configured to send an abbreviated request for data to a second user device 122 to reduce the amount of information transmitted from the user device 104. The second user device 122 may be configured to determine supplemental information to add to the abbreviated request to generate a complete request for transmission to the DA server 106. The system architecture may advantageously allow a user device 104 with limited communication capabilities and / or limited battery power (e.g., a watch or similar compact electronic device) to access services provided by the DA server 106 by using a second user device 122 with greater communication capabilities and / or battery power (e.g., a mobile phone, laptop computer, tablet computer, etc.) as a proxy to the DA server 106. Although Figure 1 Only two user devices 104 and 122 are shown, and it should be understood that the system 100 may include any number and type of user devices configured to communicate with the DA server system 106 in this proxy configuration.

[0056] Although Figure 1 The digital assistant shown in may include both a client-side portion (e.g., DA client 102) and a server-side portion (e.g., DA server 106), but in some examples, the functionality of the digital assistant may be implemented as a stand-alone application installed on a user device. In addition, the division of functionality between the client and server portions of the digital assistant may vary in different specific implementations. For example, in some examples, the DA client may be a thin client that only provides user-facing input and output processing functions and delegates all other functions of the digital assistant to a backend server.

[0057] 2. Electronic devices

[0058] Attention now turns to embodiments of electronic devices for implementing the client-side portion of a digital assistant. Figure 2Ais a block diagram showing a portable multifunction device 200 with a touch-sensitive display system 212 according to some embodiments. The touch-sensitive display 212 is sometimes referred to as a "touch screen" for convenience, and may sometimes be referred to as or referred to as a "touch-sensitive display system". The device 200 includes a memory 202 (which optionally includes one or more computer-readable storage media), a memory controller 222, one or more processing units (CPUs) 220, a peripheral device interface 218, an RF circuit 208, an audio circuit 210, a speaker 211, a microphone 213, an input / output (PO) subsystem 206, other input control devices 216, and an external port 224. The device 200 optionally includes one or more optical sensors 264. The device 200 optionally includes one or more contact force sensors 265 for detecting the intensity of contact on the device 200 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 212 of the device 200). Device 200 optionally includes one or more tactile output generators 267 for generating tactile output on device 200 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 212 of device 200 or touch pad 455 of device 400). These components optionally communicate via one or more communication buses or signal lines 203.

[0059] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate (surrogate) of the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted average) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the alternative measurement of the contact force or pressure is used directly to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the alternative measurement). In some embodiments, the alternative measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using contact intensity as an attribute of user input allows the user to access additional device functions that the user may not otherwise be able to access on a smaller device with limited real estate for displaying an indication (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical controls / mechanical controls, such as knobs or buttons).

[0060] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a touch-sensitive surface of a user (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when there is no change in the smoothness of the touch-sensitive surface, movement of the touch-sensitive surface will optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. While such interpretation of touch by the user will be limited by the user's individualized sensory perceptions, there are many sensory perceptions of touch that are common to most users. Thus, when tactile outputs are described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the tactile outputs generated correspond to physical displacements of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0061] It should be understood that device 200 is only one example of a portable multifunction device, and that device 200 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 2A The various components shown are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing circuits and / or application specific integrated circuits.

[0062] The memory 202 may include one or more computer-readable storage media. The computer-readable storage media may be tangible and non-transitory. The memory 202 may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The memory controller 222 may control other components of the device 200 to access the memory 202.

[0063] In some examples, the non-transitory computer-readable storage medium of memory 202 may be used to store instructions (e.g., for performing various aspects of process 1100 described below) for use by or in conjunction with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other system that can obtain instructions from and execute instructions by an instruction execution system, apparatus, or device. In other examples, the instructions (e.g., for performing various aspects of process 1100 described below) may be stored on a non-transitory computer-readable storage medium (not shown) of server system 108, or may be divided between the non-transitory computer-readable storage medium of memory 202 and the non-transitory computer-readable storage medium of server system 108. In the context of this document, a "non-transitory computer-readable storage medium" may be any medium that may include or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0064] The peripheral device interface 218 can be used to couple the input peripheral devices and output peripheral devices of the device to the CPU 220 and the memory 202. The one or more processors 220 run or execute various software programs and / or instruction sets stored in the memory 202 to perform various functions of the device 200 and process data. In some embodiments, the peripheral device interface 218, the CPU 220, and the memory controller 222 can be implemented on a single chip such as the chip 204. In some other embodiments, they can be implemented on separate chips.

[0065] RF (radio frequency) circuit 208 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 208 converts electrical signals into / converts electromagnetic signals into electrical signals, and communicates with a communication network and other communication devices via electromagnetic signals. RF circuit 208 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC (codec) chipset, a user identity module (SIM) card, a memory, and the like. RF circuit 208 optionally communicates with a network and other devices via wireless communication, such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). RF circuit 208 optionally includes well-known circuits for detecting a near field communication (NFC) field, such as through a short-range communication radio component. Wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11d), IEEE 802.11e, IEEE 802.11f, IEEE 802.11g, IEEE 802.11g). 802.1In and / or IEEE802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols that have not been developed as of the date of filing of this document.

[0066] The audio circuit 210, the speaker 211, and the microphone 213 provide an audio interface between the user and the device 200. The audio circuit 210 receives audio data from the peripheral device interface 218, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 211. The speaker 211 converts the electrical signals into sound waves audible to humans. The audio circuit 210 also receives electrical signals converted from sound waves by the microphone 213. The audio circuit 210 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 218 for processing. The audio data can be retrieved from and / or transmitted to the memory 202 and / or the RF circuit 208 by the peripheral device interface 218. In some embodiments, the audio circuit 210 also includes a headphone jack (e.g., Figure 3 The headset jack provides an interface between the audio circuit 210 and a removable audio input / output peripheral device, such as an output-only headset or a headset having both output (e.g., single or dual ear headphones) and input (e.g., a microphone).

[0067] The I / O subsystem 206 couples input / output peripherals on the device 200, such as a touch screen 212 and other input control devices 216, to a peripheral device interface 218. The I / O subsystem 206 optionally includes a display controller 256, an optical sensor controller 258, an intensity sensor controller 259, a tactile feedback controller 261, and one or more input controllers 260 for other input or control devices. The one or more input controllers 260 receive / send electrical signals from other input control devices 216. Other input control devices 216 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, one or more input controllers 260 are optionally coupled to any of the following (or none of the following): a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more buttons (e.g., Figure 3 308 in ) optionally includes an up / down button for volume control of the speaker 211 and / or the microphone 213. The one or more buttons optionally include a push button (e.g., Figure 3 306 in ).

[0068] A quick press of the push button may unlock the touch screen 212 or begin the process of unlocking the device using gestures on the touch screen, as described in U.S. Patent Application 11 / 322,549, filed December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," and U.S. Patent Application 7,657,849, which are incorporated herein by reference in their entirety. A longer press of the push button (e.g., 306) may turn the device 200 on or off. The user may customize the functionality of one or more buttons. The touch screen 212 is used to implement virtual or soft buttons and one or more soft keyboards.

[0069] The touch-sensitive display 212 provides an input interface and an output interface between the device and the user. The display controller 256 receives electrical signals from the touch screen 212 and / or sends electrical signals to the touch screen. The touch screen 212 displays visual output to the user. The visual output may include graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output may correspond to user interface objects.

[0070] The touch screen 212 has a touch-sensitive surface, sensor, or sensor group that accepts input from a user based on tactile and / or haptic contact. The touch screen 212 and display controller 256 (together with any associated modules and / or instruction sets in memory 202) detect contact (and any movement or interruption of the contact) on the touch screen 212 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 212. In an exemplary embodiment, the point of contact between the touch screen 212 and the user corresponds to the user's finger.

[0071] The touch screen 212 may use LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies may be used in other embodiments. The touch screen 212 and display controller 256 may use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive technology, resistive technology, infrared technology, and surface acoustic wave technology, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 212 to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as in the Apple Inc. (Cupertino, California) and iPod Technology found in.

[0072] In some embodiments, the touch-sensitive display of the touch screen 212 can be similar to the multi-touch-sensitive touch pad described in the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman) and / or U.S. Patent Publication 2002 / 0015024A1, each of which is incorporated herein by reference in its entirety. However, the touch screen 212 displays visual output from the device 200, while the touch-sensitive touch pad does not provide visual output.

[0073] In some embodiments, the touch-sensitive display of the touch screen 212 can be as described in the following patent applications: (1) U.S. patent application 11 / 381,313, filed on May 2, 2006, “Multipoint Touch Surface Controller”; (2) U.S. patent application 10 / 840,862, filed on May 6, 2004, “Multipoint Touchscreen”; (3) U.S. patent application 10 / 903,964, filed on July 30, 2004, “Gestures For Touch Sensitive Input Devices”; (4) U.S. patent application 11 / 048,264, filed on January 31, 2005, “Gestures For Touch Sensitive Input Devices”; (5) U.S. patent application 11 / 038,590, filed on January 18, 2005, “Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices”; (6) U.S. patent application 11 / 228,758, filed on September 16, 2005, “Virtual Input Device Placement On A Touch Screen User Interface”; (7) U.S. patent application 11 / 228,700, filed on September 16, 2005, “Operation Of A Computer With A Touch Screen Interface”; (8) U.S. patent application 11 / 228,737, filed on September 16, 2005, “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard”; and (9) U.S. patent application 11 / 367,749, filed on March 3, 2006, “Multi-Functional Hand-Held Device”. All of these patent applications are incorporated herein by reference in their entirety.

[0074] The touch screen 212 may have a video resolution of more than 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user may contact the touch screen 212 using any suitable object or appendage, such as a stylus, finger, etc. In some embodiments, the user interface is designed to work primarily with finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some embodiments, the device translates rough finger-based input into precise pointer / cursor positions or commands for performing the actions desired by the user.

[0075] In some embodiments, in addition to the touch screen, the device 200 may also include a touch pad (not shown) for activating or deactivating specific functions. In some embodiments, the touch pad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touch pad can be a touch-sensitive surface separate from the touch screen 212, or an extension of the touch-sensitive surface formed by the touch screen.

[0076] Device 200 also includes a power system 262 for powering the various components. Power system 262 may include a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, power fault detection circuits, power converters or inverters, power status indicators (e.g., light emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0077] The device 200 may also include one or more optical sensors 264 . Figure 2AAn optical sensor coupled to an optical sensor controller 258 in the I / O subsystem 206 is shown. The optical sensor 264 may include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 264 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with the imaging module 243 (also called a camera module), the optical sensor 264 can capture static images or videos. In some embodiments, the optical sensor is located on the rear of the device 200 opposite to the touch screen display 212 on the front of the device, so that the touch screen display can be used as a viewfinder for static images and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device so that the image of the user can be obtained for the video conference when the user views other video conference participants on the touch screen display. In some embodiments, the position of the optical sensor 264 can be changed by the user (e.g., by rotating the lens and sensor in the device housing) so that a single optical sensor 264 can be used with the touch screen display for both video conferencing and static image and / or video image acquisition.

[0078] Device 200 optionally also includes one or more contact intensity sensors 265 . Figure 2A A contact force sensor is shown coupled to a force sensor controller 259 in the I / O subsystem 206. The contact force sensor 265 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). The contact force sensor 265 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 212). In some embodiments, at least one contact force sensor is located on the rear of the device 200, opposite to the touch screen display 212 located on the front of the device 200.

[0079] Device 200 may also include one or more proximity sensors 266 . Figure 2AA proximity sensor 266 is shown coupled to the peripheral device interface 218. Alternatively, the proximity sensor 266 can be coupled to the input controller 260 in the I / O subsystem 206. The proximity sensor 266 can be implemented as described in U.S. patent applications 11 / 241,839, "Proximity Detector In Handheld Device"; 11 / 240,788, "Proximity Detector In Handheld Device"; 11 / 620,702; "Using Ambient Light Sensor To Augment Proximity Sensor Output"; 11 / 586,862, "Automated Response To And Sensing Of User Activity In Portable Devices"; and 11 / 638,251, "Methods And Systems For Automatic Configuration Of Peripherals", which are incorporated herein by reference in their entirety. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor is turned off and the touch screen 212 is disabled.

[0080] Device 200 optionally also includes one or more tactile output generators 267. Figure 2A A tactile output generator coupled to a tactile feedback controller 261 in the I / O subsystem 206 is shown. The tactile output generator 267 optionally includes one or more electroacoustic devices such as a speaker or other audio component and / or an electromechanical device that converts energy into linear motion such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). The contact force sensor 265 receives tactile feedback generation instructions from the tactile feedback module 233 and generates a tactile output on the device 200 that can be felt by a user of the device 200. In some embodiments, at least one tactile output generator is arranged in juxtaposition or proximity to a touch-sensitive surface (e.g., a touch-sensitive display system 212) and optionally generates tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 200) or laterally (e.g., backward and forward in the same plane as the surface of the device 200). In some embodiments, at least one tactile output generator sensor is located on the back of the device 200, opposite the touch screen display 212 located on the front of the device 200.

[0081] The device 200 may also include one or more accelerometers 268 . Figure 2A An accelerometer 268 coupled to the peripheral device interface 218 is shown. Alternatively, the accelerometer 268 may be coupled to the input controller 260 in the I / O subsystem 206. The accelerometer 268 may be implemented as described in U.S. Patent Publication 20050190059 entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication 20060017692 entitled "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer", both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on the touch screen display in a longitudinal view or a transverse view based on an analysis of data received from one or more accelerometers. In addition to the accelerometer 268, the device 200 optionally also includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the position and orientation (e.g., longitudinal or transverse) of the device 200.

[0082] In some embodiments, the software components stored in memory 202 include an operating system 226, a communication module (or instruction set) 228, a contact / motion module (or instruction set) 230, a graphics module (or instruction set) 232, a text input module (or instruction set) 234, a global positioning system (GPS) module (or instruction set) 235, a digital assistant client module 229, and an application (or instruction set) 236. In addition, memory 202 can store data and models, such as user data and models 231. In addition, in some embodiments, memory 202 ( Figure 2A ) or 470( Figure 4 ) storage device / global internal state 257, such as Figure 2A and Figure 4 The device / global internal state 257 includes one or more of the following states: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch screen display 212; a sensor state, which includes information obtained from the device's various sensors and input control devices 216; and position information about the device's position and / or posture.

[0083] The operating system 226 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware components and software components.

[0084] The communication module 228 facilitates communication with other devices through one or more external ports 224, and also includes various software components for processing data received by the RF circuit 208 and / or the external port 224. The external port 224 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for direct coupling to other devices or indirect coupling through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is connected to (trademark of Apple Inc.) devices.

[0085] The contact / motion module 230 optionally detects contact with the touch screen 212 (in conjunction with the display controller 256) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 230 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger press event), determining the contact strength (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 230 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction) and / or acceleration (change in magnitude and / or direction) of the contact point, and the movement of the contact point is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact) or multiple points of simultaneous contact (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 230 and display controller 256 detect contact on the touch pad.

[0086] In some embodiments, the contact / motion module 230 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators and can be adjusted without changing the physical hardware of the device 200). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of the touchpad or touchscreen display can be set to any one of a large range of predefined thresholds. In addition, in some specific implementations, a software setting is provided to the user of the device for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0087] The contact / motion module 230 optionally detects gesture input made by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event.

[0088] The graphics module 232 includes various known software components for rendering and displaying graphics on the touch screen 212 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0089] In some embodiments, the graphics module 232 stores data to be used to represent graphics. Each graphic is optionally assigned a corresponding code. The graphics module 232 receives one or more codes specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data if necessary, and then generates screen image data to be output to the display controller 256.

[0090] Haptic feedback module 233 includes various software components for generating instructions used by tactile output generator 267 to produce tactile output at one or more locations on device 200 in response to user interaction with device 200 .

[0091] Text input module 234, which may be a component of graphics module 232, provides a soft keyboard for entering text in a variety of applications (eg, contacts 237, email 240, instant messaging 241, browser 247, and any other application requiring text input).

[0092] The GPS module 235 determines the location of the device and provides this information for use in various applications (e.g., to the phone 238 for location-based dialing; to the camera 243 as picture / video metadata; and to applications that provide location-based services, such as the Weather widget, the Local Pages widget, and the Maps / Navigation widget).

[0093] The digital assistant client module 229 may include various client-side digital assistant instructions to provide client-side functions of the digital assistant. For example, the digital assistant client module 229 may be able to accept sound input (e.g., voice input), text input, touch input, and / or gesture input through various user interfaces of the portable multifunction device 200 (e.g., microphone 213, accelerometer 268, touch-sensitive display system 212, optical sensor 264, other input control devices 216, etc.). The digital assistant client module 229 may also be able to provide audio output (e.g., voice output), visual output, and / or tactile output through various output interfaces of the portable multifunction device 200 (e.g., speaker 211, touch-sensitive display system 212, one or more tactile output generators 267, etc.). For example, the output may be provided as voice, sound, prompt, text message, menu, graphic, video, animation, vibration, and / or a combination of two or more of the above. During operation, the digital assistant client module 229 may use the RF circuit 208 to communicate with the DA server 106.

[0094] User data and models 231 may include various data associated with the user (e.g., user-specific vocabulary data, user preference data, user-specified name pronunciation, data from the user's electronic address book, to-do items, shopping lists, etc.) to provide client-side functionality of the digital assistant. In addition, user data and models 231 may include various models for processing user input and determining user intent (e.g., speech recognition models, statistical language models, natural language processing models, knowledge ontologies, task flow models, service models, etc.).

[0095] In some examples, the digital assistant client module 229 may utilize the various sensors, subsystems, and peripherals of the portable multifunction device 200 to sample additional information from the surrounding environment of the portable multifunction device 200 to establish a context associated with the user, the current user interaction, and / or the current user input. In some examples, the digital assistant client module 229 may provide the context information, or a subset thereof, to the DA server 106 along with the user input to help infer the user's intent. In some examples, the digital assistant may also use the context information to determine how to prepare and deliver output to the user. The context information may be referred to as context data.

[0096] In some examples, contextual information accompanying the user input may include sensor information, such as lighting, ambient noise, ambient temperature, images or videos of the surrounding environment, etc. In some examples, contextual information may also include the physical state of the device, such as device orientation, device location, device temperature, power level, speed, acceleration, motion pattern, cellular signal strength, etc. In some examples, information related to the software state of the DA server 106 (e.g., running processes, installed programs, past and current network activity, background services, error logs, resource usage, etc.) and information related to the software state of the portable multifunction device 200 may be provided to the DA server 106 as contextual information associated with the user input.

[0097] In some examples, the digital assistant client module 229 may selectively provide information stored on the portable multifunction device 200 (e.g., user data 231) in response to a request from the DA server 106. In some examples, the digital assistant client module 229 may also elicit additional input from the user via a natural language dialog or other user interface upon request by the DA server 106. The digital assistant client module 229 may communicate the additional input to the DA server 106 to assist the DA server 106 in inferring intent and / or satisfying the user intent expressed in the user request.

[0098] A more detailed description of the digital assistant is provided below. 7A to 7C It should be appreciated that the digital assistant client module 229 may include any number of sub-modules of the digital assistant module 726 described below.

[0099] The application 236 may include the following modules (or instruction sets), or a subset or superset thereof:

[0100] Contacts module 237 (sometimes called address book or contact list);

[0101] Telephone module 238;

[0102] Video conferencing module 239;

[0103] Email client module 240;

[0104] Instant messaging (IM) module 241;

[0105] Fitness support module 242;

[0106] A camera module 243 for still images and / or video images;

[0107] Image management module 244;

[0108] Video player module;

[0109] Music player module;

[0110] Browser module 247;

[0111] Calendar module 248;

[0112] A widget module 249, which may include one or more of the following: a weather widget 249-1, a stock widget 249-2, a calculator widget 249-3, an alarm widget 249-4, a dictionary widget 249-5, and other widgets obtained by the user and a widget 249-6 created by the user;

[0113] A widget creator module 250 for making user-created widgets 249-6;

[0114] Search module 251;

[0115] Video and music player module 252, which merges the video player module and the music player module;

[0116] Notepad module 253;

[0117] Map module 254; and / or

[0118] Online video module 255.

[0119] Examples of other applications 236 that may be stored in memory 202 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0120] In combination with the touch screen 212, display controller 256, touch / motion module 230, graphics module 232, and text input module 234, the contact module 237 can be used to manage an address book or contact list (for example, stored in the application internal state 292 of the contact module 237 in memory 202 or memory 470), including: adding names to the address book; deleting one or more names from the address book; associating telephone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing telephone numbers or email addresses to initiate and / or facilitate communications via telephone 238, video conferencing module 239, email 240, or IM 241; and the like.

[0121] In conjunction with RF circuit 208, audio circuit 210, speaker 211, microphone 213, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, phone module 238 may be used to enter a character sequence corresponding to a phone number, access one or more phone numbers in contact module 237, modify an entered phone number, dial a corresponding phone number, conduct a conversation, and disconnect or hang up when the conversation is complete. As described above, wireless communication may use any of a number of communication standards, protocols, and technologies.

[0122] In combination with the RF circuit 208, the audio circuit 210, the speaker 211, the microphone 213, the touch screen 212, the display controller 256, the optical sensor 264, the optical sensor controller 258, the touch / motion module 230, the graphics module 232, the text input module 234, the contact module 237 and the telephone module 238, the video conferencing module 239 includes executable instructions for initiating, conducting and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0123] In conjunction with RF circuit 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, email client module 240 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 244, email client module 240 makes it very easy to create and send emails with still images or video images captured by camera module 243.

[0124] In conjunction with RF circuit 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text input module 234, instant messaging module 241 includes executable instructions for the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, transmitting a corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving an instant message, and viewing the received instant message. In some embodiments, the transmitted and / or received instant message may include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging service (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0125] In combination with the RF circuit 208, the touch screen 212, the display controller 256, the contact / motion module 230, the graphics module 232, the text input module 234, the GPS module 235, the map module 254, and the music player module, the fitness support module 242 includes executable instructions for the following operations: creating a fitness (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (sports equipment); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for fitness; and displaying, storing, and transmitting fitness data.

[0126] In combination with the touch screen 212, the display controller 256, one or more optical sensors 264, the optical sensor controller 258, the touch / motion module 230, the graphics module 232, and the image management module 244, the camera module 243 includes executable instructions for the following operations: capturing still images or videos (including video streams) and storing them in the memory 202, modifying the characteristics of the still images or videos, or deleting the still images or videos from the memory 202.

[0127] In conjunction with touch screen 212, display controller 256, touch / motion module 230, graphics module 232, text input module 234, and camera module 243, image management module 244 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0128] In combination with the RF circuit 208, the touch screen 212, the display controller 256, the touch / motion module 230, the graphics module 232 and the text input module 234, the browser module 247 includes executable instructions for browsing the Internet (including searching, linking to, receiving and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0129] In combination with the RF circuit 208, the touch screen 212, the display controller 256, the touch / motion module 230, the graphics module 232, the text input module 234, the email client module 240 and the browser module 247, the calendar module 248 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0130] In combination with the RF circuit 208, the touch screen 212, the display controller 256, the contact / motion module 230, the graphics module 232, the text input module 234, and the browser module 247, the desktop widget module 249 is a mini-application that can be downloaded and used by the user (e.g., the weather desktop widget 249-1, the stock desktop widget 249-2, the calculator desktop widget 249-3, the alarm desktop widget 249-4, and the dictionary desktop widget 249-5), or a mini-application created by the user (e.g., the user-created desktop widget 249-6). In some embodiments, the desktop widget includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop widget includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! Desktop Widget).

[0131] In combination with the RF circuit 208, the touch screen 212, the display controller 256, the touch / motion module 230, the graphics module 232, the text input module 234 and the browser module 247, the desktop widget creator module 250 can be used by users to create desktop widgets (for example, converting a user-specified portion of a web page into a desktop widget).

[0132] In combination with the touch screen 212, display controller 256, contact / motion module 230, graphics module 232 and text input module 234, the search module 251 includes executable instructions for searching the memory 202 for text, music, sound, images, videos and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0133] In conjunction with touch screen 212, display controller 256, contact / motion module 230, graphics module 232, audio circuitry 210, speaker 211, RF circuitry 208, and browser module 247, video and music player module 252 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch screen 212 or on an external display connected via external port 224). In some embodiments, device 200 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

[0134] In conjunction with the touch screen 212, display controller 256, contact / motion module 230, graphics module 232 and text input module 234, the notepad module 253 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0135] In combination with the RF circuit 208, the touch screen 212, the display controller 256, the touch / motion module 230, the graphics module 232, the text input module 234, the GPS module 235 and the browser module 247, the map module 254 can be used to receive, display, modify and store maps and data associated with the maps (e.g., driving directions, data related to stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.

[0136] In conjunction with touch screen 212, display controller 256, contact / motion module 230, graphics module 232, audio circuit 210, speaker 211, RF circuit 208, text input module 234, email client module 240, and browser module 247, online video module 255 includes instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), playback (e.g., on the touch screen or on an external display connected via external port 224), send an email with a link to a specific online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 241 is used instead of email client module 240 to send a link to a specific online video.

[0137] Additional description of the online video application can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed on June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11 / 968,067, filed on December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are hereby incorporated by reference in their entirety.

[0138] Each of the above modules and applications corresponds to an executable instruction set for performing one or more of the above functions and methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) do not have to be implemented as independent software programs, processes, or modules, and therefore various subsets of these modules may be combined or otherwise rearranged in various embodiments. For example, a video player module may be combined with a music player module into a single module (e.g., Figure 2A In some embodiments, the memory 202 may store a subset of the modules and data structures identified above. In addition, the memory 202 may store additional modules and data structures not described above.

[0139] In some embodiments, the device 200 is a device where the operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for the operation of the device 200, the number of physical input control devices (such as push buttons, dials, etc.) on the device 200 can be reduced.

[0140] The predefined set of functions that are performed exclusively through the touch screen and / or trackpad optionally includes navigating between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 200 to a main menu, a home menu, or a root menu from any user interface displayed on the device 200. In such embodiments, a "menu button" is implemented using a touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a trackpad.

[0141] Figure 2Bis a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 202 ( Figure 2A ) or memory 470( Figure 4 ) includes an event classifier 270 (e.g., in the operating system 226) and a corresponding application 236-1 (e.g., any one of the aforementioned applications 237 to 251, 255, 480 to 490).

[0142] The event classifier 270 receives event information and determines the application 236-1 and the application view 291 of the application 236-1 to which the event information is to be delivered. The event classifier 270 includes an event monitor 271 and an event distributor module 274. In some embodiments, the application 236-1 includes an application internal state 292 that indicates one or more current application views displayed on the touch-sensitive display 212 when the application is active or executing. In some embodiments, the device / global internal state 257 is used by the event classifier 270 to determine which application(s) is currently active, and the application internal state 292 is used by the event classifier 270 to determine the application view 291 to which the event information is to be delivered.

[0143] In some embodiments, the application internal state 292 includes additional information, such as one or more of the following: resumption information to be used when the application 236-1 resumes execution, user interface state information indicating information being displayed by the application 236-1 or information ready to be displayed by the application 236-1, a state queue for enabling a user to return to a previous state or view of the application 236-1, and a repeat / undo queue of previous actions taken by the user.

[0144] Event monitor 271 receives event information from peripherals interface 218. Event information includes information about sub-events (e.g., user touches on touch-sensitive display 212 as part of a multi-touch gesture). Peripherals interface 218 transmits information it receives from I / O subsystem 206 or sensors such as proximity sensor 266, accelerometer 268, and / or microphone 213 (through audio circuit 210). The information that peripherals interface 218 receives from I / O subsystem 206 includes information from touch-sensitive display 212 or a touch-sensitive surface.

[0145] In some embodiments, event monitor 271 sends requests to peripheral device interface 218 at predetermined intervals. In response, peripheral device interface 218 transmits event information. In other embodiments, peripheral device interface 218 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0146] In some embodiments, the event classifier 270 also includes a hit view determination module 272 and / or an active event identifier determination module 273.

[0147] When the touch-sensitive display 212 displays more than one view, the hit view determination module 272 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0148] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected can correspond to a programmatic level within a programmatic or view hierarchy of the application. For example, the lowest-level view in which a touch is detected can be called a hit view, and the set of events recognized as correct input can be determined based at least in part on the hit view of the initial touch that started the touch-based gesture.

[0149] The hit view determination module 272 receives information related to sub-events of a contact-based gesture. When an application has multiple views organized in a hierarchy, the hit view determination module 272 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module 272, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0150] Active event recognizer determination module 273 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 273 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 273 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with a particular view, views higher in the hierarchy will still remain actively participating views.

[0151] Event distributor module 274 distributes event information to event identifiers (e.g., event identifier 280). In embodiments including active event identifier determination module 273, event distributor module 274 delivers the event information to the event identifier determined by active event identifier determination module 273. In some embodiments, event distributor module 274 stores the event information in an event queue, which is retrieved by corresponding event receiver 282.

[0152] In some embodiments, operating system 226 includes event classifier 270. Alternatively, application 236-1 includes event classifier 270. In other embodiments, event classifier 270 is a standalone module or is part of another module stored in memory 202, such as contact / motion module 230.

[0153] In some embodiments, the application 236-1 includes multiple event handlers 290 and one or more application views 291, each of which includes instructions for handling touch events occurring in a corresponding view of the user interface of the application. Each application view 291 of the application 236-1 includes one or more event identifiers 280. Typically, the corresponding application view 291 includes multiple event identifiers 280. In other embodiments, one or more event identifiers 280 are part of an independent module, such as a user interface toolkit (not shown) or a higher level object from which the application 236-1 inherits methods and other properties. In some embodiments, the corresponding event handler 290 includes one or more of the following: data updater 276, object updater 277, GUI updater 278, and / or event data 279 received from the event classifier 270. The event handler 290 can utilize or call the data updater 276, object updater 277, or GUI updater 278 to update the application internal state 292. Alternatively, one or more application views 291 include one or more corresponding event handlers 290. Additionally, in some embodiments, one or more of the data updater 276, object updater 277, and GUI updater 278 are included in the corresponding application view 291.

[0154] The corresponding event identifier 280 receives event information (e.g., event data 279) from the event classifier 270 and identifies the event from the event information. The event identifier 280 includes an event receiver 282 and an event comparator 284. In some embodiments, the event identifier 280 also includes at least a subset of metadata 283 and event delivery instructions 288 (which may include sub-event delivery instructions).

[0155] The event receiver 282 receives event information from the event classifier 270. The event information includes information about sub-events such as touches or touch movements. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information may also include the rate and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another orientation (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0156] Event comparator 284 compares event information with predefined event or sub-event definitions, and based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 284 includes event definition 286. Event definition 286 includes definitions of events (e.g., predefined sub-event sequences), e.g., event 1 (287-1), event 2 (287-2), and others.

[0157] In some embodiments, sub-events in event (287) include, for example, touch start, touch end, touch movement, touch cancellation, and multi-touch. In one example, event 1 (287-1) is defined as a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift-off (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on the displayed object, and a second lift-off (touch end) of a predetermined duration. In another example, event 2 (287-2) is defined as a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on the displayed object, movement of the touch on the touch-sensitive display 212, and lift-off of the touch (touch end).

[0158] In some embodiments, the event also includes information for one or more associated event handlers 290.

[0159] In some embodiments, event definition 286 includes a definition of an event for a corresponding user interface object. In some embodiments, event comparator 284 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 212, when a touch is detected on touch-sensitive display 212, event comparator 284 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 290, the event comparator uses the result of the hit test to determine which event handler 290 should be activated. For example, event comparator 284 selects an event handler associated with a sub-event and the object that triggered the hit test.

[0160] In some embodiments, the definition of the corresponding event 287 also includes a delay action that delays the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0161] When a corresponding event recognizer 280 determines that a series of sub-events does not match any event in event definition 286, the corresponding event recognizer 280 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0162] In some embodiments, the corresponding event identifier 280 includes metadata 283 having configurable properties, flags, and / or lists that indicate how the event delivery system should perform delivery of sub-events to actively participating event identifiers. In some embodiments, metadata 283 includes configurable properties, flags, and / or lists that indicate how event identifiers can interact with each other or how they are enabled to interact with each other. In some embodiments, metadata 283 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0163] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 280 activates an event handler 290 associated with the event. In some embodiments, the corresponding event recognizer 280 delivers event information associated with the event to the event handler 290. Activating the event handler 290 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 280 throws a tag associated with the identified event, and the event handler 290 associated with the tag obtains the tag and performs a predefined process.

[0164] In some embodiments, event delivery instructions 288 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler.

[0165] Instead, the sub-event delivery instruction delivers the event information to an event handler associated with the sub-event series or to an actively participating view. The event handler associated with the sub-event series or with an actively participating view receives the event information and performs a predetermined process.

[0166] In some embodiments, data updater 276 creates and updates data used in application 236-1. For example, data updater 276 updates phone numbers used in contact module 237 or stores video files used in video player module. In some embodiments, object updater 277 creates and updates objects used in application 236-1. For example, object updater 277 creates new user interface objects or updates the position of user interface objects. GUI updater 278 updates the GUI. For example, GUI updater 278 prepares display information and sends it to graphics module 232 for display on a touch-sensitive display.

[0167] In some embodiments, the one or more event handlers 290 include or have access to the data updater 276, the object updater 277, and the GUI updater 278. In some embodiments, the data updater 276, the object updater 277, and the GUI updater 278 are included in a single module of the corresponding application 236-1 or application view 291. In other embodiments, they are included in two or more software modules.

[0168] It should be understood that the above discussion of event processing for user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 200, and not all user input is initiated on a touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events defining an event to be recognized.

[0169] Figure 3A portable multifunction device 200 with a touch screen 212 according to some embodiments is shown. The touch screen optionally displays one or more graphics within a user interface (UI) 300. In this embodiment and in other embodiments described below, a user can select one or more graphics by, for example, making gestures on the graphics using one or more fingers 302 (not drawn to scale in the accompanying drawings) or using one or more styluses 303 (not drawn to scale in the accompanying drawings). In some embodiments, selection of one or more graphics will occur when the user interrupts contact with the one or more graphics. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of fingers that have been in contact with the device 200 (from right to left, from left to right, up and / or down). In some specific implementations or in some cases, inadvertent contact with a graphic will not select the graphic. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0170] The device 200 may also include one or more physical buttons, such as a "home" or menu button 304. As previously described, the menu button 304 may be used to navigate to any application 236 in a set of applications executable on the device 200. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 212.

[0171] In one embodiment, the device 200 includes a touch screen 212, a menu button 304, a push button 306 for powering the device on / off and for locking the device, one or more volume adjustment buttons 308, a user identity module (SIM) card slot 310, a headset jack 312, and a docking / charging external port 224. The push button 306 is optionally used to turn the device on / off by pressing the button and keeping the button in the pressed state for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In an alternative embodiment, the device 200 also accepts voice input for activating or deactivating certain functions through the microphone 213. The device 200 also optionally includes one or more contact intensity sensors 265 for detecting the intensity of contact on the touch screen 212 and / or one or more tactile output generators 267 for generating tactile output for the user of the device 200.

[0172] Figure 44 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. Device 400 does not have to be portable. In some embodiments, device 400 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home controller or an industrial controller). Device 400 typically includes one or more processing units (CPUs) 410, one or more network or other communication interfaces 460, a memory 470, and one or more communication buses 420 for interconnecting these components. Communication bus 420 optionally includes circuits (sometimes referred to as chipsets) that interconnect system components and control communications between system components. Device 400 includes an input / output (I / O) interface 430 with a display 440, which is typically a touch screen display. I / O interface 430 also optionally includes a keyboard and / or mouse (or other pointing device) 450 and a touchpad 455, a tactile output generator 457 for generating tactile output on device 400 (e.g., similar to the above reference). Figure 2A The one or more tactile output generators 267 described above), sensors 459 (e.g., optical sensors, acceleration sensors, proximity sensors, touch sensors, and / or sensors similar to those described above) Figure 2A The memory 470 may include one or more storage devices located remotely from the one or more CPUs 410. In some embodiments, the memory 470 stores and stores data stored in the portable multifunction device 200 ( Figure 2A ) in memory 202 of portable multifunction device 200, or a subset thereof. In addition, memory 470 optionally stores additional programs, modules, and data structures that are not present in memory 202 of portable multifunction device 200. For example, memory 470 of device 400 optionally stores a drawing module 480, a presentation module 482, a word processing module 484, a website creation module 486, a disk editing module 488, and / or a spreadsheet module 490, while portable multifunction device 200( Figure 2A )'s memory 202 optionally does not store these modules.

[0173] Figure 4Each of the above elements in the above elements may be stored in one or more of the aforementioned memory devices. Each of the above modules corresponds to an instruction set for performing the above functions. The above modules or programs (e.g., instruction sets) do not have to be implemented as separate software programs, processes or modules, and therefore various subsets of these modules may be combined or otherwise rearranged in various embodiments. In some embodiments, memory 470 may store a subset of the modules and data structures identified above. In addition, memory 470 may store additional modules and data structures not described above.

[0174] Attention is now turned to embodiments of a user interface that may be implemented on, for example, portable multifunction device 200.

[0175] Figure 5A An exemplary user interface of an application menu on portable multifunction device 200 is shown according to some embodiments. A similar user interface can be implemented on device 400. In some embodiments, user interface 500 includes the following elements, or a subset or superset thereof:

[0176] One or more signal strength indicators 502 of one or more wireless communications, such as cellular signals and Wi-Fi signals;

[0177] Time 504;

[0178] Bluetooth indicator 505;

[0179] Battery status indicator 506;

[0180] A tray 508 with common application icons, such as:

[0181] o an icon 516 of the phone module 238 labeled “Phone” which optionally includes an indicator 514 of the number of missed calls or voice messages;

[0182] o an icon 518 of the email client module 240 labeled “Mail”, which optionally includes an indicator 510 of the number of unread emails;

[0183] o An icon 520 labeled "Browser" of the browser module 247; and

[0184] o an icon 522 labeled “iPod” of a video and music player module 252 (also referred to as an iPod (trademark of Apple Inc.) module 252); and

[0185] Icons of other applications, such as:

[0186] o Icon 524 of IM module 241 labeled “Message”;

[0187] o Icon 526 labeled “Calendar” of calendar module 248;

[0188] o Icon 528 labeled “Photos” of the image management module 244;

[0189] o An icon 530 labeled “Camera” of the camera module 243;

[0190] o Icon 532 labeled “Online Video” of the online video module 255;

[0191] o Icon 534 labeled "Stock Market" of the Stock Market widget 249-2;

[0192] o an icon 536 labeled “Map” of the map module 254;

[0193] o Icon 538 labeled "Weather" of the Weather widget 249-1;

[0194] o Icon 540 labeled “Clock” of the Alarm Clock widget 249-4;

[0195] o an icon 542 labeled “Fitness Support” of the fitness support module 242;

[0196] o An icon 544 of the notepad module 253 labeled "Notepad"; and

[0197] o An icon 546 labeled “Settings” for a settings application or module that provides access to settings for the device 200 and its various applications 236.

[0198] Please note that Figure 5A The icon labels shown in are exemplary only. For example, icon 522 of video and music player module 252 may optionally be labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label of a corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0199] Figure 5B A touch-sensitive surface 551 (eg, touch screen display 212) is shown having a touch-sensitive surface 551 (eg, Figure 4 tablet or touch pad 455) device (e.g., Figure 4Device 400 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 459) for detecting contact intensity on touch-sensitive surface 551, and / or one or more tactile output generators 457 for generating tactile output for a user of device 400.

[0200] Although some of the examples that follow will be given with reference to input on a touch screen display 212 (where a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 5B In some embodiments, the touch-sensitive surface (e.g., Figure 5B 551) has a main axis (e.g., Figure 5B 553) corresponding to the principal axis (for example, Figure 5B According to these embodiments, the device detects a position corresponding to a corresponding position on the display (e.g., Figure 5B , 560 corresponds to 568 and 562 corresponds to 570) at a contact with touch-sensitive surface 551 (eg, Figure 5B Thus, on a touch-sensitive surface (e.g., Figure 5B 551) and a display of a multi-function device (e.g., Figure 5B When the user input detected by the device on the touch-sensitive surface (e.g., contacts 560 and 562 and their movement) is separated, the device is used to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0201] In addition, although the following examples are given primarily with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of a cursor along the path of the swipe (e.g., instead of movement of the contact).

[0202] As another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting contact and then ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and finger contact are optionally used simultaneously.

[0203] Fig. 6AAn exemplary personal electronic device 600 is shown. Device 600 includes a body 602. In some embodiments, device 600 may include a device 602 for devices 200 and 400 (e.g., Figures 2A to 4 ) Some or all of the features described in ). In some embodiments, the device 600 has

[0204] Touch-sensitive display screen 604, hereinafter referred to as touch screen 604. As an alternative or in addition to touch screen 604, device 600 has a display and a touch-sensitive surface. As in the case of device 200 and device 400, in some embodiments, touch screen 604 (or touch-sensitive surface) may have one or more intensity sensors for detecting the intensity of contact (e.g., touch) being applied. One or more intensity sensors of touch screen 604 (or touch-sensitive surface) can provide output data representing the intensity of touch. The user interface of device 600 can respond to touch based on the intensity of touch, which means that touches of different intensities can invoke different user interface operations on device 600.

[0205] Technology for detecting and processing touch intensity can be found, for example, in related applications: International Patent Application PCT / US2013 / 040061, filed on May 8, 2013, entitled “Device, Method, and Graphical User Interface for Displaying UserInterface Objects Corresponding to an Application,” and International Patent Application PCT / US2013 / 069483, filed on November 11, 2013, entitled “Device, Method, and Graphical UserInterface for Transitioning Between Touch Input to Display OutputRelationships,” each of which is hereby incorporated by reference in its entirety.

[0206] In some embodiments, the device 600 has one or more input mechanisms 606 and 608. The input mechanisms 606 and 608, if included, can be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 600 has one or more attachment mechanisms. Such attachment mechanisms, if included, can allow the device 600 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watchbands, bracelets, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms can allow the user to wear the device 600.

[0207] Figure 6B An exemplary personal electronic device 600 is shown. In some embodiments, the device 600 may include a reference Figure 2A , Figure 2B and Figure 4 Some or all of the components described. Device 600 has a bus 612 that operably couples an I / O portion 614 to one or more computer processors 616 and a memory 618. I / O portion 614 may be connected to a display 604, which may have a touch-sensitive component 622 and optionally a touch-intensity sensitive component 624. In addition, I / O portion 614 may be connected to a communication unit 630 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular and / or other wireless communication technologies. Device 600 may include input mechanisms 606 and / or 608. For example, input mechanism 606 may be a rotatable input device or a depressible and rotatable input device. In some examples, input mechanism 608 may be a button.

[0208] In some examples, input mechanism 608 may be a microphone. Personal electronic device 600 may include various sensors, such as GPS sensor 632, accelerometer 634, orientation sensor 640 (e.g., compass), gyroscope 636, motion sensor 638, and / or combinations thereof, all of which may be operably connected to I / O portion 614.

[0209] The memory 618 of the personal electronic device 600 may be a non-transitory computer-readable storage medium for storing computer-executable instructions that, when executed by one or more computer processors 616, may cause the computer processors to perform processes 800 to 900 ( Figures 8 to 9 ) described below. The computer executable instructions may also be stored and / or transmitted in any non-transitory computer readable storage medium for use by or in conjunction with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other system that can obtain instructions from an instruction execution system, apparatus, or device and execute instructions. The personal electronic device 600 is not limited to Figure 6B components and configurations, but may include other components or additional components in a variety of configurations.

[0210] As used herein, the term "indicator" refers to an indication that can be displayed on device 200, 400, and / or 600 (FIG. 2, Figure 4 6 ). For example, an image (e.g., an icon), a button, and text (e.g., a link) can each constitute an affordance.

[0211] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface that a user is interacting with. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that a focus selector is displayed on a touch-sensitive surface (e.g., a button, window, slider, or other user interface element) when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element). Figure 4 Touchpad 455 or Figure 5B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 551 in FIG. 5 , the particular user interface element is adjusted according to the detected input. In the case that a touch screen display (e.g., Figure 2A A touch sensitive display system 212 or Figure 5A In some implementations of the touch screen 212 in FIG. 1 , the contact detected on the touch screen acts as a “focus selector”. 5 In some implementations, the focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these implementations, the focus selector moves in accordance with the movement of the focus between different areas of the user interface. Regardless of the specific form that the focus selector takes, the focus selector is generally a user interface element (or contact on the touch screen display) that is controlled by the user to deliver the user's intended interaction with the user interface (e.g., by indicating to the device an element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touch screen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user intends to activate the corresponding button (rather than other user interface elements shown on the device's display).

[0212] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of a contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact starts to move, before contact ends, before or after contact intensity is detected to increase, and / or before or after contact intensity is detected to decrease). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half-maximum value of the contact intensity, the 90% maximum value of the contact intensity, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the contact intensity over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an action.

[0213] For example, the set of one or more strength thresholds may include a first strength threshold and a second strength threshold. In this example, a contact whose characteristic strength does not exceed the first threshold results in a first operation, a contact whose characteristic strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose characteristic strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the characteristic strength and one or more thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding operation or abandon the corresponding operation), rather than to determine whether to perform the first operation or the second operation.

[0214] In some embodiments, a portion of a gesture is identified for the purpose of determining a characteristic strength. For example, a touch-sensitive surface may receive a continuous swipe contact that transitions from a starting position and reaches an end position, where the strength of the contact increases. In this example, the characteristic strength of the contact at the end position may be based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only a portion of the swipe contact at the end position). In some embodiments, a smoothing algorithm may be applied to the strength of the swipe gesture before determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or depressions in the swipe contact strength for the purpose of determining the characteristic strength.

[0215] The intensity of a contact on a touch-sensitive surface can be characterized relative to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or a trackpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from the operation typically associated with clicking a button of a physical mouse or a trackpad. In some embodiments, when a contact is detected with a characteristic intensity below the light press intensity threshold (e.g., and above a nominal contact detection intensity threshold, contacts below the nominal contact detection intensity threshold are no longer detected), the device will move the focus selector in accordance with the movement of the contact on the touch-sensitive surface without performing an operation associated with the light press intensity threshold or the deep press intensity threshold.

[0216] In general, these intensity thresholds are consistent across different sets of user interface drawings unless otherwise stated.

[0217] An increase in the characteristic intensity of a contact from an intensity below a light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in the characteristic intensity of a contact from an intensity below a deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in the characteristic intensity of a contact from an intensity below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting contact on the touch surface. A decrease in the characteristic intensity of a contact from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting contact lifted from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0218] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the strength of the contact (or multiple contacts) increases to above a press input strength threshold. In some embodiments, the corresponding operation is performed in response to detecting that the corresponding contact strength increases to above the press input strength threshold (e.g., a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the corresponding contact strength to above the press input strength threshold and the contact strength subsequently decreases to below the press input strength threshold, and the corresponding operation is performed in response to detecting that the corresponding contact strength subsequently decreases to below the press input threshold (e.g., an "up stroke" of the corresponding press input).

[0219] In some embodiments, the device employs intensity hysteresis to avoid unexpected inputs sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the corresponding contact intensity to above the press input intensity threshold and a subsequent decrease in the contact intensity to below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the corresponding contact intensity subsequently decreases to below the hysteresis intensity threshold (e.g., an "upward stroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact intensity increases from an intensity equal to or below the hysteresis intensity threshold to an intensity equal to or above the press input intensity threshold and, optionally, that the contact intensity subsequently decreases to an intensity equal to or below the hysteresis intensity, and a corresponding operation is performed in response to detecting the press input (e.g., an increase in contact intensity or a decrease in contact intensity, depending on the circumstances).

[0220] For ease of explanation, optionally, a description of an operation performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input is triggered in response to detecting any of the following: contact intensity increasing to above the press input intensity threshold, contact intensity increasing from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, contact intensity decreasing to below the press input intensity threshold, and / or contact intensity decreasing to below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where an operation is described as being performed in response to detecting a decrease in contact intensity below the press input intensity threshold, the operation is optionally performed in response to detecting a decrease in contact intensity below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold.

[0221] 3. Digital Assistant System

[0222] Fig. 7A A block diagram of a digital assistant system 700 according to various examples is shown. In some examples, the digital assistant system 700 may be implemented on a stand-alone computer system. In some examples, the digital assistant system 700 may be distributed across multiple computers. In some examples, some of the modules and functions of the digital assistant may be divided into a server portion and a client portion, where the client portion is located on one or more user devices (e.g., devices 104, 122, 200, 400, or 600) and communicates with the server portion (e.g., server system 108) via one or more networks, such as Figure 1In some examples, the digital assistant system 700 may be Figure 1 Specific implementation of the server system 108 (and / or DA server 106) shown in . It should be noted that the digital assistant system 700 is only one example of a digital assistant system, and the digital assistant system 700 may have more or fewer components than shown, may combine two or more components, or may have a different configuration or arrangement of components. Fig. 7A The various components shown in may be implemented in hardware, software instructions for execution by one or more processors, firmware (including one or more signal processing integrated circuits and / or application specific integrated circuits), or a combination thereof.

[0223] The digital assistant system 700 may include a memory 702, one or more processors 704, an input / output (I / O) interface 706, and a network communication interface 708. These components may communicate with each other via one or more communication buses or signal lines 710.

[0224] In some examples, memory 702 may include non-transitory computer-readable media such as high-speed random access memory and / or non-volatile computer-readable storage media (e.g., one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

[0225] In some examples, the I / O interface 706 can couple the input / output devices 716 of the digital assistant system 700, such as a display, keyboard, touch screen, and microphone, to the user interface module 722. The I / O interface 706 in conjunction with the user interface module 722 can receive user input (e.g., voice input, keyboard input, touch input, etc.) and process the input accordingly. In some examples, for example, when the digital assistant is implemented on a stand-alone user device, the digital assistant system 700 may include interfaces respectively corresponding to Figure 2A , Figure 4 , FIG. 6A to FIG. 6B Any of the components and I / O communication interfaces described for devices 200, 400, or 600 in the digital assistant system. In some examples, the digital assistant system 700 may represent a server portion of a specific implementation of a digital assistant, and may interact with a user through a client-side portion located on a user device (e.g., device 104, 200, 400, or device 600).

[0226] In some examples, the network communication interface 708 may include one or more wired communication ports 712, and / or wireless transmission and reception circuits 714. One or more wired communication ports may receive and send communication signals via one or more wired interfaces such as Ethernet, Universal Serial Bus (USB), FireWire, etc. The wireless circuit 714 may receive RF signals and / or optical signals from a communication network and other communication devices, and send RF signals and / or optical signals to the communication network and other communication devices. Wireless communication may use any of a variety of communication standards, protocols and technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol. The network communication interface 708 may enable the digital assistant system 700 to communicate with other devices through a network such as the Internet, an intranet and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN) and / or a metropolitan area network (MAN)).

[0227] In some examples, the memory 702 or the computer-readable storage medium of the memory 702 may store programs, modules, instructions, and data structures, including all or a subset of the following: operating system 718, communication module 720, user interface module 722, one or more application programs 724, and digital assistant module 726. In particular, the memory 702 or the computer-readable storage medium of the memory 702 may store instructions for performing the processes 800, 900 described below. The one or more processors 704 may execute these programs, modules, and instructions, and read data from or write data to data structures.

[0228] The operating system 718 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) may include various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitating communication between various hardware, firmware, and software components.

[0229] The communication module 720 can facilitate communication between the digital assistant system 700 and other devices through the network communication interface 708. For example, the communication module 720 can communicate with electronic devices (such as Figure 2A , Figure 4 , FIG. 6A to FIG. 6B The communication module 720 may also include various components for processing data received by the wireless circuit 714 and / or the wired communication port 712.

[0230] The user interface module 722 may receive commands and / or inputs from a user (e.g., from a keyboard, touch screen, pointing device, controller, and / or microphone) via the PO interface 706 and generate user interface objects on the display. The user interface module 722 may also prepare output (e.g., speech, sound, animation, text, icon, vibration, tactile feedback, lighting, etc.) and deliver it to the user via the PO interface 706 (e.g., through a display, audio channel, speaker, touchpad, etc.).

[0231] Applications 724 may include programs and / or modules configured to be executed by one or more processors 704. For example, if the digital assistant system is implemented on a stand-alone user device, applications 724 may include user applications such as games, calendar applications, navigation applications, or mail applications. If the digital assistant system 700 is implemented on a server, applications 724 may include, for example, resource management applications, diagnostic applications, or scheduling applications.

[0232] The memory 702 may also store a digital assistant module 726 (or the server portion of the digital assistant). In some examples, the digital assistant module 726 may include the following submodules or subsets or supersets thereof: an input / output processing module 728, a speech-to-text (STT) processing module 730, a natural language processing module 732, a dialog flow processing module 734, a task flow processing module 736, a service processing module 738, and a speech synthesis module 740. Each of these modules may have access to one or more of the following systems or data and models of the digital assistant module 726 or subsets or supersets thereof: a knowledge ontology 760, a vocabulary index 744, user data 748, a task flow model 754, a service model 756, and an ASR system.

[0233] In some examples, using the processing modules, data, and models implemented in the digital assistant module 726, the digital assistant can perform at least some of the following operations: convert voice input into text; recognize user intent expressed in natural language input received from the user; proactively elicit and obtain information needed to infer user intent (e.g., by disambiguating words, games, intents, etc.); determine a task flow for satisfying the inferred intent; and execute the task flow to satisfy the inferred intent.

[0234] In some examples, such as Figure 7B As shown in FIG. , the I / O processing module 728 can be Fig. 7A The I / O device 716 in the Fig. 7AThe network communication interface 708 in the system interacts with a user device (e.g., device 104, 200, 400, or 600) to obtain user input (e.g., voice input) and provide a response to the user input (e.g., as voice output). The I / O processing module 728 may optionally obtain context information associated with the user input from the user device along with or shortly after receiving the user input. The context information may include user-specific data, vocabulary, and / or preferences related to the user input. In some examples, the context information also includes the software state and hardware state of the user device when the user request is received, and / or information related to the user's surrounding environment when the user request is received. In some examples, the I / O processing module 728 may also send follow-up questions related to the user request to the user and receive answers from the user. When the user request is received by the I / O processing module 728 and the user request may include voice input, the I / O processing module 728 may forward the voice input to the STT processing module 730 (or speech recognizer) for speech-to-text conversion.

[0235] The STT processing module 730 may include one or more ASR systems. The one or more ASR systems may process the speech input received by the I / O processing module 728 to generate recognition results. Each ASR system may include a front-end speech preprocessor. The front-end speech preprocessor may extract representative features from the speech input.

[0236] For example, the front-end speech preprocessor may perform Fourier transform on the speech input to extract the spectral features characterizing the speech input as a sequence of representative multidimensional vectors. In addition, each ASR system may include one or more speech recognition models (e.g., sound model and / or language model), and may implement one or more speech recognition engines. Examples of speech recognition models may include hidden Markov models, Gaussian mixture models, deep neural network models, n-gram language models, and other statistical models. Examples of speech recognition engines may include engines based on dynamic time warping and engines based on weighted finite state transducers (WFST). One or more speech recognition models and one or more speech recognition engines may be used to process the extracted representative features of the front-end speech preprocessor to produce intermediate recognition results (e.g., phonemes, phoneme strings, and subwords), and ultimately produce text recognition results (e.g., sequences of words, word strings, or symbols). In some examples, the speech input may be processed at least in part by a third-party service or processed on a user's device (e.g., device 104, 200, 400, or 600) to produce a recognition result. Once the STT processing module 730 generates a recognition result including a text string (e.g., a word, or a sequence of words, or a sequence of symbols), the recognition result may be transmitted to the natural language processing module 732 for intent inference.

[0237] Further details regarding speech-to-text processing are described in U.S. Utility Patent Application Serial No. 13 / 236,942, filed on September 20, 2011, entitled “Consolidating Speech Recognition Results,” the entire disclosure of which is incorporated herein by reference.

[0238] In some examples, the STT processing module 730 may include a vocabulary of recognizable words, and / or the vocabulary may be accessed via the phonetic alphabet conversion module 731. Each vocabulary word may be associated with one or more candidate pronunciations of the word represented by a speech recognition phonetic alphabet. Specifically, the vocabulary of recognizable words may include words associated with multiple candidate pronunciations. For example, the vocabulary may include the word "tomato" associated with candidate pronunciations of / ts'meirou / and / to'matoo / . In addition, vocabulary words may be associated with custom candidate pronunciations based on previous voice input from the user. Such custom candidate pronunciations may be stored in the STT processing module 730 and may be associated with a specific user via a user profile on the device. In some examples, the candidate pronunciations of a word may be determined based on the spelling of the word and one or more linguistic and / or phonetic rules. In some examples, the candidate pronunciations may be manually generated, for example, manually generated based on a known standard pronunciation.

[0239] In some examples, the candidate pronunciations may be ranked based on their prevalence. The ranking can be higher than Because the former is a more common pronunciation (e.g., among all users, for users in a particular geographic region, or for any other suitable subset of users). In some examples, the candidate pronunciations can be ranked based on whether the candidate pronunciation is a custom candidate pronunciation associated with the user. For example, a custom candidate pronunciation can be ranked higher than a standard candidate pronunciation. This can be used to identify proper nouns with unique pronunciations that deviate from the standard pronunciation. In some examples, the candidate pronunciation can be associated with one or more speech features (such as geographic origin, country, or ethnicity). For example, the candidate pronunciation can be associated with the United States, while the candidate pronunciation The candidate pronunciations may be associated with the United Kingdom. In addition, the ranking of the candidate pronunciations may be based on one or more characteristics of the user (e.g., geographic origin, country, ethnicity, etc.) in a user profile stored on the device. For example, it may be determined from the user profile that the user is associated with the United States. Based on the user being associated with the United States, the candidate pronunciations may be ranked (associated with the United States) can be ranked higher than the candidate pronunciations (Associated with the United Kingdom.) In some examples, one of the ranked candidate pronunciations may be selected as the predicted pronunciation (eg, the most likely pronunciation).

[0240] When speech input is received, the STT processing module 730 may be used to determine (e.g., using an acoustic model) a phoneme corresponding to the speech input, and then attempt to determine (e.g., using a language model) a word that matches the phoneme. For example, if the STT processing module 730 may first identify a phoneme sequence corresponding to a portion of the speech input, It can then determine based on the lexical index 744 that the sequence corresponds to the word "tomato".

[0241] In some examples, the STT processing module 730 may use fuzzy matching techniques to determine the words in the utterance. Thus, for example, the STT processing module 730 may determine the phoneme sequence corresponds to the word "tomato", even though this particular phoneme sequence is not one of the candidate phoneme sequences for that word.

[0242] The natural language processing module 732 ("natural language processor") of the digital assistant may obtain a sequence of words or symbols ("symbol sequence") generated by the STT processing module 730 and attempt to associate the symbol sequence with one or more "executable intents" recognized by the digital assistant. An "executable intent" may represent a task that can be executed by the digital assistant and may have an associated task flow implemented in the task flow model 754. An associated task flow may be a series of programmed actions and steps taken by the digital assistant to perform a task. The scope of the digital assistant's capabilities may depend on the number and variety of task flows that have been implemented and stored in the task flow model 754, or in other words, on the number and variety of "executable intents" recognized by the digital assistant. However, the effectiveness of the digital assistant may also depend on the assistant's ability to infer the correct "one or more executable intents" from a user request expressed in natural language.

[0243] In some examples, in addition to the sequence of words or symbols obtained from the STT processing module 730, the natural language processing module 732 may also receive context information associated with the user request (e.g., from the I / O processing module 728). The natural language processing module 732 may optionally use the context information to clarify, supplement, and / or further qualify the information contained in the sequence of symbols received from the STT processing module 730. The context information may include, for example: user preferences, hardware and / or software state of the user's device, sensor information collected before, during, or shortly after the user's request, previous interactions (e.g., conversations) between the digital assistant and the user, etc. As described herein, the context information may be dynamic and may change with the time, location, content, and other factors of the conversation.

[0244] In some examples, natural language processing may be based on, for example, knowledge ontology 760. Knowledge ontology 760 may be a hierarchical structure containing many nodes, each node representing an "executable intent" or an "attribute" related to one or more of an "executable intent" or other "attributes." As described above, an "executable intent" may represent a task that the digital assistant is able to perform, i.e., the task is "executable" or can be performed.

[0245] A "property" may represent a parameter associated with an executable intent or a sub-aspect of another property. Links between executable intent nodes and property nodes in ontology 760 may define how the parameter represented by the property node is related to the task represented by the executable intent node.

[0246] In some examples, ontology 760 may be composed of executable intent nodes and attribute nodes. Within ontology 760, each executable intent node may be directly linked to or linked to one or more attribute nodes through one or more intermediate attribute nodes. Similarly, each attribute node may be directly linked to or linked to one or more executable intent nodes through one or more intermediate attribute nodes. For example, Figure 7C As shown, ontology 760 may include a "restaurant reservation" node (i.e., an executable intent node). The attribute nodes "restaurant", "date / time" (for reservations), and "number of people traveling together" may each be directly linked to an executable intent node (i.e., the "restaurant reservation" node).

[0247] In addition, the attribute nodes "cuisine", "price range", "telephone number", and "location" can be child nodes of the attribute node "restaurant", and each can be linked to the "restaurant reservation" node (i.e., an executable intent node) through the intermediate attribute node "restaurant". Figure 7C As shown, ontology 760 may also include a "set reminder" node (i.e., another executable intent node). The attribute nodes "date / time" (for setting reminders) and "subject" (for reminders) may each be linked to the "set reminder" node.

[0248] The task of setting a reminder is related to both, so the attribute node "Date / Time" can be linked to both the "Restaurant Reservation" node and the "Set Reminder" node in the knowledge ontology 760.

[0249] Executable intent nodes, together with their linked concept nodes, can be described as "domains". In this discussion, each domain can be associated with a corresponding executable intent and involves a set of nodes (and the relationships between these nodes) associated with a particular executable intent. For example, Figure 7CThe knowledge ontology 760 shown in the figure may include an example of a restaurant reservation domain 762 within the knowledge ontology 760 and an example of a reminder domain 764. The restaurant reservation domain includes an executable intent node "restaurant reservation", attribute nodes "restaurant", "date / time" and "number of companions", and sub-attribute nodes "cuisine", "price range", "phone number" and "location". The reminder domain 764 may include an executable intent node "set reminder" and attribute nodes "subject" and "date / time". In some examples, the knowledge ontology 760 may be composed of multiple domains. Each domain may share one or more attribute nodes with one or more other domains. For example, in addition to the restaurant reservation domain 762 and the reminder domain 764, the "date / time" attribute node may also be associated with many different domains (e.g., itinerary domain, travel reservation domain, movie ticket domain, etc.).

[0250] although Figure 7C Two example domains within ontology 760 are shown, but other domains may include, for example, "find a movie," "make a phone call," "find directions," "schedule a meeting," "send a message," and "provide answers to questions," "read a list," "provide navigation instructions," "provide instructions for a task," etc. The "send message" domain may be associated with a "send message" executable intent node, and may also include attribute nodes such as "one or more recipients," "message type," and "message body." The attribute node "recipients" may be further qualified, for example, by child attribute nodes such as "recipient name" and "message address."

[0251] In some examples, ontology 760 may include all domains (and therefore executable intents) that a digital assistant can understand and act upon. In some examples, ontology 760 may be modified, such as by adding or removing entire domains or nodes, or by modifying the relationships between nodes within ontology 760.

[0252] In some examples, nodes associated with multiple related executable intents may be clustered under a "superdomain" in the knowledge ontology 760. For example, the "travel" superdomain may include a cluster of attribute nodes and executable intent nodes related to travel. Executable intent nodes related to travel may include "flight booking", "hotel booking", "car rental", "route planning", "finding points of interest", etc. Executable intent nodes under the same superdomain (e.g., the "travel" superdomain) may have multiple shared attribute nodes. For example, executable intent nodes for "airline booking", "hotel booking", "car rental", "route planning", and "finding points of interest" may share one or more of the attribute nodes "starting location", "destination", "departure date / time", "arrival date / time", and "number of people traveling with you".

[0253] In some examples, each node in the knowledge ontology 760 can be associated with a set of words and / or phrases related to the attribute or executable intent represented by the node. The corresponding set of words and / or phrases associated with each node can be a so-called "vocabulary" associated with the node. The corresponding set of words and / or phrases associated with each node can be stored in a vocabulary index 744 associated with the attribute or executable intent represented by the node. For example, returning Figure 7B , the vocabulary associated with the node of the "restaurant" attribute may include words such as "food", "drinks", "cuisine", "hunger", "eat", "pizza", "fast food", "meal", etc. For another example, the vocabulary associated with the node of the "make phone call" executable intent may include words and phrases such as "call", "make a phone call", "dial", "talk to...", "call the number", "call", etc. The vocabulary index 744 may optionally include words and phrases in different languages.

[0254] The natural language processing module 732 may receive a sequence of symbols (e.g., a text string) from the STT processing module 730 and determine which nodes are involved by the words in the sequence of symbols. In some examples, if a word or phrase in the sequence of symbols is found to be associated with one or more nodes in the knowledge ontology 760 (via the vocabulary index 744), the word or phrase may "trigger" or "activate" these nodes. Based on the number and / or relative importance of the activated nodes, the natural language processing module 732 may select one of the executable intents as the task that the user intends the digital assistant to perform. In some examples, the domain with the most "triggered" nodes may be selected. In some examples, the domain with the highest confidence (e.g., based on the relative importance of its various triggered nodes) may be selected. In some examples, the domain may be selected based on a combination of the number and importance of the triggered nodes. In some examples, additional factors are also considered in the process of selecting nodes, such as whether the digital assistant has previously correctly interpreted similar requests from the user.

[0255] User data 748 may include user-specific information, such as user-specific vocabulary, user preferences, user address, the user's default language and second language, the user's contact list, and other short-term or long-term information for each user. In some examples, natural language processing module 732 may use user-specific information to supplement the information contained in the user input to further qualify the user's intent. For example, for a user request to "invite my friends to my birthday party," natural language processing module 732 may be able to access user data 748 to determine who the "friends" are and when and where the "birthday party" will be held, without requiring the user to explicitly provide such information in their request.

[0256] Additional details of searching knowledge ontology based on symbolic strings are described in U.S. utility patent application serial number 12 / 341,743, entitled “Method and Apparatus for Searching Using An Active Ontology,” filed on December 22, 2008, the entire disclosure of which is incorporated herein by reference.

[0257] In some examples, once the natural language processing module 732 identifies an executable intent (or domain) based on a user request, the natural language processing module 732 may generate a structured query to represent the identified executable intent. In some examples, the structured query may include parameters for one or more nodes within the domain of the executable intent, and at least some of the parameters are filled with specific information and requirements specified in the user request. For example, a user may say "help me reserve a seat at 7pm at the sushi restaurant." In this case, the natural language processing module 732 may be able to correctly identify the executable intent as "restaurant reservation" based on the user input. According to the knowledge ontology, a structured query for the "restaurant reservation" domain may include parameters such as {cuisine}, {time}, {date}, {number of companions}, etc. In some examples, based on voice input and text derived from the voice input using the STT processing module 730, the natural language processing module 732 may generate a partial structured query for the restaurant reservation domain, wherein the partial structured query includes parameters {cuisine = "sushi category"} and {time = "7pm"}. However, in this example, the user utterance contains insufficient information to complete the structured query associated with the domain. Therefore, based on the currently available information, other necessary parameters such as {number of people in the group} and {date} may not be specified in the structured query. In some examples, the natural language processing module 732 may use the received contextual information to fill in some parameters of the structured query. For example, in some examples, if a sushi restaurant "nearby" is requested, the natural language processing module 732 may use the GPS coordinates from the user device to fill in the {location} parameter in the structured query.

[0258] In some examples, the natural language processing module 732 may transmit the generated structured query (including any completed parameters) to the task flow processing module 736 ("task flow processor"). The task flow processing module 736 may be configured to receive the structured query from the natural language processing module 732, complete the structured query (if necessary), and perform the actions required to "complete" the user's final request. In some examples, the various processes necessary to complete these tasks may be provided in the task flow model 754. In some examples, the task flow model 754 may include processes for obtaining additional information from the user, and task flows for performing actions associated with executable intents.

[0259] As described above, in order to complete the structured query, the task flow processing module 736 may need to initiate an additional dialogue with the user in order to obtain additional information and / or clarify the utterances that may be ambiguous. When it is necessary to conduct such an interaction, the task flow processing module 736 may call the dialogue flow processing module 734 to participate in the dialogue with the user. In some examples, the dialogue flow processing module 734 may determine how (and / or when) to request additional information from the user, and receive and process the user response. The question may be provided to the user through the I / O processing module 728 and an answer may be received from the user. In some examples, the dialogue processing module 734 may present the dialogue output to the user via audio and / or visual output, and receive input from the user via verbal or physical (e.g., click) response. Continuing the above example, when the task flow processing module 736 calls the dialogue flow processing module 734 to determine the "number of people traveling together" and "date" information for the structured query associated with the domain "restaurant reservation", the dialogue flow processing module 734 may generate questions such as "how many people in a row?" and "which day to book?" to pass to the user. Once a response is received from the user, the dialog flow processing module 734 may fill in the structured query with the missing information, or pass the information to the task flow processing module 736 to complete the missing information based on the structured query.

[0260] Once the task flow processing module 736 has completed the structured query for the executable intent, the task flow processing module 736 can proceed to execute the final task associated with the executable intent. Therefore, the task flow processing module 736 can execute the steps and instructions in the task flow model according to the specific parameters contained in the structured query. For example, the task flow model for the executable intent "restaurant reservation" may include steps and instructions for contacting a restaurant and actually requesting a reservation for a specific number of people at a specific time. For example, using a structured query such as: {restaurant reservation, restaurant = ABC Cafe, date = 3 / 12 / 2012, time = 7pm, number of people = 5}, the task flow processing module 736 can perform the following steps: (1) Log in to the server of ABC Cafe or a restaurant reservation system such as (2) entering date, time, and party size information into a form on the website, (3) submitting the form, and (4) making a calendar entry for the reservation on the user's calendar.

[0261] In some examples, the task flow processing module 736 can complete the task requested in the user input or provide the informational answer requested in the user input with the assistance of the service processing module 738 ("service processing module"). For example, the service processing module 738 can initiate a phone call, set a calendar entry, call a map search, call other user applications installed on the user device or interact with the other applications, and call a third-party service (e.g., a restaurant reservation portal, a social networking site, a bank portal, etc.), or interact with a third-party service on behalf of the task flow processing module 736. In some examples, the protocols and application programming interfaces (APIs) required for each service can be specified by the corresponding service model in the service model 756. The service processing module 738 can access the appropriate service model for the service and generate a request for the service according to the protocol and API required by the service based on the service model.

[0262] For example, if the restaurant has enabled an online reservation service, the restaurant may submit a service model that specifies the necessary parameters for making a reservation and transmits the values ​​of the necessary parameters to the API of the online reservation service. When requested by the task flow processing module 736, the service processing module 738 may use the web address stored in the service model to establish a network connection with the online reservation service and send the necessary parameters for the reservation (e.g., time, date, number of companions) to the online reservation interface in a format that complies with the API of the online reservation service.

[0263] In some examples, the natural language processing module 732, the dialogue processing module 734, and the task flow processing module 736 can be used together and repeatedly to infer and define the user's intention, obtain information, further clarify and refine the user's intention, and finally generate a response (i.e., output to the user or complete the task) to meet the user's intention. The generated response can be a dialogue response to the voice input that at least partially meets the user's intention. In addition, in some examples, the generated response can be output as a voice output. In these examples, the generated response can be sent to a speech synthesis module 740 (e.g., a speech synthesizer), in which the generated response can be processed to synthesize the dialogue response in the form of speech. In other examples, the generated response can be data content related to satisfying the user's request in the voice input.

[0264] The speech synthesis module 740 may be configured to synthesize speech output for presentation to the user. The speech synthesis module 740 synthesizes speech output based on the text provided by the digital assistant. For example, the generated dialogue response may be in the form of a text string. The speech synthesis module 740 may convert the text string into an audible speech output. The speech synthesis module 740 may use any suitable speech synthesis technology to generate speech output from text, including but not limited to: concatenation synthesis, unit selection synthesis, diphone synthesis, domain-specific synthesis, formant synthesis, pronunciation synthesis, synthesis based on a hidden Markov model (HMM), and sine wave synthesis. In some examples, the speech synthesis module 740 may be configured to synthesize individual words based on phoneme strings corresponding to these words. For example, a phoneme string may be associated with a word in the generated dialogue response. The phoneme string may be stored in metadata associated with the word. The speech synthesis model 740 may be configured to directly process the phoneme string in the metadata to synthesize the word in speech form.

[0265] In some examples, instead of using speech synthesis module 740 (or in addition), speech synthesis can be performed on a remote device (e.g., server system 108), and the synthesized speech can be sent to the user device for output to the user. For example, this can occur in some specific implementations where the output of the digital assistant is generated at the server system. And because the server system typically has more processing power or more resources than the user device, it is possible to obtain higher quality speech output than client-side synthesis would achieve.

[0266] Additional details about the digital assistant can be found in U.S. utility patent application Ser. No. 12 / 987,982, filed on Jan. 10, 2011, and entitled “Intelligent Automated Assistant,” and U.S. utility patent application Ser. No. 13 / 251,088, filed on Sept. 30, 2011, and entitled “Generating and Processing Task Items That Represent Tasks to Perform,” the disclosures of which are incorporated herein by reference in their entirety.

[0267] 4. Digital Assistant Operation Process

[0268] Figure 8A flowchart of a process 800 for operating a digital assistant according to some embodiments is shown. For example, process 800 is performed using one or more electronic devices (e.g., devices 104, 108, 200, 400, or 600) that implement a digital assistant. In some examples, a client-server system (e.g., system 100) is used to perform process 800, and the frames of process 800 can be divided between a server (e.g., DA server 106) and a client device in any manner. In other examples, process 800 can be divided between a server and multiple client devices (e.g., a mobile phone and a smart watch). Therefore, although part of process 800 is described herein as being performed by a specific device of a client-server system, it should be understood that process 800 is not limited thereto. In other examples, process 800 is performed using only a client device (e.g., user device 104) or only multiple client devices. In process 800, some frames are optionally combined, the order of some frames is optionally changed, and some frames are optionally omitted. In some examples, additional steps can be performed in conjunction with process 800.

[0269] At block 805, natural language user input is received by a user device such as Figure 1 The natural language user input is received by the user device 104). Natural language user input is, for example, voice input included in audio input or text input. In some examples, the natural language user input may include a request for the user device and / or another device to perform a task. For example, in the example of "send a car to 1200 Main Street", the natural language user input may include a request for the user device to book a car using a ride booking service. In some examples, the natural language user input may further specify one or more parameters for the requested task. For example, "1200 Main Street" specifies the pick-up and drop-off location for booking a car. In this example, "order from Domino's as I usually do", this natural language user input may include a request for the user device to order food from the pizza chain Domino's, and "as I usually do" may further specify what food to order based on the context.

[0270] At block 810, an intent and optionally one or more parameters associated with the intent are identified. For example, the intent and parameters may be derived from a natural language user input. In some examples, a text string is obtained from the natural language user input, and the intent and parameters are derived from the text string. The text string may be obtained using any known speech-to-text method.

[0271] As described, an intent may correspond to a task requested by a user. Thus, identifying (e.g., determining) an intent may include identifying a task specified in a natural language user input and / or inferring an intent corresponding to a task requested based on the language and / or context of the natural language user input. An intent may correspond to any type of task performed by a user device, and in particular, may correspond to a task performed by one or more applications of a user device, as described in further detail below.

[0272] Because a first type of device can perform different tasks than a second type of device, intents and / or parameters can be identified based on the type of device. For example, a first type of device can perform tasks associated with an intent to reserve a car, and a second type of device can perform tasks associated with an intent to provide a status of a car reservation after the reservation is made.

[0273] In some examples, intents are associated with (e.g., included in) one or more domains (e.g., intent categories, intent sets). Each domain may include a particular category of intents, allowing intents to be intuitively grouped. For example, intents for booking a car, canceling a car reservation, and / or any other intent for tasks typically associated with ride reservations may be included in the ride reservation domain. In another example, intents for checking in, canceling a flight, rescheduling a flight, getting flight information, and / or any other intent for tasks typically associated with air travel may be included in the air travel domain. In another example, intents for providing directions, getting traffic information, and / or any other intent for tasks typically associated with navigation may be included in the navigation domain. In yet another example, intents for paying, collecting money, and / or any other intent for tasks typically associated with financial transactions may be included in the financial transaction domain.

[0274] Identifying parameters may include identifying a portion of a natural language input that specifies the manner in which a task corresponding to an intent is to be performed. For example, a parameter may specify a location (e.g., an address or place of interest), a time, a date, a contact, a type, text (e.g., to be inserted into an email or message), an amount (e.g., a distance, money), and in some cases the name of a software application to perform the task. The parameters may further specify other conditions for the task, examples of which are described herein.

[0275] For example, one or more detectors may be used to identify parameters. Each detector may be configured to parse natural language user input (e.g., a textual representation of the natural language user input) and identify one or more corresponding data types. For example, a first detector may be configured to identify user contacts, while a second detector may be configured to identify addresses. Other detectors may identify data types including, but not limited to, phone numbers, names, persons of interest, places of interest, URLs, times, flight numbers, package tracking numbers, and dates.

[0276] In some examples, words of the custom vocabulary may be identified in natural language user input as, for example, parameters. In one example, one or more detectors may be configured to recognize user-specific terms. User-specific terms may include any terms associated with and / or specified by a user of the electronic device, such as a contact name, address, phone number, etc. As another example, one or more detectors may be configured to recognize custom vocabularies for one or more applications, respectively. The custom vocabulary of an application may include the name of the application (e.g., Uber, Lyft, Instagram, Flickr, WeChat, WhatsApp, LINE, Viber) and / or may include other terms uniquely associated with the application (e.g., UberX, DM, Lyftline, ZipCar). In some examples, the custom vocabulary may be included in the vocabulary index 744 ( Figure 7B )middle.

[0277] In some examples, one or more applications may register with an application registration service. The service may be hosted or otherwise accessible by server 108 and / or user device 104. Registering in this manner may include specifying one or more custom vocabulary terms associated with the application and optionally one or more language models for the custom vocabulary terms. For example, a language model may provide one or more pronunciations for each custom vocabulary term. The language model provided in this manner may thereafter be used to help identify the use of such custom terms during analysis of natural language user input.

[0278] In some examples, one or more parameters can be inferred from the natural language user input. For example, in the example of "get me a ride to the stadium", one parameter associated with the intent can be inferred to be the user's current location. In another example, "give John back the money for the meal", one parameter associated with the intent can be inferred to be the amount of money.

[0279] In some examples, an intent is identified based on natural language user input, and then parameters associated with the intent are identified. Additionally, in some examples, parameters not associated with the intent are not identified. For example, an intent corresponding to directions (e.g., a driving route) may be associated with parameters specifying one or more locations (e.g., a starting point and / or a destination) and / or a mode of transportation. For example, consider the example of "Give me real-time driving directions to 1200 Main Street."

[0280] In this case, the identified intent corresponds to a task to provide a route, "travel" is a parameter that specifies a mode of transportation, and "1200 Main Street" is a parameter that specifies a location. The "real-time" portion of the user input is not a parameter associated with the identified intent. Therefore, while "real-time" can be a valid parameter for some intents, "real-time" will not be recognized as a parameter during operation.

[0281] In other examples, one or more parameters may be identified first, and the intent may be identified based on the identified one or more parameters. In other examples, the intent and the parameters associated with the intent may be identified simultaneously.

[0282] In some examples, the intent and parameters of the natural language user input are determined by a user device (such as Figure 1 In other examples, the user device provides the natural language user input (or a representation thereof) to the server, such as Figure 1 The server 108 of the present invention is used to identify (e.g., determine) the intent and parameters of the natural language user input, as described. Thereafter, the server provides (e.g., transmits) the identified intent and parameters to the user device.

[0283] Optionally, once the intent and any parameters have been identified, the user device confirms the identified intent and / or parameters to the user of the user device, and in some cases, confirms the inferred parameters to the user of the user device. Confirming in this manner may include prompting the user to confirm the identified intent and all identified parameters associated with the intent in response to a natural language query. For example, in response to a user input "Get me a ride to the airport", the user device may provide a natural language query "Do you want a ride to the airport from your current location?" The natural language query provided by the user device may be provided to the user as text using a touch-sensitive display of the user device, and / or may be provided to the user as speech using an audio output component of the user device (e.g., speaker 211 of FIG. 2 ). The user may respond to the natural language query, for example by providing natural language user input to the user device.

[0284] Optionally, the user device confirms each parameter. In some examples, this may include prompting the user to confirm one or more parameters. For example, in response to a user input "Let me take a ride to the station", the user device may provide a natural language query "Do you mean Penn Station?" As another example, in response to a user input "Pay John $5", the user device may provide a natural language query "Do you mean John Smith?" The natural language query provided by the user device may be provided to the user as text using a touch-sensitive display of the user device, and / or may be provided to the user as speech using an audio output component of the user device (e.g., speaker 211 of FIG. 2 ). The user may respond to the natural language query, for example, by providing a natural language user input to the user device.

[0285] In some examples, one or more parameters are contextual. Therefore, the user device may determine (e.g., parse) one or more parameters based on context information. The context information may be context information of the user device (or any data stored therein) and / or context information of the user of the user device. For example, a natural language user input may be stated as "Send a car to pick me up at my house." Because "my house" is a context parameter and no actual location is specified, the user device may determine the location of the user device and identify the determined location as a parameter (i.e., instead of "my house"). As another example, a natural language user input may be stated as "Call him back." Because "he" is a context parameter and no specific contact is specified, the user device may determine the contact referred to by "he" and pass the contact as a parameter (i.e., instead of "he").

[0286] In some examples, the identified intents and parameters are implemented as intent objects. When implemented in this way, each intent object is an object (e.g., a data structure, a programming object) and corresponds to a corresponding intent. Each intent object may include one or more fields (e.g., instance variables) that correspond to one or more parameters, respectively. For example, an intent object corresponding to a ride booking intent may be generated (e.g., instantiated) like the following pseudo code:

[0287]

[0288]

[0289] As one of ordinary skill will appreciate, the pseudo-code above is exemplary and the Intent object may be implemented in other ways.

[0290] By implementing intents as intent objects, intents can be language-independent. As described, intents can be derived from natural language user input. Thus, the same intent can be derived from natural language input provided in any number of spoken languages. For example, natural language user input in English "Send an Uber to 1200Park Avenue" and natural language user input in German "Senden Sie eine Uber 1200Park Avenue" will each result in the same intent being recognized (and resulting in the same Intent object).

[0291] At block 815, a software application associated with the intent may be identified (eg, selected). Typically, this may include identifying one or more software applications configured to perform a task corresponding to the intent.

[0292] In some examples, identifying a software application may include determining one or more domains corresponding to an intent and identifying an application corresponding to the one or more domains. In some examples, one or more machine learning mechanisms (e.g., neural networks) are used to identify one or more candidate domains. For example, a relevance score is assigned to each candidate domain, and the candidate domains are ranked using the relevance score. The relevance score may be determined by providing the text of the natural language user input and / or the context of the natural language user input to the neural network. The top one or more candidate domains may be determined as one or more domains corresponding to the intent. In some cases, the relevance scores of two or more top candidate domains may be within a threshold. The user device may use a knowledge ontology (such as knowledge ontology 760) to distinguish between these candidate domains.

[0293] As described, one or more software applications may register with an application registration service. Registering in this manner may include specifying which domains (e.g., a ride reservation domain, an airline travel domain, a navigation domain) correspond to the software applications. An application corresponding to a domain may support every intent specified by the domain, or may only support some of the intents specified by the domain. In some examples, applications may register with individual intents, and identifying an application may include identifying an application corresponding to the identified intent.

[0294] In some examples, applications are identified based on identified parameters. For example, based on user input "Send a black car to take me to the airport," an intent to book a car using a ride booking service and a parameter specifying a car type (i.e., a "black car") may be identified. Although some available applications may be configured to book cars generally, only those applications that are configured to book "black cars" may be identified. As another example, based on user input "Send a message to Sam saying 'Hi!'" although several applications may be configured to send the message, only those applications that have contact information for the contact Sam may be identified.

[0295] In some examples, only applications installed on the user device and / or applications accessible to the user device are identified. For example, although several available applications may be configured to perform a task, only applications accessible to the user device are identified. Accessible applications include applications that reside and / or are installed on the user device, and also include applications that can be accessed remotely by the user device, such as applications on one or more other devices.

[0296] Therefore, in at least some embodiments, the identified application is an application that is configured to perform tasks according to the identified parameters and that the user device can access. In some examples, although multiple applications can meet these criteria, the user device may wish to identify fewer applications or a single application. Therefore, the application can be further identified based on the user device's previous use of the application. In some examples, for a given intent, the application that was recently used to perform the task corresponding to the intent is identified. Consider the user input "Call Rob", where the intent (i.e., make a call) and the parameter (i.e., "Rob") can be identified. In this example, the application that was last used to make a call is identified. In other examples, the application that is most often used to perform the task corresponding to the intent is identified. For the same example, the application that is most often used to make calls is identified. In some examples, the application is further selected according to one or more parameters. For example, the application that was recently used to make a call to the contact Rob is identified, or the application that is most often used to make a call to the contact Rob is identified. In some examples, a default application can be specified for one or more specific tasks and / or parameters, for example, by a user or a digital assistant. For example, a user may specify that a first application is to be used when calling a first contact, and a second application is to be used when calling a second contact.

[0297] As described, the natural language user input may include custom vocabulary that may be identified as one or more parameters. In some examples, such custom vocabulary includes application names, and accordingly the application may be identified based on the presence of the custom vocabulary in the input. For example, the natural language user input may, for example, be stated as "Call Rob using Skype." In response, the software application Skype may be the identified software application. In another example, the natural language input may be stated as "Play The Beatles on Spotify," and in response, the software application Spotify may be the identified software application.

[0298] A custom vocabulary may also include terms that are uniquely associated with an application. Thus, such terms may be identified as parameters and, optionally, used to identify an application. In the example of "Send me an UberX," "UberX" is a term of the custom vocabulary of the software application Uber, and thus Uber is identified as a software application. In the example of "Tweet I hope the Sharks win the trophy," "Tweet" is a term of the custom vocabulary of the software application Twitter, and thus Twitter is identified as a software application.

[0299] A custom vocabulary may also include terms that are uniquely associated with a user. Thus, such terms may be identified as parameters and optionally used to identify an application. In the example of "call boss," "boss" is a user-specific term of the custom vocabulary and may be identified as a parameter, for example, to identify a contact in the user's contact list.

[0300] In some examples, the user device may receive a custom vocabulary from another device. Thus, the user device may be configured to recognize additional vocabulary that is not known to the device. In some examples, the user device may receive the custom vocabulary periodically and / or in response to establishing a connection with another device. In other examples, in response to determining that the user has provided an unrecognized term, the user device may retrieve the custom vocabulary from another device.

[0301] In some examples, the application configured to perform the task according to the identified parameters is not accessible to the user device. Therefore, the user device may access (e.g., download and / or install) the application configured to perform the task according to the identified parameters. In some examples, the user device may identify multiple software applications and provide a list of software applications to the user. The user may select one or more applications, and the user device may access the one or more selected applications.

[0302] At block 820, the intent and parameters are provided to the identified software application. In some examples, the intent and parameters are provided to the software application as an intent object.

[0303] In some examples, intents and parameters can be selectively provided to a software application based on the state of a user device. As an example, intents and parameters can be selectively provided to a user device based on whether the user device is in a locked state. In some examples, an application can be allowed to receive specific intents and parameters when the device is in a locked state. In other examples, an application can be allowed to receive specific intents and parameters when the user device is not in a locked state. For example, during a registration process using an application registration service, a software application can specify whether the application can receive specific intents given a specific state of a user device.

[0304] At block 825, the user device may receive one or more responses from the software application. In some examples, the user device receives a response for each parameter provided to the software application. For example, each response may indicate whether the parameter is valid or whether additional user input is required. If the response indicates that the parameter is valid, no further action is taken on the parameter.

[0305] If the response provided by the software application does not indicate that the parameter is valid, the response may indicate that clarification of the parameter is needed. For example, the parameter may be incorrect (i.e., invalid), and the software application may request additional input from the user. As an example, consider a user input of "Send a blue car to 1200 Main Street." Although the user has requested a blue car, the ride booking application (e.g., Uber, Lyft) may not allow the selection of a blue car (e.g., blue car may not be a supported parameter, or the application may determine that no blue cars are currently available). Therefore, if the parameter is inappropriate (e.g., the user has specified an invalid type of car), the application may request that an appropriate (e.g., valid) value for the parameter (e.g., the type of car) be provided. Reference Fig. 10AFor example, based on a response from a software application indicating that the parameter is not suitable, the user device may provide a natural language query 1002 that prompts the user to select a valid parameter. In this example, the user device provides a natural language query that requires the user to select a valid type of car. The natural language query 1002 may be provided to the user as text using a touch-sensitive display of the user device, and / or may be provided to the user as voice using an audio output component of the user device. As shown in the figure, in some examples, the user device may provide (e.g., display) one or more candidate parameters 1004 to the user for selection. In this example, the candidate parameters 1004 include "budget", "black car", "SUV" and "shared". The candidate parameters 1004 provided in this manner may be provided by the software application in some examples. The user device may select one of the candidate parameters by providing a touch input and / or by providing a natural language user input to the user device, and in response, the user device may provide the selected candidate parameter to the software application.

[0306] If the response provided by the software application does not indicate that the parameters are valid, the response may indicate that the parameters need to be disambiguated. For example, in the example where the user inputs "Call Tom", parameters for the contact Tom may be provided to a software application configured to make a phone call. If, in the process of determining contact information (e.g., phone number) for the contact Tom, the software application determines that there are multiple contacts with the name "Tom", the software application may request an indication of which "Tom" is being referred to.

[0307] The software application may request the user device to disambiguate the parameter based on the user input. As part of the request, the software application optionally includes in a response a disambiguation list having a plurality of candidate parameters. If the software application requests disambiguation based on the user input, the user device may provide a natural language query requiring the user to select a candidate parameter. In addition, the candidate parameters of the disambiguation list may be displayed to allow the user to select. The user may select one of the candidate parameters, for example, by providing a touch input and / or by providing a natural language user input, and the user device may provide the selected candidate parameter to the software application.

[0308] The software application may request the user device to disambiguate the parameter without user input. Thus, the user device may automatically disambiguate the parameter based on the context of the user device, the software application, and / or the parameter. For example, the user device may select a contact "Tom" that the user recently contacted.

[0309] In some cases, the software application may not have access to information needed to resolve parameters and / or disambiguate parameters. The software application may not have access to a list of contacts and therefore cannot determine whether there is an entry (let alone multiple entries) for a name. Therefore, the software application may request the user device to disambiguate any parameters that the software application cannot resolve. Additionally or alternatively, the software application may request the user device to provide candidate parameters to the software application so that the software application can perform disambiguation. Referring to the previous example, the software application may request the device to disambiguate "Tom", or provide a list of all "Tom" contacts so that the software application can disambiguate "Tom" itself.

[0310] Performing a task may require a user to specify one or more parameters of a particular type. In some examples, however, natural language user input may omit one or more required parameters. Thus, the software application may optionally provide one or more responses indicating which (if any) required parameters were not specified. Consider the user input "Send a car to 1200 Main Street." Although a car is typically requested via user input, a ride booking application identified based on the user input may require a particular type of car to be selected. Thus, in the event that a parameter is missing (e.g., the user did not specify a type of car), the application may request an identified value for the parameter (e.g., the type of car). Again referring to Fig. 10A , the user device may provide a natural language query 1002 prompting the user to specify valid parameters. Thereafter, the user may select a candidate parameter, for example, from a list of candidate parameters, and may provide the selected parameter to the software application as described.

[0311] Once the software application has indicated that each parameter is valid and that no additional information is needed, the user device may confirm the intent to the software application at block 830. Specifically, the user device may request notification that the software application can successfully perform the task corresponding to the intent given the intent and the parameters associated with the intent.

[0312] Once the software application provides a notification indicating that the software application can perform a task, the user device optionally confirms the intent to the user. For example, the user device may provide a natural language query "I found an uberX at your location. Should I hail this car?" The user may confirm or deny the intent through touch input or natural language input. In some examples, the notification provided by the software application may include information to be provided to the user. Such information may, for example, allow the user to make a more informed decision when prompted for confirmation. For example, the user device may provide a natural language query "I found an uberX that will arrive at your location in 9 minutes. Should I hail this car?"

[0313] Thereafter, the user device causes (eg, instructs) the software application to perform a task corresponding to the intention according to the parameters.

[0314] In other examples, once the software application provides a notification indicating that the software application can perform a task, the user device causes the software application to perform the task without user confirmation. For example, a user may provide natural language user input "Send me a Lyft to the airport," and once the recognized application Lyft indicates that it can reserve a car for the user, the software application automatically performs the task without user confirmation. In some examples, whether a task is performed without user confirmation is based on the type of user device. A first type of device, such as a mobile phone, may require user confirmation of one or more tasks, and a second type of device, such as a smartwatch, may perform one or more tasks without first receiving user confirmation.

[0315] At block 835 , the user device receives a result response from the software application indicating whether the software application successfully performed the task.

[0316] The result response indicating the failure to perform the task may further indicate one or more reasons for the failure. In some examples, the user device may provide an output indicating one or more reasons for the failure to the user, such as a natural language output or a text output.

[0317] The result response indicating successful performance of the task may include one or more response items. Each response item may be a result determined (e.g., received, generated) by the software application when performing the task. For example, the response items corresponding to booking a car with a ride booking application may include the type of car, license plate number, driver name, arrival time, current location of the car, pick-up location, destination, estimated travel time, estimated cost and estimated travel route, service type (e.g., Uberpool vs. uberX). As another example, the response items corresponding to initiating a workout session using a fitness application may include confirmation that the session has been initiated, workout duration, activity type, and one or more goals.

[0318] In some examples, one or more response items may be provided to the user. Fig. 10B , for example, one or more response items may be provided to the user as natural language output 1012, as text input, and / or audio output. The response items may also be provided visually. For example, a map 1014 of the estimated travel route may be provided to the user. It will be appreciated that the response items may be provided to the user in any desired manner.

[0319] In some examples, the software application may specify the manner in which one or more response items are provided to the user. That is, the software application may determine the manner in which the response items are displayed and / or spoken to the user, and the digital assistant may provide each response item accordingly.

[0320] In some examples, the software application may specify how to provide response items using the UI extension of the digital assistant. For example, the user device may provide a set of view controller parameters (e.g., fields that may be provided to the view controller for display) to the software application, and in response, the software application may provide a set of view controller parameter values. The set of view controller parameter values ​​may indicate which response items to display in various fields of the view controller and / or the manner in which the response items are displayed in each field. In this way, the digital assistant may control (e.g., limit) the manner in which the software application displays the response items.

[0321] In other examples, the digital assistant can determine how to provide a response item. In other examples, the software application is called so that the user can interact directly with the software application. In some examples, calling the application in this way can terminate the session with the digital assistant.

[0322] In some examples, once the task has been successfully performed, the user device (e.g., a digital assistant of the user device) may provide one or more notifications to the user indicating the status of the operation associated with the task. As an example, once a car has been reserved in response to user input, the user device may provide the location, estimated time of arrival, and / or route of the reserved car. The notification may further indicate whether an error related to the operation has occurred. For example, if the reserved car is no longer available, the user may be notified. Notifications provided in this manner may be provided periodically and / or in real time in some examples.

[0323] In some examples, once the task has been successfully performed, the user device can adjust the operation associated with the task. Adjusting the operation associated with the task can include adjusting (e.g., removing, adding, and / or modifying) parameters associated with the intent, and providing the intent to the software application with the adjusted parameters. For example, the software application can reserve a car and indicate to the user that the car has been successfully reserved. Thereafter, the car reservation can be adjusted by specifying a new destination for the ride. In other examples, adjusting the operation includes providing a different intent to the software application. For example, the intent can be an intent of a different domain.

[0324] In some examples, the operation is adjusted in response to a selection of an affordance displayed by the device. As an example, once a task has been successfully performed, the device may display one or more affordances that the user may select to confirm successful performance of the task and / or indicate how the user wishes to adjust the operation associated with the task. In some examples, affordances may be displayed based on context. As an example, an affordance for a car reservation may allow modification of a destination address or cancellation of a car reservation ( Fig.13A ). As another example, an affordance for making a dinner reservation may allow modification of the reservation time or number of people. As yet another example, an affordance for a travel route may allow modification of the destination address or mode of transportation. As yet another example, an affordance for electronic payment may allow cancellation of payment or modification of the total payment amount.

[0325] In some examples, the software application may fail to perform a task. In response, the device may display one or more affordances that the user may select to confirm the failure of the task and / or request the performance of the task. As an example, selection of an affordance may cause the user device to perform the same task that previously failed. This may include providing the same intent to the software application. The intent may have the same or different parameters and / or parameter values. As another example, selection of an affordance may cause the device to perform a different task, such as a task associated with a different domain.

[0326] As an example, during operation, a software application may not be able to respond to a user request to make a payment ( Fig. 13B ). In response, the device may display a first enable indication, and in response to selection of the first enable indication, the user device may provide an intent to the software application. The intent may be the same as an intent previously provided to the software application (i.e., an intent to make a payment). The user device may further display a second enable indication, and in response to selection of the second enable indication, the user device may provide the intent to a different software application. The user device may further display a third enable indication, and in response to selection of the third enable indication, the user device may provide an intent to send a message to another software application. For example, the user may wish to communicate that a payment failed.

[0327] In some examples, for example, before providing the intent and parameters to the software application, the permission of the software application is verified. A user device, for example, may determine whether the software application is permitted to access data associated with a particular intent. This determination may be made based on the permissions configured on the user device. In the "Send a black car to my location" example, "location" may be a context parameter that requires context information (e.g., location data) of the user device. Therefore, before resolving the location of the user device and providing the location as a parameter to the software application, the user device may first determine whether the software application is permitted to access the information. If the software application is permitted to access the data, the operation proceeds as described. If the software application is not permitted to access the data, the intent and parameters are not provided to the software application, and the task is not performed. In some cases, the permissions of the software application may be exchanged between devices. For example, a first device including a software application may provide a permission for the software application to a second device of a user including the same software application.

[0328] In some examples, the natural language input may include multiple task requests. Thus, based on the natural language input, multiple intents and / or multiple applications may be identified. Parameters associated with each intent are also optionally identified. For example, the natural language input "Get me a ride to the airport and tell me my flight status" may include an intent to book a car and an intent to retrieve the user's flight status. In some examples, the tasks corresponding to each intent may be performed sequentially or simultaneously.

[0329] In some examples, the natural language input may include multiple related task requests. For example, in some examples, a requested task of the natural language user input may depend on the completion of another requested task of the natural language input. In the example of "email me directions to the airport", two tasks are requested: the first task is to provide directions, and the second task is to send the email. The parameter "me" specifies a specific contact and is a parameter for the intent to send the email, while the parameter "airport" is a parameter that specifies the destination for the intent to provide directions. Because sending directions by email requires providing the directions first, the second task (sending the email) depends on the first task (providing the directions). Therefore, the task of retrieving the directions is performed first.

[0330] In some examples, intents can be provided between applications. For example, an application can provide an intent object to another application to cause the application to perform a task. In this example, two intents (provide directions and send an email) can be provided to a map application to provide a requested route. For example, the second intent, or the intent to send an email, can be provided as a parameter to the map application. Based on the first intent, or the intent to provide directions, the map application can provide the requested route. The map application can then provide the second intent to the email application, for example as an intent object that includes the route as a parameter. In response, the email application can email the directions as requested.

[0331] As another example, a user may provide user input "Get me a ride to a Sharks game" while viewing a sports application (e.g., an ESPN application). In response, the sports application, which has information about the game, may pass an intent (e.g., reserve a car) and parameters (e.g., the address of the game) to a ride booking application. In some examples, the intent and parameters may be provided as an intent object.

[0332] As yet another example, a user may provide user input "Pay my brother $5.00" while using a ride booking application. In response, the ride booking application may pass the intent (e.g., pay) and the parameter (5.00) to a payment application (e.g., PayPal, Venmo, an online payment service). As described, in some examples, the intent and parameters may be provided as an intent object.

[0333] In some examples, intents may be provided between devices. As described, different types of devices may be configured to perform different corresponding tasks. In some cases, a natural language user input may be provided to a first device that is not configured to perform a task corresponding to the intent of the natural language user input. The first device may provide the intent to a second device so that a task associated with the intent is performed. As an example, a first device (e.g., a mobile phone) may be configured to book a car, and a second device (e.g., a smartwatch) may be configured to provide the status of the booked car. For example, if a user requests a car reservation using a second device (e.g., "Send an Uber to take me to the airport"), the second device may derive the intent from the user input and determine that the second device cannot perform the task associated with the derived intent. Thereafter, the second device may provide the derived intent to the first device so that the task can be performed by the first device.

[0334] As described, the device may provide natural language output to the user during operation. For example, a natural language query may be provided to the user to request confirmation of parameters. In some examples, natural language output may be provided to the user according to a task flow associated with one or more software applications. For example, the task flow may specify the manner and type of natural language output provided. For example, a user may provide natural language input (e.g., "Send a Lyft to take me home"). After receiving the intent and any identified parameters, the software application may indicate that one or more parameters require clarification and / or additional parameters (e.g., the user's location, the address of "home", the type of vehicle, etc.) are required to perform the task associated with the intent. Therefore, the device may provide one or more natural language queries specified by the task flow to the user so that all required parameters can be provided to the software application.

[0335] In some examples, one or more task flows are stored on the device. Thus, the device can provide natural language output (e.g., natural language queries) to the user without having to retrieve a separate query for the task flow from an external device during operation. Storing task flows in this manner can improve operational efficiency. In some examples, the task flow is retrieved and / or stored on the user device when the corresponding software application is stored on the device.

[0336] Fig. 9 A flow chart of a process for operating a digital assistant according to some embodiments is shown. Process 900 can be used, for example, to implement Figure 8 At least a portion of process 800 includes but is not limited to Figure 8 815 and / or box 820. For example, process 900 is performed using one or more electronic devices (e.g., devices 104, 108, 200, 400, or 600) that implement a digital assistant. In some examples, process 900 is performed using a client-server system (e.g., system 100), and the boxes of process 900 can be divided between a server (e.g., DA server 106) and a client device in any manner. Therefore, although portions of process 900 are described herein as being performed by specific devices of a client-server system, it should be understood that process 900 is not limited to this. In other examples, process 900 is performed only using a client device (e.g., user device 104). In process 900, some boxes are optionally combined, the order of some boxes is optionally changed, and some boxes are optionally omitted. In some examples, additional steps may be performed in conjunction with process 900.

[0337] At block 905, natural language user input is received by a user device such as Figure 1As described, the natural language input may include a request for the user device and / or another device to perform a task, and may also specify one or more parameters for the requested task.

[0338] At box 910, an intent and optionally one or more parameters associated with the intent are identified. The intent and parameters can be derived from natural language user input. As previously described, an intent can correspond to any type of task performed by a user device, and specifically, can correspond to a task performed by one or more applications of the user device. The parameters associated with the intent can identify a portion of the natural language input that specifies the manner in which the task corresponding to the intent is to be performed. In the example "Get me a ride to the airport", the intent corresponds to the task of booking a car, and "airport" is a parameter that specifies the destination. Given that the user's intent is to book a car, the user's location can also be a parameter (e.g., an inferred parameter).

[0339] At block 915, a determination is made as to whether a task corresponding to the intent can be completed. In some examples, the determination may include determining whether an application configured to perform the task according to the parameters is accessible to the user device. In this example, it is determined whether an application configured to book a car at the location of the user device is accessible to the user device. As described, accessible applications are applications stored locally on the user device and applications that are remotely accessible to the user device.

[0340] Based on determining that the task corresponding to the intent can be completed, at block 920, the intent and parameters are provided to a software application. For example, if it is determined at block 915 that the user device has access to a software application that is configured to perform the requested task according to any of the identified parameters, then the intent and parameters are provided to the application to perform the task. In this example, this includes determining that a software application configured to book a car at the location of the user device is accessible to the user device. For example, the ride booking application Lyft may be installed on the user device and may be used to book a car according to the examples described herein.

[0341] Based on determining that the task corresponding to the intent cannot be completed, at block 925, a list of one or more software applications is provided. For example, the list of one or more software applications may include one or more software applications that are configured to perform the task associated with the intent according to any identified parameters. In some examples, for example, the one or more software applications of the list may be identified based on one or more domains associated with the intent (recalling that applications may be registered with one or more domains). Fig. 10COnce the list of software applications (e.g., ride booking applications) has been determined, the user device can provide the list to the user. As shown, providing the list can include providing a natural language input 1022 requesting the user to select an application from a list of one or more software applications. In some examples, the list is generated by the user device. In other examples, as described, the list of software applications is generated by a server and provided to the user device, which in turn can provide the list to the user.

[0342] At block 930, the user device receives user input indicating a selection of one or more software applications from the list of one or more software applications. The user input may be a touch input on a touch-sensitive display of the user device and / or may be a natural language user input.

[0343] At block 935, the intent and parameters are provided to the software application selected by the user. In some examples, providing the intent and parameters includes downloading and / or installing the software application so that the user device can access the software application locally. In other examples, this includes remotely accessing the selected application.

[0344] As described, in response to the intent and parameters, the software application may provide one or more responses. Once the parameters are verified, the user device may confirm the intent to the software application and cause the software application to perform the task. Thereafter, the user device may receive the resulting response and optionally provide one or more response items of the resulting response to the user.

[0345] Reference is made herein to providing natural language output and / or natural language queries to a user of a user device. In some examples, the manner in which the natural language output and queries are provided to the user may depend on the type or state of the user device. For example, if the user device is a mobile phone, the user device may provide queries using both text and audio. On the other hand, if the user device is a speaker, the user device may provide queries using only audio. As another example, if the user device is a mobile phone that is not paired with a headset, the user device may provide queries using text and / or relatively short natural language queries. If the user device is paired with a headset, the user device may provide queries using only relatively long natural language queries.

[0346] FIG. 10D to FIG. 10E An exemplary data flow for a digital assistant system according to some embodiments is shown. FIG. 10D to FIG. 10E The data flow may use one or more of processes 800 , 900 in some examples. Fig. 10D An exemplary data flow of a digital assistant system 1030 according to some embodiments is shown. Specifically, Fig. 10DThe data flow for the application registration process and the data flow for executing tasks are shown. Data flows 1031-1041 may be associated with the application registration process (in Fig. 10D ), and data flows 1043-1051 may be associated with the execution of tasks (in Fig. 10D denoted as a solid line).

[0347] Typically, the data flow associated with the application registration process involves registering the application with an application registration service (e.g., an authentication service), through which the application and its corresponding custom vocabulary can be accessed and / or used by a digital assistant to perform tasks.

[0348] In operation, at data flow 1031, an application is submitted to an application review module 1032. A language model corresponding to the application and the intent of the application may also be submitted. The language model may include a custom vocabulary for the application. Next, at data flow 1033, the application review module 1032 may provide the application, the custom vocabulary, and / or the intent of the application to a verification service 1034. The verification service 1034 may determine whether to verify the application, for example, based on whether the application is operable with a digital assistant. This may include, for example, ensuring that any intent of the application corresponds to one or more domains of the application. For example, the verification service may reject an instant messaging application associated with an intent to book a car because the domain and intent do not match. The verification service 1034 at data flow 1035 may provide a verification response indicating whether the application is valid.

[0349] If the verification service 1034 indicates that the application is valid, the application review module 1032 provides the application (as verified) to the application store 1036. In general, as indicated by data flow 1039, the application can be downloaded and / or accessed by the user device 1040 via the DA server 1038 in the application store 1036. In some examples, the user device 1040 can be Figure 1 The user device 104, and the DA server 1038 may be Figure 1 DA server 106. This may, for example, cause a list of applications (e.g., info.plist) of user device 1040 to be updated and / or synchronized with DA server 1038. At data flow 1041, the authentication service may provide the DA server 1038 with a custom vocabulary (e.g., runtime vocabulary) of the application to assist in parsing natural language input, as described.

[0350] Typically, the data flow associated with the execution of a task involves providing an intent and, optionally, one or more parameters to an application program to perform the task corresponding to the intent.

[0351] In operation, at data stream 1043, the user device 1040 may provide natural language input to the DA server 1038. In some examples, the natural language input may be provided by a digital assistant 1042 of the user device 1040. Based on the natural language input, the DA server 1038 may identify one or more tasks requested in the natural language input and one or more parameters associated with the intent. In addition, the DA server 1038 may identify an application for performing tasks with the intent. In some examples, the name (or other form of identifier) ​​of the identified application 1044 may be a parameter of the intent. At data stream 1045, the DA server 1038 thereafter provides the intent, parameters, and identification of the identified application 1044 to the user device 1040 (e.g., the digital assistant 1042 of the user device 1040). In some examples, the intent and parameters may be provided to the user device 1040 as an intent object.

[0352] In response, the digital assistant determines whether the identified application 1044 is permitted to access information associated with the identified parameter. For example, if the parameter is the location of the user device 1040, the digital assistant queries the data permission 1046 to determine whether the application 1044 is permitted to access the location data.

[0353] In the case where the application is granted access to the data of each parameter, the user device (e.g., the digital assistant 1042 of the user device) provides the intent to the application. As shown, the application may reside on the user device 1040. In other examples, the application may reside on one or more other devices, and the intent may be transmitted to the application via one or more networks. As described, if the application determines that one or more parameters are missing, incorrect, and / or ambiguous, the application 1044 may thereafter request input from the user of the user device 1040. In some examples, the query for user input may be provided as a natural language query generated by the DA server 1038. Therefore, at data stream 1051, the user device 1040 may request and subsequently receive one or more natural language queries. Once all parameters have been parsed, the application 1044 may perform the task corresponding to the intent and provide a result response indicating whether the task was successfully executed.

[0354] Fig. 10DOne or more data flows in the DA server 1038 are implemented (e.g., generated) using, for example, one or more electronic devices (e.g., devices 104, 108, 200, 400, or 600) that implement a digital assistant. Specifically, the data flows provided between the DA server 1038 and the digital assistant 1042 of the user device 1040 are shown as relying on a client-server architecture. In other examples, the DA server 1038 may be implemented as a process and / or service on the user device 1040. Thus, in some examples, the data flows exchanged between the DA server 1038 and the digital assistant 1042 may be exchanged only on the user device 1040.

[0355] Fig. 10E An exemplary data flow of a digital assistant system 1060 according to some embodiments is shown. Specifically, Fig. 10E The application registration process is shown and can be used to implement Fig. 10D An exemplary data flow of the application registration process is shown. Fig. 10E The several elements correspond to Fig. 10D Components of the present invention are provided with the same reference numerals. For the sake of brevity, the explanation of their functions and operations will not be repeated.

[0356] At data flow 1065, the verified vocabulary is provided from the verification service 1034 to the Global Application Vocabulary Store 1060. In general, the Global Application Vocabulary Store can store language models and / or vocabulary for any number and / or version of software applications. At data flows 1061 and 1063, the speech training module 1062 and the natural language training module 1064 are trained to recognize and naturally language process the application-specific vocabulary provided with the verified application. Based on this data, the Global Application Vocabulary Store can generate and / or train one or more language models that allow the digital assistant to recognize and process utterances containing application-specific vocabulary.

[0357] In user equipment (such as Fig. 10D During operation of a user device 1040 of the user device, the runtime-specific global application vocabulary library can receive a vocabulary and / or language model for one or more applications of the user device from the global application vocabulary library 1060. The vocabulary can be specific to a user ID of a user of the user device and / or can be specific to a version of an application and / or operating system of the user device. Based on the vocabulary, for example, one or more terms of a natural language input can be identified as parameters.

[0358] Fig.11A flowchart of a process 1100 for operating a digital assistant according to some embodiments is shown. For example, process 1100 is performed using one or more electronic devices (e.g., devices 104, 108, 200, 400, or 600) that implement a digital assistant. In some examples, a client-server system (e.g., system 100) is used to perform process 1100, and the frames of process 1100 can be divided between a server (e.g., DA server 106) and a client device in any manner. In other examples, process 1100 can be divided between a server and multiple client devices (e.g., a mobile phone and a smart watch). Therefore, although part of process 1100 is described herein as being performed by a specific device of a client-server system, it should be understood that process 1100 is not limited thereto. In other examples, process 1100 is performed using only a client device (e.g., user device 104) or using multiple client devices. In process 1100, some frames are optionally combined, the order of some frames is optionally changed, and some frames are optionally omitted. In some examples, additional steps may be performed in conjunction with process 1100 .

[0359] At box 1105, the electronic device receives audio input including natural language user input.

[0360] At block 1110, the electronic device identifies an intended object in a set of intended objects. The intended object is derived from a natural language user input. In some examples, identifying the intended object in the set of intended objects includes identifying the intended object in the set of intended objects based on a type of the electronic device.

[0361] At block 1115, the electronic device identifies a software application associated with the intent object in the set of intent objects. In some examples, identifying the software application associated with the intent object in the set of intent objects includes identifying a first parameter associated with the intent object.

[0362] In some examples, an electronic device receives multiple terms from another electronic device; receiving natural language user input includes: receiving natural language user input including a term from the multiple terms; and identifying a software application associated with the intent object from the group of intent objects includes: identifying the software application based on the term from the multiple terms.

[0363] At box 1120, the electronic device provides the intent object to the software application. Providing the intent object causes the software application to perform a task associated with the intent object. In some examples, providing the intent object to the software application to cause the software application to perform a task associated with the intent object includes providing a first parameter to the software application. In some examples, providing the intent object to the software application includes, after identifying the software application, determining whether user input requesting cancellation or modification of the task is received within a period of time, and providing the intent object to the software application based on determining that user input requesting cancellation or modification of the task is not received within the period of time. In some examples, providing the intent object to the software application includes providing the intent object to the software application without requiring a user to confirm the intent object.

[0364] In some examples, the electronic device identifies multiple candidate parameters associated with the intent and provides the multiple candidate parameters to the software application.

[0365] In some examples, an electronic device receives a disambiguation request corresponding to a parameter from a software application, identifies a candidate parameter associated with the parameter from a set of candidate parameters, and provides the selected candidate parameter to the software application.

[0366] In some examples, the natural language user input is a first natural language user input. The electronic device receives a request from a software application for a second parameter associated with the intent object, provides a natural language query based on the request, receives a second natural language user input, identifies a second parameter, the second parameter being derived from the second natural language user input, and provides the second parameter to the software application.

[0367] In some examples, an electronic device receives a request for parameters associated with the intent object from a software application, identifies, using the electronic device, a natural language query based on the request, the natural language query associated with the software application and the parameters; provides the natural language query, receives fourth user input, identifies the parameters based on the fourth user input, and provides the parameters to the software application.

[0368] At block 1125, the electronic device receives a result response indicating whether the task was successfully performed. The result response may be received from a software application.

[0369] At block 1130, the electronic device provides an output indicating whether the task was performed. In some examples, the output indicates that the software application successfully performed the task. In some examples, the output indicates that the software application failed to perform the task. In some examples, providing an output indicating whether the task was performed includes receiving a set of view controller parameter values ​​from the software application and providing an output based on the set of view controller parameter values.

[0370] In some examples, after providing the output, the electronic device receives a second input, and in response to the second input, adjusts the operation associated with the task. In some examples, adjusting the operation associated with the task includes adjusting the intent object and providing the adjusted intent object to the software application. In some examples, adjusting the intent object includes adjusting a parameter of the intent object. In some examples, receiving the second input includes detecting a selection of an affordance, and adjusting the operation associated with the task includes adjusting the operation associated with the task in response to the selection of the affordance.

[0371] In some examples, after providing the output, the electronic device receives a third input, and in response to the third input, provides another intent object to the software application so that the software application performs a task associated with the other intent object. In some examples, receiving the third input includes detecting a selection of an affordance, and providing another intent object to the software application so that the software application performs a task associated with the other intent object includes: providing the other intent object to the software application in response to the selection of the affordance. In some examples, the intent object and the other intent object are the same type of intent object, and the parameters associated with the intent object and the parameters associated with the other intent object have the same value. In some examples, the intent object and the other intent object are the same intent object.

[0372] Fig.12A flowchart of a process 1200 for operating a digital assistant according to some embodiments is shown. For example, process 1200 is performed using one or more electronic devices (e.g., devices 104, 108, 200, 400, or 600) that implement a digital assistant. In some examples, process 1200 is performed using a client-server system (e.g., system 100), and the frames of process 1200 can be divided between a server (e.g., DA server 106) and a client device in any manner. In other examples, process 1200 can be divided between a server and multiple client devices (e.g., a mobile phone and a smart watch). Therefore, although part of process 1200 is described herein as being performed by a specific device of a client-server system, it should be understood that process 1200 is not limited thereto. In other examples, process 1200 is performed using only a client device (e.g., user device 104) or using multiple client devices. In process 1200, some frames are optionally combined, the order of some frames is optionally changed, and some frames are optionally omitted. In some examples, additional steps may be performed in conjunction with process 1200 .

[0373] At block 1205 , the electronic device identifies an intent object from a set of intent objects, wherein the intent object is derived from natural language user input.

[0374] At block 1210 , the electronic device identifies a software application associated with the intent object in the set of intent objects.

[0375] At block 1215, the electronic device receives a task flow associated with the software application from a second electronic device. In some examples, receiving the task flow associated with the software application includes receiving the task flow from another electronic device and storing the task flow on the electronic device.

[0376] At block 1220, the electronic device provides the intent object to the software application to cause the software application to perform a task associated with the intent object.

[0377] At box 1225, the electronic device receives a request from a software application.

[0378] At block 1230, the electronic device determines a query based on the request. The query is determined according to a task flow. In some examples, determining the query based on the request includes retrieving a query from a task flow stored on the electronic device.

[0379] At box 1235, the electronic device provides the query using the touch-sensitive display.

[0380] At block 1240, the electronic device receives a second user input indicating a query response. In some examples, the second user input is received after providing the query. In some examples, the query response indicates a parameter associated with the object of intent.

[0381] At block 1245, the electronic device provides the query response to the software application. In some examples, providing the query response to the software application includes adjusting the intent object and providing the adjusted intent object to the software application.

[0382] In some examples, the electronic device receives a result response from the software application indicating whether the task was successfully performed and provides an output indicating whether the task was performed.

[0383] According to some embodiments, Fig.14 According to various embodiments (including reference Fig.11 The functional blocks of the device are optionally implemented by hardware, software, or a combination of hardware and software that implements the principles of the various described embodiments. It should be understood by those skilled in the art that Fig.14 The functional blocks described in the present invention are optionally combined or separated into sub-blocks to implement the principles of various embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0384] like Fig.14 As shown, the electronic device 1400 includes a touch-sensitive display unit 1402 and a processing unit 1408 optionally coupled to the touch-sensitive display unit 1402. In some embodiments, the processing unit 1408 includes a receiving unit 1410, an identifying unit 1412, a providing unit 1414, and optionally, an adjusting unit 1416, a detecting unit 1418, and a determining unit 1420.

[0385] In some examples, processing unit 1408 is configured to receive audio input including natural language user input (e.g., using receiving unit 1410); identify an intent object from a set of intent objects (e.g., using recognition unit 1412), wherein the intent object is derived from the natural language user input; identify a software application associated with the intent object from the set of intent objects (e.g., using recognition unit 1412); provide the intent object to the software application to cause the software application to perform a task associated with the intent object (e.g., using providing unit 1414); receive a result response from the software application indicating whether the task was successfully executed (e.g., using receiving unit 1410); and provide an output indicating whether the task was executed (e.g., using providing unit 1414).

[0386] In some examples, identifying a software application associated with the intent object in the group of intent objects includes (e.g., using identification unit 1412) identifying a first parameter associated with the intent object, and providing the intent object to the software application so that the software application performs a task associated with the intent object includes (e.g., using provision unit 1414) providing the first parameter to the software application.

[0387] In some examples, the natural language user input is a first natural language input, and the processing unit 1408 is further configured to: receive a request for a second parameter associated with the intent object from a software application (e.g., using a receiving unit 1410); provide a natural language query based on the request (e.g., using a providing unit 1414); receive a second natural language user input (e.g., using a receiving unit 1410); and identify a second parameter (e.g., using an identifying unit 1412) that is derived from the second natural language user input; and provide the second parameter to the software application.

[0388] In some examples, the output indicates that the software application successfully performed the task.

[0389] In some examples, processing unit 1408 is further configured to, after providing the output, receive a second input (e.g., using receiving unit 1410); and in response to the second input, adjust the operation associated with the task (e.g., using adjusting unit 1416).

[0390] In some examples, adjusting operations associated with the task includes adjusting the intent object (eg, using adjusting unit 1416 ) and providing the adjusted intent object to the software application (eg, using providing unit 1414 ).

[0391] In some examples, adjusting the intent object includes adjusting (eg, using adjustment unit 1416 ) a parameter of the intent object.

[0392] In some examples, receiving the second input includes detecting selection of the enabling representation (e.g., using detection unit 1418), and adjusting the operation associated with the task includes adjusting the operation associated with the task in response to selection of the enabling representation (e.g., using adjustment unit 1416).

[0393] In some examples, the output indicates that the software application failed to perform a task.

[0394] In some examples, processing unit 1408 is further configured to, after providing the output, receive a third input (e.g., using receiving unit 1410); and in response to the third input, provide another intent object to the software application (e.g., using providing unit 1414) to cause the software application to perform a task associated with the other intent object.

[0395] In some examples, receiving a third input includes detecting a selection of the enable representation (e.g., using detection unit 1418), and providing another intention object to the software application so that the software application performs a task associated with the other intention object includes: in response to the selection of the enable representation, providing the other intention object to the software application (e.g., using provision unit 1414).

[0396] In some examples, the intent object and the other intent object are the same type of intent object, and a parameter associated with the intent object and a parameter associated with the other intent object have the same value.

[0397] In some examples, the intent object and the other intent object are the same intent object.

[0398] In some examples, providing output indicating whether a task is performed includes receiving a set of view controller parameter values ​​from a software application (e.g., using receiving unit 1410); and providing output based on the set of view controller parameter values ​​(e.g., using providing unit 1414).

[0399] In some examples, processing unit 1408 is further configured to receive multiple terms from another electronic device (e.g., using receiving unit 1410), receiving natural language user input includes (e.g., using receiving unit 1410) receiving natural language user input including a term from the multiple terms, and identifying a software application associated with the intent object in the group of intent objects includes (e.g., using identification unit 1412) identifying the software application based on the term from the multiple terms.

[0400] In some examples, identifying an intended object in a set of intended objects includes identifying the intended object in the set of intended objects based on a type of the electronic device (eg, using identification unit 1412 ).

[0401] In some examples, processing unit 1408 is further configured to identify (e.g., using identification unit 1412) multiple candidate parameters associated with the intent; and provide (e.g., using provision unit 1414) the multiple candidate parameters to the software application.

[0402] In some examples, processing unit 1408 is further configured to receive a disambiguation request corresponding to a parameter from a software application (e.g., using receiving unit 1410), identify a candidate parameter associated with the parameter from a set of candidate parameters (e.g., using identifying unit 1412), and provide the selected candidate parameter to the software application (e.g., using providing unit 1414).

[0403] In some examples, providing an intent object to a software application so that the software application performs a task associated with the intent object includes, after identifying the software application, determining (e.g., using determination unit 1420) whether user input requesting cancellation or modification of the task is received within a period of time; and based on determining that no user input requesting cancellation or modification of the task is received within a period of time, providing the intent object to the software application (e.g., using provision unit 1414).

[0404] In some examples, based on determining that no user input requesting cancellation or modification of a task has been received within a period of time, providing the intention object to the software application includes (e.g., utilizing providing unit 1414) providing the intention object to the software application without requiring user confirmation of the intention object.

[0405] In some examples, processing unit 1408 is further configured to: receive a request for parameters associated with an intent object from a software application (e.g., using receiving unit 1410); using an electronic device, identify a natural language query based on the request (e.g., using identification unit 1412), the natural language query being associated with the software application and the parameters; provide the natural language query (e.g., using providing unit 1414); receive a fourth user input (e.g., using receiving unit 1410); identify parameters based on the fourth user input (e.g., using identification unit 1412); and provide the parameters to the software application (e.g., using providing unit 1414).

[0406] The above is relative to Fig.11 The described operations are optionally performed by Figure 1 , Figure 2A , Figure 4 , FIG. 6A to FIG. 6B , Fig. 7A and Fig.14 For example, receiving operations 1105 and 1125, identifying operations 1110 and 1115, and providing operations 1120 and 1130 are optionally implemented by one or more processors 120. A person skilled in the art will clearly know how to implement the processing based on Figure 1 , Figure 2A , Figure 4 , FIG. 6A to FIG. 6B , Fig. 7A and Fig.14 The components depicted in the figure are used to implement other processes.

[0407] Those skilled in the art should understand that Fig.14 The functional blocks described in the embodiment are optionally combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For example, the processing unit 1408 may have an associated "controller" unit that is operably coupled to the processing unit 1408 to enable operation. The controller unit is Fig.14 1408, but it should be understood that it is within the grasp of a person of ordinary skill in the art who designs a device (e.g., device 1400) having a processing unit 1408. For another example, in some embodiments, one or more units such as receiving unit 1410 may be hardware units other than processing unit 1408. Therefore, the description herein optionally supports the combination, separation, and / or further definition of the functional blocks described herein.

[0408] According to some embodiments, Fig.15 According to various embodiments (including reference Fig.11 The functional blocks of the device are optionally implemented by hardware, software, or a combination of hardware and software that implements the principles of the various described embodiments. It should be understood by those skilled in the art that Fig.15 The functional blocks described in the present invention are optionally combined or separated into sub-blocks to implement the principles of various embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0409] like Fig.15 As shown, one or more electronic devices 1500 include one or more processing units 1508 and one or more touch-sensitive displays 1502. In some embodiments, the one or more processing units 1508 include a receiving unit 1510, an acquiring unit 1512, a determining unit 1514, a providing unit 1516, and optionally an executing unit 1518.

[0410] In some examples, one or more processing units 1508 are configured to receive audio input including natural language user input (e.g., using receiving unit 1510); obtain a text string from the natural language user input (e.g., using obtaining unit 1512); determine an intent object in a set of intent objects based on the text string (e.g., using determining unit 1514); determine a software application associated with the intent object in the set of intent objects (e.g., using determining unit 1514); provide the intent object to the software application so that the software application performs a task associated with the intent object (e.g., using providing unit 1516); receive a result response from the software application indicating whether the task was successfully executed (e.g., using receiving unit 1510); and provide an output indicating whether the task was executed (e.g., using providing unit 1516).

[0411] In some examples, determining an intent object in a set of intent objects based on a text string includes: (e.g., using execution unit 1518) performing natural language processing on the text string to determine the intent of the natural language user input, and (e.g., using recognition unit 1520) identifying an intent object associated with the intent.

[0412] In some examples, providing an output indicating whether the task was performed includes providing (eg, using providing unit 1516 ) a natural language output.

[0413] In some examples, determining the software application associated with the intent object in the group of intent objects includes (e.g., using determination unit 1514) determining parameters associated with the intent object, and providing the intent object to the software application so that the software application performs a task associated with the intent object includes (e.g., using provision unit 1516) providing the parameters to the software application.

[0414] In some examples, determining an intended object in a set of intended objects includes determining the intended object in the set of intended objects based on a type of the electronic device (eg, using determination unit 1514).

[0415] The above is relative to Fig.11 The described operations are optionally performed by Figure 1 , Figure 2A , Figure 4 , FIG. 6A to FIG. 6B , Fig. 7A and Fig.15 For example, receiving operations 1105 and 1125, identifying operations 1110 and 1115, and providing operations 1120 and 1130 are optionally implemented by one or more processors 120. A person skilled in the art will clearly know how to implement the processing based on Figure 1 , Figure 2A , Figure 4 , FIG. 6A to FIG. 6B , Fig. 7A and Fig.15 The components depicted in the figure are used to implement other processes.

[0416] Those skilled in the art should understand that Fig.15 The functional blocks described in the embodiment are optionally combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For example, one or more processing units 1508 may have an associated "controller" unit that is operably coupled to at least one of the one or more processing units 1508 to enable operation. The controller unit is Fig.15 1508, but it should be understood that it is within the grasp of a person of ordinary skill in the art who designs a device (e.g., device 1500) having one or more processing units 1508. For another example, in some embodiments, one or more units such as receiving unit 1510 may be hardware units other than one or more processing units 1508. Therefore, the description herein optionally supports the combination, separation, and / or further definition of the functional blocks described herein.

[0417] According to some embodiments, Fig.16 According to various embodiments (including reference Fig.12 The functional blocks of the device are optionally implemented by hardware, software, or a combination of hardware and software that implements the principles of the various described embodiments. It should be understood by those skilled in the art that Fig.16 The functional blocks described in the present invention are optionally combined or separated into sub-blocks to implement the principles of various embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0418] like Fig.16 As shown, one or more electronic devices 1600 include a touch-sensitive display unit 1602 and a processing unit 1608 optionally coupled to the touch-sensitive display unit. In some embodiments, the one or more processing units 1608 include an identification unit 1610, a receiving unit 1612, a providing unit 1614, a determining unit 1616, and optionally an adjustment unit 1618 and a retrieval unit 1620.

[0419] In some examples, processing unit 1608 is configured to identify an intent object from a set of intent objects (e.g., using recognition unit 1610), the intent object being derived from natural language user input; identify a software application associated with the intent object from the set of intent objects (e.g., using recognition unit 1610); receive a task flow associated with the software application from a second electronic device (e.g., using receiving unit 1612); provide the intent object to the software application (e.g., using providing unit 1614) so ​​that the software application performs a task associated with the intent object; receive a request from the software application; determine a query based on the request (e.g., using determination unit 1616), wherein the query is determined according to the task flow; provide the query using touch-sensitive display 1602 and (e.g., using providing unit 1614); after providing the query, receive a second user input indicating a query response (e.g., using receiving unit 1612); and provide the query response to the software application (e.g., using providing unit 1614).

[0420] In some examples, processing unit 1608 is further configured to receive a result response from the software application indicating whether the task was successfully executed (e.g., using receiving unit 1612) and provide an output indicating whether the task was executed (e.g., using providing unit 1614).

[0421] In some examples, providing the query response to the software application includes adjusting the intent object (eg, using adjustment unit 1618) and providing the adjusted intent object to the software application.

[0422] In some examples, receiving a task flow associated with a software application includes receiving the task flow from another electronic device (eg, using receiving unit 1612) and storing the task flow on the electronic device.

[0423] In some examples, determining a query based on the request includes retrieving (eg, using retrieval unit 1620) a query from a task flow stored on the electronic device.

[0424] In some examples, the query response indicates parameters associated with the intent object.

[0425] The above is relative to Fig.12 The described operations are optionally performed by Figure 1 , Figure 2A , Figure 4 , FIG. 6A to FIG. 6B , Fig. 7A and Fig.16For example, identifying operations 1205 and 1210, receiving operations 1215, 1225, and 1240, providing operations 1220, 1235, and 1245, and determining operation 1230 are optionally implemented by one or more processors 120. A person skilled in the art will clearly know how to implement the above-mentioned operations based on the above-mentioned components. Figure 1 , Figure 2A , Figure 4 , FIG. 6A to FIG. 6B , Fig. 7A and Fig.16 The components depicted in the figure are used to implement other processes.

[0426] Those skilled in the art should understand that Fig.16 The functional blocks described in the embodiment are optionally combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For example, one or more processing units 1608 may have an associated "controller" unit that is operably coupled to at least one of the one or more processing units 1608 to enable operation. The controller unit is Fig.16 1608, but it should be understood that it is within the grasp of a person of ordinary skill in the art who designs a device (e.g., device 1600) having one or more processing units 1608. For another example, in some embodiments, one or more units such as identification unit 1610 may be hardware units other than one or more processing units 1608. Therefore, the description herein optionally supports the combination, separation, and / or further definition of the functional blocks described herein.

[0427] According to some embodiments, Fig.17 According to various embodiments (including reference Fig.12 The functional blocks of the device are optionally implemented by hardware, software, or a combination of hardware and software that implements the principles of the various described embodiments. It should be understood by those skilled in the art that Fig.17 The functional blocks described in the present invention are optionally combined or separated into sub-blocks to implement the principles of various embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0428] like Fig.17 As shown, the electronic device 1700 includes a touch-sensitive display unit 1402 and one or more processing units 1708. In some embodiments, the one or more processing units 1708 include a determination unit 1710, an identification unit 1712, a providing unit 1714, a receiving unit 1716, and optionally an acquisition unit 1718 and a storage unit 1720.

[0429] In some examples, one or more processing units 1708 are configured to determine an intent object from a set of intent objects (e.g., using a determination unit 1710); identify a software application associated with the intent object from the set of intent objects (e.g., using an identification unit 1712); determine a task flow associated with the software application (e.g., using a determination unit 1710); provide the intent object to the software application (e.g., using a provision unit 1714) so ​​that the software application performs a task associated with the intent object; receive a request from the software application (e.g., using a receiving unit 1716); determine a query based on the request (e.g., using a determination unit 1710), the query being determined based on the task flow; use a touch-sensitive display and provide the query (e.g., using a provision unit 1714); after providing the query, receive a second user input indicating a query response (e.g., using a receiving unit 1716); and provide the query response to the software application (e.g., using a provision unit 1714).

[0430] In some examples, one or more processing units 1708 are further configured to receive audio input including natural language user input (e.g., using receiving unit 1716) and obtain a text string from the natural language user input (e.g., using obtaining unit 1718), and determine an intended object in a set of intended objects including determining an intended object in a set of intended objects based on the text string (e.g., using determining unit 1710).

[0431] In some examples, one or more processing units 1708 are further configured to receive a result response from the software application indicating whether the task was successfully executed (e.g., using receiving unit 1716) and provide an output indicating whether the task was executed (e.g., using providing unit 1714).

[0432] In some examples, determining a task flow associated with a software application includes determining the task flow at a first electronic device (e.g., using a determination unit 1710), providing the task flow to a second electronic device (e.g., using a provision unit 1714), and storing the task flow on the first electronic device (e.g., using a storage unit 1720).

[0433] In some examples, determining a query based on the request includes (eg, using determining unit 1710) determining a query for the first electronic device.

[0434] The above is relative to Fig.12 The described operations are optionally performed by Figure 1 , Figure 2A , Figure 4 , FIG. 6A to FIG. 6B , Fig. 7A and Fig.17 For example, identifying operations 1205 and 1210, receiving operations 1215, 1225, and 1240, providing operations 1220, 1235, and 1245, and determining operation 1230 are optionally implemented by one or more processors 120. A person skilled in the art will clearly know how to implement the above-mentioned operations based on the above-mentioned components. Figure 1 , Figure 2A , Figure 4 , FIG. 6A to FIG. 6B , Fig. 7A and Fig.17 The components depicted in the figure are used to implement other processes.

[0435] Those skilled in the art should understand that Fig.14 The functional blocks described in the embodiment are optionally combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For example, the processing unit 1708 may have an associated "controller" unit that is operably coupled to the processing unit 1708 to enable operation. The controller unit is Fig.17 1708, but it should be understood that it is within the grasp of a person of ordinary skill in the art who designs a device (e.g., device 1700) having a processing unit 1708. For another example, in some embodiments, one or more units such as the determination unit 1710 may be hardware units other than the processing unit 1708. Therefore, the description herein optionally supports the combination, separation, and / or further definition of the functional blocks described herein.

[0436] For the purpose of explanation, the foregoing description is described by reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive, nor is it intended to limit the invention to the precise form disclosed. Based on the above teachings, many modifications and variations are possible. These embodiments are selected and described in order to best explain the principles of these technologies and their practical applications. Others skilled in the art can thus best utilize these technologies and various embodiments with various modifications suitable for the specific purposes envisioned.

[0437] Although the disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. It should be understood that such changes and modifications are considered to be included within the scope of the disclosure and examples defined by the claims.

Claims

1. A method comprising: At an electronic device having one or more processors and a display: receiving audio input including natural language user input; determining user intent based on the natural language user input; Based on the determined user intent, identifying an intent object from a set of intent objects, wherein the intent object is a data structure corresponding to the determined user intent and the intent object includes one or more data fields for one or more parameters associated with the determined user intent; identifying, based on the intent object, a software application for performing a task corresponding to the determined user intent; providing the intent object to the software application to cause the software application to perform the task corresponding to the determined user intent; Based on determining that the task cannot be completed, providing a list of one or more software applications identified as performing the task; receiving an indication of a software application selected from the one or more software applications listed; receiving a response from the selected software application indicating whether the task was successfully performed; as well as An output is provided indicating whether the task was performed.

2. The method according to claim 1, wherein: Identifying a software application associated with the determined user intent includes: identifying a first parameter associated with the intent object, and Providing the intent object to the software application so that the software application performs the task corresponding to the determined user intent includes providing the first parameter to the software application.

3. The method according to any one of claims 1-2, wherein the natural language user input is a first natural language user input, and wherein the method further comprises: receiving, from the software application, a request for a second parameter associated with the intent object; providing a natural language query based on the request; receiving a second natural language user input; identifying the second parameter, wherein the second parameter is derived from the second natural language user input; as well as The second parameter is provided to the software application.

4. The method of any one of claims 1-2, wherein the output indicates that the software application successfully performed the task.

5. The method according to claim 4, further comprising: After providing the output, receiving a second input; as well as In response to the second input, an operation associated with the task is adjusted.

6. The method of claim 5, wherein adjusting operations associated with the task comprises: adjusting the intention object, wherein adjusting the intention object comprises adjusting a parameter of the intention object; as well as The adjusted intent object is provided to the software application.

7. The method of claim 5, wherein receiving a second input comprises detecting a selection of an affordance, and The adjustment of the operation associated with the task includes: In response to the selection of the affordance, the operation associated with the task is adjusted.

8. The method of any of claims 1-2, wherein the output indicates that the software application failed to perform the task.

9. The method according to claim 8, further comprising: After providing the output, receiving a third input; as well as In response to the third input, another intention object is provided to the software application to cause the software application to perform a task associated with the another intention object.

10. The method of claim 9, wherein receiving the third input comprises: Detecting the choice of affordances, and Wherein providing another intent object to the software application so that the software application performs the task associated with the another intent object comprises: In response to the selection of the affordance, the other intention object is provided to the software application.

11. The method of claim 10, wherein the intent object and the further intent object are the same type of intent object, and wherein a parameter associated with the intent object and a parameter associated with the further intent object have the same value.

12. The method of any one of claims 1-2, wherein providing an output indicating whether the task is performed comprises: receiving a set of view controller parameter values ​​from the software application; as well as Output is provided according to the set of view controller parameter values.

13. The method according to any one of claims 1 to 2, further comprising: receiving a plurality of terms from another electronic device, wherein receiving the natural language user input comprises: receiving a natural language user input including a term from a plurality of terms, and Wherein identifying a software application for performing a task corresponding to the determined user intent includes: identifying the software application based on the term in the plurality of terms.

14. The method of any one of claims 1-2, wherein identifying the intended object in the set of intended objects based on the determined user intent comprises: The intended object is identified based on the type of the electronic device.

15. The method according to any one of claims 1-2, further comprising: identifying a plurality of candidate parameters associated with the determined user intent; as well as The plurality of candidate parameters are provided to the software application.

16. The method according to any one of claims 1-2, further comprising: receiving, from the software application, a disambiguation request corresponding to a parameter; identifying, from a set of candidate parameters, the candidate parameter associated with the parameter; as well as The selected candidate parameters are provided to the software application.

17. The method of any one of claims 1-2, wherein providing the intent object to the software application to cause the software application to perform the task corresponding to the determined user intent comprises: After identifying the software application, determining whether user input requesting cancellation or modification of the task is received within a period of time; as well as Based on determining that the user input requesting cancellation or modification of the task is not received within the period of time, the intent object is provided to the software application.

18. The method of claim 17, wherein providing the intention object to the software application based on determining that the user input requesting cancellation or modification of the task is not received within the period of time comprises: The intent object is provided to the software application without requiring user confirmation of the intent object.

19. The method according to any one of claims 1-2, further comprising: receiving, from the software application, a request for parameters associated with the determined user intent; identifying, using the electronic device, a natural language query based on the request, wherein the natural language query is associated with the software application and the parameter; providing the natural language query; receiving a fourth user input; identifying the parameter based on the fourth user input; as well as The parameters are provided to the software application.

20. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the one or more electronic devices to: receiving audio input including natural language user input; determining user intent based on the natural language user input; Based on the determined user intent, identifying an intent object from a set of intent objects, wherein the intent object is a data structure corresponding to the determined user intent and the intent object includes one or more data fields for one or more parameters associated with the determined user intent; identifying, based on the intent object, a software application for performing a task corresponding to the determined user intent; providing the intent object to the software application to cause the software application to perform the task corresponding to the determined user intent; Based on determining that the task cannot be completed, providing a list of one or more software applications identified as performing the task; receiving an indication of a software application selected from the one or more software applications listed; receiving a response from the selected software application indicating whether the task was successfully performed; as well as An output is provided indicating whether the task was performed.

21. The computer-readable storage medium of claim 20, wherein: Identifying a software application associated with the determined user intent includes: identifying a first parameter associated with the intent object, and Providing the intent object to the software application so that the software application performs the task corresponding to the determined user intent includes providing the first parameter to the software application.

22. The computer-readable storage medium of any one of claims 20-21, wherein the natural language user input is a first natural language user input, and wherein the one or more programs further comprise instructions that, when executed by the one or more processors of the electronic device, cause the electronic device to perform the following operations: receiving, from the software application, a request for a second parameter associated with the intent object; providing a natural language query based on the request; receiving a second natural language user input; identifying the second parameter, wherein the second parameter is derived from the second natural language user input; as well as The second parameter is provided to the software application.

23. The computer-readable storage medium of any one of claims 20-21, wherein the output indicates that the software application successfully performed the task.

24. The computer-readable storage medium of claim 23, wherein the one or more programs further comprise instructions that, when executed by the one or more processors of the electronic device, cause the electronic device to: After providing the output, receiving a second input; and In response to the second input, an operation associated with the task is adjusted.

25. The computer-readable storage medium of claim 24, wherein adjusting operations associated with the task comprises: adjusting the intention object, wherein adjusting the intention object comprises adjusting a parameter of the intention object; as well as The adjusted intent object is provided to the software application.

26. The computer-readable storage medium of claim 24, wherein receiving the second input comprises detecting a selection of an affordance, and The adjustment of the operation associated with the task includes: In response to the selection of the affordance, the operation associated with the task is adjusted.

27. The computer-readable storage medium of any one of claims 20-21, wherein the output indicates that the software application failed to perform the task.

28. The computer-readable storage medium of claim 27, wherein the one or more programs further comprise instructions that, when executed by the one or more processors of the electronic device, cause the electronic device to: After providing the output, receiving a third input; and In response to the third input, another intention object is provided to the software application to cause the software application to perform a task associated with the another intention object.

29. The computer-readable storage medium of claim 28, wherein receiving the third input comprises: Detecting the choice of affordances, and Wherein providing another intent object to the software application so that the software application performs the task associated with the another intent object comprises: In response to the selection of the affordance, the other intention object is provided to the software application.

30. The computer-readable storage medium of claim 29, wherein the intent object and the another intent object are the same type of intent object, and wherein a parameter associated with the intent object and a parameter associated with the another intent object have the same value.

31. The computer-readable storage medium of any one of claims 20-21, wherein providing an output indicating whether the task is performed comprises: receiving a set of view controller parameter values ​​from the software application; as well as Output is provided according to the set of view controller parameter values.

32. The computer-readable storage medium of any one of claims 20-21, wherein the one or more programs further include instructions that, when executed by the one or more processors of the electronic device, cause the electronic device to perform the following operations: receiving a plurality of terms from another electronic device, Wherein receiving the natural language user input comprises: receiving a natural language user input including a term from the plurality of terms, and Wherein identifying a software application for performing a task corresponding to the determined user intent includes: identifying the software application based on the term in the plurality of terms.

33. The computer-readable storage medium of any one of claims 20-21, wherein identifying the intent object in the set of intent objects based on the determined user intent comprises: The intended object is identified based on the type of the electronic device.

34. The computer-readable storage medium of any one of claims 20-21, wherein the one or more programs further include instructions that, when executed by the one or more processors of the electronic device, cause the electronic device to perform the following operations: identifying a plurality of candidate parameters associated with the determined user intent; and The plurality of candidate parameters are provided to the software application.

35. The computer-readable storage medium of any one of claims 20-21, wherein the one or more programs further include instructions that, when executed by the one or more processors of the electronic device, cause the electronic device to perform the following operations: receiving, from the software application, a disambiguation request corresponding to a parameter; identifying the candidate parameter associated with the parameter from a set of candidate parameters; and The selected candidate parameters are provided to the software application.

36. The computer-readable storage medium of any one of claims 20-21, wherein providing the intent object to the software application to cause the software application to perform the task corresponding to the determined user intent comprises: After identifying the software application, determining whether user input requesting cancellation or modification of the task is received within a period of time; as well as Based on determining that the user input requesting cancellation or modification of the task is not received within the period of time, the intent object is provided to the software application.

37. The computer-readable storage medium of claim 36, wherein providing the intention object to the software application based on determining that the user input requesting cancellation or modification of the task is not received within the period of time comprises: The intent object is provided to the software application without requiring user confirmation of the intent object.

38. The computer-readable storage medium of any one of claims 20-21, wherein the one or more programs further include instructions that, when executed by the one or more processors of the electronic device, cause the electronic device to perform the following operations: receiving, from the software application, a request for parameters associated with the determined user intent; identifying, using the electronic device, a natural language query based on the request, wherein the natural language query is associated with the software application and the parameter; providing the natural language query; receiving a fourth user input; identifying the parameter based on the fourth user input; as well as The parameters are provided to the software application.

39. An electronic device comprising: one or more processors; Memory; monitor; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for: receiving audio input including natural language user input; determining user intent based on the natural language user input; Based on the determined user intent, identifying an intent object from a set of intent objects, wherein the intent object is a data structure corresponding to the determined user intent and the intent object includes one or more data fields for one or more parameters associated with the determined user intent; identifying, based on the intent object, a software application for performing a task corresponding to the determined user intent; providing the intent object to the software application to cause the software application to perform the task corresponding to the determined user intent; Based on determining that the task cannot be completed, providing a list of one or more software applications identified as performing the task; receiving an indication of a software application selected from the one or more software applications listed; receiving a response from the selected software application indicating whether the task was successfully performed; as well as An output is provided indicating whether the task was performed.

40. The electronic device of claim 39, wherein: Identifying a software application associated with the determined user intent includes: identifying a first parameter associated with the intent object, and Providing the intent object to the software application so that the software application performs the task corresponding to the determined user intent includes providing the first parameter to the software application.

41. The electronic device of any one of claims 39-40, wherein the natural language user input is a first natural language user input, and wherein the one or more programs further include instructions for: receiving, from the software application, a request for a second parameter associated with the intent object; providing a natural language query based on the request; receiving a second natural language user input; identifying the second parameter, wherein the second parameter is derived from the second natural language user input; as well as The second parameter is provided to the software application.

42. The electronic device of any of claims 39-40, wherein the output indicates that the software application successfully performed the task.

43. The electronic device of claim 42, wherein the one or more programs further comprise instructions for: After providing the output, receiving a second input; and In response to the second input, an operation associated with the task is adjusted.

44. The electronic device of claim 43, wherein adjusting operations associated with the task comprises: adjusting the intention object, wherein adjusting the intention object comprises adjusting a parameter of the intention object; as well as The adjusted intent object is provided to the software application.

45. The electronic device of claim 43, wherein receiving the second input comprises detecting a selection of the affordance, and The adjustment of the operation associated with the task includes: In response to the selection of the affordance, the operation associated with the task is adjusted.

46. ​​An electronic device as claimed in any of claims 39-40, wherein the output indicates that the software application has failed to perform the task.

47. The electronic device of claim 46, wherein the one or more programs further comprise instructions for: After providing the output, receiving a third input; and In response to the third input, another intention object is provided to the software application to cause the software application to perform a task associated with the another intention object.

48. The electronic device of claim 47, wherein receiving the third input comprises: Detecting the choice of affordances, and Wherein providing another intent object to the software application so that the software application performs the task associated with the another intent object comprises: In response to the selection of the affordance, the other intention object is provided to the software application.

49. The electronic device of claim 48, wherein the intent object and the another intent object are the same type of intent objects, and wherein a parameter associated with the intent object and a parameter associated with the another intent object have the same value.

50. The electronic device of any one of claims 39-40, wherein providing an output indicating whether the task is performed comprises: receiving a set of view controller parameter values ​​from the software application; as well as Output is provided according to the set of view controller parameter values.

51. The electronic device of any one of claims 39-40, wherein the one or more programs further comprise instructions for: receiving a plurality of terms from another electronic device, Wherein receiving the natural language user input comprises: receiving a natural language user input including a term from the plurality of terms, and Wherein identifying a software application for performing a task corresponding to the determined user intent includes: identifying the software application based on the term in the plurality of terms.

52. The electronic device of any one of claims 39-40, wherein identifying the intent object in the set of intent objects based on the determined user intent comprises: The intended object is identified based on the type of the electronic device.

53. The electronic device of any one of claims 39-40, wherein the one or more programs further comprise instructions for: identifying a plurality of candidate parameters associated with the determined user intent; and The plurality of candidate parameters are provided to the software application.

54. The electronic device of any one of claims 39-40, wherein the one or more programs further comprise instructions for: receiving, from the software application, a disambiguation request corresponding to a parameter; identifying the candidate parameter associated with the parameter from a set of candidate parameters; and The selected candidate parameters are provided to the software application.

55. The electronic device of any one of claims 39-40, wherein providing the intent object to the software application to cause the software application to perform the task corresponding to the determined user intent comprises: After identifying the software application, determining whether user input requesting cancellation or modification of the task is received within a period of time; as well as Based on determining that the user input requesting cancellation or modification of the task is not received within the period of time, the intent object is provided to the software application.

56. The electronic device of claim 55, wherein providing the intent object to the software application based on determining that the user input requesting cancellation or modification of the task is not received within the period of time comprises: The intent object is provided to the software application without requiring user confirmation of the intent object.

57. The electronic device of any one of claims 39-40, wherein the one or more programs further comprise instructions for: receiving, from the software application, a request for parameters associated with the determined user intent; identifying, using the electronic device, a natural language query based on the request, wherein the natural language query is associated with the software application and the parameter; providing the natural language query; receiving a fourth user input; identifying the parameter based on the fourth user input; as well as The parameters are provided to the software application.

58. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: Device for carrying out the method according to any one of claims 1 to 2.

59. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system, the computer system communicating with a display generation component and one or more input devices, the one or more programs including instructions for executing a method according to any one of claims 1-2.

Images