SYSTEMS AND METHODS FOR PROACTIVE IDENTIFICATION AND DISPLAY OF RELEVANT CONTENT ON A TOUCH-SENSITIVE DEVICE

Description

TECHNICAL FIELD

[0001] The embodiments disclosed herein generally relate to electronic devices with touch-sensitive displays and, more specifically, to systems and methods for proactively identifying and surfacing relevant content on an electronic device that is in communication with a display and a touch-sensitive surface.BACKGROUND

[0002] Handheld electronic devices with touch-sensitive displays are ubiquitous. Users of these ubiquitous handheld electronic devices now install numerous applications on their devices and use these applications to help them perform their daily activities more efficiently. In order to access these applications, however, users typically must unlock their devices, locate a desired application (e.g., by navigating through a home screen to locate an icon associated with the desired application or by searching for the desired application within a search interface), and then also locate a desired function within the desired application. Therefore, users often spend a significant amount of time locating desired applications and desired functions within those applications, instead of simply being able to immediately execute (e.g., with a single touch input) the desired application and / or perform the desired function.

[0003] Moreover, the numerous installed applications inundate users with a continuous stream of information that cannot be thoroughly reviewed immediately. As such, users often wish to return at a later point in time to review a particular piece of information that they noticed earlier or to use a particular piece of information at a later point in time. Oftentimes, however, users are unable to locate or fail to remember how to locate the particular piece of information.

[0004] As such, it is desirable to provide an intuitive and easy-to-use system and method for proactively identifying and surfacing relevant content (e.g., the particular piece of information) on an electronic device that is in communication with a display and a touch-sensitive surface

[0005] EP2120142A2 discloses a location aware device e.g. wireless telephone has processor that presents user interface on display based on temporal parameter information from clock and geographic position information from position receiver.

[0006] US2014 / 282178A1 discloses a method for surfacing commands in user interface of CAD application on tablet, involves performing prediction calculations using command log views of data and community data to select predicted commands, and displaying predicted commands.

[0007] US2013 / 191911A1 discloses a device, method, and graphical user interface for accessing an application in a locked device.SUMMARY

[0008] The invention is defined by the appended set of claims.

[0009] Accordingly, there is a need for electronic devices with faster, more efficient methods and interfaces for quickly accessing applications and desired functions within those applications. Moreover, there is a need for electronic devices that assist users with managing the continuous stream of information they receive daily by proactively identifying and providing relevant information (e.g., contacts, nearby places, applications, news articles, addresses, and other information available on the device) before the information is explicitly requested by a user. Such methods and interfaces optionally complement or replace conventional methods for accessing applications. Such methods and interfaces produce a more efficient human-machine interface by requiring fewer inputs in order for users to locate desired information. For battery-operated devices, such methods and interfaces conserve power and increase the time between battery charges (e.g., by requiring a fewer number of touch inputs in order to perform various functions). Moreover, such methods and interfaces help to extend the life of the touch-sensitive display by requiring a fewer number of touch inputs (e.g., instead of having to continuously and aimlessly tap on a touch-sensitive display to locate a desired piece of information, the methods and interfaces disclosed herein proactively provide that piece of information without requiring user input).

[0010] The above deficiencies and other problems associated with user interfaces for electronic devices with touch-sensitive surfaces are addressed by the disclosed devices. In some embodiments, the device is a desktop computer. In some embodiments, the device is portable (e.g., a notebook computer, tablet computer, or handheld device). In some embodiments, the device has a touchpad. In some embodiments, the device has a touch-sensitive display (also known as a "touch screen" or "touch-screen display"). In some embodiments, the device has a graphical user interface (GUI), one or more processors, memory and one or more modules, programs or sets of instructions stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI primarily through stylus and / or finger contacts and gestures on the touch-sensitive surface. In some embodiments, the functions optionally include image editing, drawing, presenting, word processing, website creating, disk authoring, spreadsheet making, game playing, telephoning, video conferencing, e-mailing, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playing, and / or digital video playing. Executable instructions for performing these functions are, optionally, included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. (A1) In accordance with some embodiments, a method is performed at an electronic device (e.g., portable multifunction device 100, FIG. 1A, configured in accordance with any one of Computing Device A-D, Figure 1E) with a touch-sensitive display (touch screen 112, FIG. 1C). The method includes: executing, on the electronic device, an application in response to an instruction from a user of the electronic device. While executing the application, the method further includes: collecting usage data. The usage data at least includes one or more actions (or types of actions) performed by the user within the application. The method also includes: (i) automatically, without human intervention, obtaining at least one trigger condition based on the collected usage data and (ii) associating the at least one trigger condition with a particular action of the one or more actions performed by the user within the application. Upon determining that the at least one trigger condition has been satisfied, the method includes: providing an indication to the user that the particular action associated with the trigger condition is available. (A2) In some embodiments of the method of A1, obtaining the at least one trigger condition includes sending, to one or more servers that are remotely located from the electronic device, the usage data and receiving, from the one or more servers, the at least one trigger condition. (A3) According to the invention, providing the indication includes displaying, on a lock screen on the touch-sensitive display, a user interface object corresponding to the particular action associated with the trigger condition. (A4) According to the invention, the user interface object includes a description of the particular action associated with the trigger condition. (A5) In some embodiments of the method of A4, the user interface object further includes an icon associated with the application. (A6) In some embodiments of the method of any one of A3-A5, the method further includes: detecting a first gesture at the user interface object. In response to detecting the first gesture: (i) displaying, on the touch-sensitive display, the application and (ii) while displaying the application, the method includes: performing the particular action associated with the trigger condition. (A7) In some embodiments of the method of A6, the first gesture is a swipe gesture over the user interface object. (A8) According to the invention, the method further includes: detecting a second gesture at the user interface object. In response to detecting the second gesture and while continuing to display the lock screen on the touch-sensitive display, performing the particular action associated with the trigger condition. (A9) In some embodiments of the method of A8, the second gesture is a single tap at a predefined area of the user interface object. (A10) In some embodiments of the method of any one of A3-A9, the user interface object is displayed in a predefined central portion of the lock screen. (A11) In some embodiments of the method of A1, providing the indication to the user that the particular action associated with the trigger condition is available includes performing the particular action. (A12) In some embodiments of the method of A3, the user interface object is an icon associated with the application and the user interface object is displayed substantially in a corner of the lock screen on the touch-sensitive display. (A13) In some embodiments of the method of any one of A1-A12, the method further includes: receiving an instruction from the user to unlock the electronic device. In response to receiving the instruction, the method includes: displaying, on the touch-sensitive display, a home screen of the electronic device. The method also includes: providing, on the home screen, the indication to the user that the particular action associated with the trigger condition is available. (A14) In some embodiments of the method of A13, the home screen includes (i) a first portion including one or more user interface pages for launching a first set of applications available on the electronic device and (ii) a second portion, that is displayed adjacent to the first portion, for launching a second set of applications available on the electronic device. The second portion is displayed on all user interface pages included in the first portion and providing the indication on the home screen includes displaying the indication over the second portion. (A15) In some embodiments of the method of A14, the second set of applications is distinct from and smaller than the first set of applications. (A16) In some embodiments of the method of any one of A1-A15, determining that the at least one trigger condition has been satisfied includes determining that the electronic device has been coupled with a second device, distinct from the electronic device. (A17) In some embodiments of the method of any one of A1-A16, determining that the at least one trigger condition has been satisfied includes determining that the electronic device has arrived at an address corresponding to a home or a work location associated with the user. (A18) In some embodiments of the method of A17, determining that the electronic device has arrived at an address corresponding to the home or the work location associated with the user includes monitoring motion data from an accelerometer of the electronic device and determining, based on the monitored motion data, that the electronic device has not moved for more than a threshold amount of time. (A19) In some embodiments of the method of any one of A1-A18, the usage data further includes verbal instructions, from the user, provided to a virtual assistant application while continuing to execute the application. The at least one trigger condition is further based on the verbal instructions provided to the virtual assistant application. (A20) In some embodiments of the method of A19, the verbal instructions comprise a request to create a reminder that corresponds to a current state of the application, the current state corresponding to a state of the application when the verbal instructions were provided. (A21) In some embodiments of the method of A20, the state of the application when the verbal instructions were provided is selected from the group consisting of: a page displayed within the application when the verbal instructions were provided, content playing within the application when the verbal instructions were provided, a notification displayed within the application when the verbal instructions were provided, and an active portion of the page displayed within the application when the verbal instructions were provided. (A22) In some embodiments of the method of A20, the verbal instructions include the term "this" in reference to the current state of the application. (A23) In another aspect, a method is performed at one or more electronic devices (e.g., portable multifunction device 100, Figure 5, and one or more servers 502, Figure 5). The method includes: executing, on a first electronic device of the one or more electronic devices, an application in response to an instruction from a user of the first electronic device. While executing the application, the method includes: automatically, without human intervention, collecting usage data, the usage data at least including one or more actions (or types of actions) performed by the user within the application. The method further includes: automatically, without human intervention, establishing at least one trigger condition based on the collected usage data. The method additionally includes: associating the at least one trigger condition with particular action of the one or more actions performed by the user within the application. Upon determining that the at least one trigger condition has been satisfied, the method includes: providing an indication to the user that the particular action associated with the trigger condition is available. (A24) In another aspect, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive display, one or more processors, and memory storing one or more programs which, when executed by the one or more processors, cause the electronic device to perform the method described in any one of A1-A22. (A25) In yet another aspect, an electronic device is provided and the electronic device includes: a touch-sensitive display and means for performing the method described in any one of A 1-A22. (A26) In still another aspect, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores executable instructions that, when executed by an electronic device with a touch-sensitive display, cause the electronic device to perform the method described in any one of A1-A22. (A27) In still one more aspect, a graphical user interface on an electronic device with a touch-sensitive display is provided. In some embodiments, the graphical user interface includes user interfaces displayed in accordance with the method described in any one of A 1-A22. (A28) In one additional aspect, an electronic device is provided that includes a display unit (e.g., display unit 4201, Figure 42), a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4203, Figure 42), and a processing unit (e.g., processing unit 4205, Figure 42). In some embodiments, the electronic device is configured in accordance with any one of the computing devices shown in Figure 1E (i.e., Computing Devices A-D). For ease of illustration, Figure 42 shows display unit 4201 and touch-sensitive surface unit 4203 as integrated with electronic device 4200, however, in some embodiments one or both of these units are in communication with the electronic device, although the units remain physically separate from the electronic device. The processing unit is coupled with the touch-sensitive surface unit and the display unit. In some embodiments, the touch-sensitive surface unit and the display unit are integrated in a single touch-sensitive display unit (also referred to herein as a touch-sensitive display). The processing unit includes an executing unit (e.g., executing unit 4207, Figure 42), a collecting unit (e.g., collecting unit 4209, Figure 42), an obtaining unit (e.g., obtaining unit 4211, Figure 42), an associating unit (e.g., associating unit 4213, Figure 42), a providing unit (e.g., providing unit 4215, Figure 42), a sending unit (e.g., sending unit 4217, Figure 42), a receiving unit (e.g., receiving unit 4219, Figure 42), a displaying unit (e.g., displaying unit 4221, Figure 42), a detecting unit (e.g., detecting unit 4223, Figure 42), a performing unit (e.g., performing unit 4225, Figure 42), a determining unit (e.g., determining unit 4227, Figure 42), and a monitoring unit (e.g., monitoring unit 4229, Figure 42). The processing unit (or one or more components thereof, such as the units 4207-4229) is configured to: execute (e.g., with the executing unit 4207), on the electronic device, an application in response to an instruction from a user of the electronic device; while executing the application, collect usage data (e.g., with the collecting unit 4209), the usage data at least including one or more actions performed by the user within the application; automatically, without human intervention, obtain (e.g., with the obtaining unit 4211) at least one trigger condition based on the collected usage data; associate (e.g., with the associating unit 4213) the at least one trigger condition with a particular action of the one or more actions performed by the user within the application; and upon determining that the at least one trigger condition has been satisfied, provide (e.g., with the providing unit 4215) an indication to the user that the particular action associated with the trigger condition is available. (A29) In some embodiments of the electronic device of A28, obtaining the at least one trigger condition includes sending (e.g., with the sending unit 4217), to one or more servers that are remotely located from the electronic device, the usage data and receiving (e.g., with the receiving unit 4219), from the one or more servers, the at least one trigger condition. (A30) In some embodiments of the electronic device of any one of A28-A29, providing the indication includes displaying (e.g., with the displaying unit 4217 and / or the display unit 4201), on a lock screen on the touch-sensitive display unit, a user interface object corresponding to the particular action associated with the trigger condition. (A31) In some embodiments of the electronic device of A30, the user interface object includes a description of the particular action associated with the trigger condition. (A32) In some embodiments of the electronic device of A31, the user interface object further includes an icon associated with the application. (A33) In some embodiments of the electronic device of any one of A30-A32, the processing unit is further configured to: detect (e.g., with the detecting unit 4223 and / or the touch-sensitive surface unit 4203) a first gesture at the user interface object. In response to detecting the first gesture: (i) display (e.g., with the displaying unit 4217 and / or the display unit 4201), on the touch-sensitive display unit, the application and (ii) while displaying the application, perform (e.g., with the performing unit 4225) the particular action associated with the trigger condition. (A34) In some embodiments of the electronic device of A33, the first gesture is a swipe gesture over the user interface object. (A35) In some embodiments of the electronic device of any one of A30-A33, the processing unit is further configured to: detect (e.g., with the detecting unit 4223 and / or the touch-sensitive surface unit 4203) a second gesture at the user interface object. In response to detecting the second gesture and while continuing to display the lock screen on the touch-sensitive display unit, the processing unit is configured to: perform (e.g., with the performing unit 4225) the particular action associated with the trigger condition. (A36) In some embodiments of the electronic device of A35, the second gesture is a single tap at a predefined area of the user interface object. (A37) In some embodiments of the electronic device of any one of A30-A36, the user interface object is displayed in a predefined central portion of the lock screen. (A38) In some embodiments of the electronic device of A28, providing the indication to the user that the particular action associated with the trigger condition is available includes performing (e.g., with the performing unit 4225) the particular action. (A39) In some embodiments of the electronic device of A30, the user interface object is an icon associated with the application and the user interface object is displayed substantially in a corner of the lock screen on the touch-sensitive display unit. (A40) In some embodiments of the electronic device of any one of A28-A39, the processing unit is further configured to: receive (e.g., with the receiving unit 4219) an instruction from the user to unlock the electronic device. In response to receiving the instruction, the processing unit is configured to: display (e.g., with the displaying unit 4217 and / or the display unit 4201), on the touch-sensitive display unit, a home screen of the electronic device. The processing unit is also configured to: provide (e.g., with the providing unit 4215), on the home screen, the indication to the user that the particular action associated with the trigger condition is available. (A41) In some embodiments of the electronic device of A40, the home screen includes (i) a first portion including one or more user interface pages for launching a first set of applications available on the electronic device and (ii) a second portion, that is displayed adjacent to the first portion, for launching a second set of applications available on the electronic device. The second portion is displayed on all user interface pages included in the first portion and providing the indication on the home screen includes displaying (e.g., with the displaying unit 4217 and / or the display unit 4201) the indication over the second portion. (A42) In some embodiments of the electronic device of A41, the second set of applications is distinct from and smaller than the first set of applications. (A43) In some embodiments of the electronic device of any one of A28-A42, determining that the at least one trigger condition has been satisfied includes determining (e.g., with the determining unit 4227) that the electronic device has been coupled with a second device, distinct from the electronic device. (A44) In some embodiments of the electronic device of any one of A28-A43, determining that the at least one trigger condition has been satisfied includes determining (e.g., with the determining unit 4227) that the electronic device has arrived at an address corresponding to a home or a work location associated with the user. (A45) In some embodiments of the electronic device of A44, determining that the electronic device has arrived at an address corresponding to the home or the work location associated with the user includes monitoring (e.g., with the monitoring unit 4229) motion data from an accelerometer of the electronic device and determining (e.g., with the determining unit 4227), based on the monitored motion data, that the electronic device has not moved for more than a threshold amount of time. (A46) In some embodiments of the electronic device of any one of A28-A45, the usage data further includes verbal instructions, from the user, provided to a virtual assistant application while continuing to execute the application. The at least one trigger condition is further based on the verbal instructions provided to the virtual assistant application. (A47) In some embodiments of the electronic device of A46, the verbal instructions comprise a request to create a reminder that corresponds to a current state of the application, the current state corresponding to a state of the application when the verbal instructions were provided. (A48) In some embodiments of the electronic device of A47, the state of the application when the verbal instructions were provided is selected from the group consisting of: a page displayed within the application when the verbal instructions were provided, content playing within the application when the verbal instructions were provided, a notification displayed within the application when the verbal instructions were provided, and an active portion of the page displayed within the application when the verbal instructions were provided. (A49) In some embodiments of the electronic device of A46, the verbal instructions include the term "this" in reference to the current state of the application.

[0011] As described above (and in more detail below), one aspect of the present technology is the gathering and use of data available from various sources (e.g., based on speech provided during voice communications) to improve the delivery to users of content that may be of interest to them. The present disclosure contemplates that in some instances, this gathered data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data can include demographic data, location-based data, telephone numbers, email addresses, home addresses, or any other identifying information.

[0012] The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, the personal information data can be used to deliver targeted content that is of greater interest to the user. Accordingly, use of such personal information data enables calculated control of the delivered content. Further, other uses for personal information data that benefit the user are also contemplated by the present disclosure.

[0013] The present disclosure further contemplates that the entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data will comply with well-established privacy policies and / or privacy practices. In particular, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining personal information data private and secure. For example, personal information from users should be collected for legitimate and reasonable uses of the entity and not shared or sold outside of those legitimate uses. Further, such collection should occur only after receiving the informed consent of the users. Additionally, such entities would take any needed steps for safeguarding and securing access to such personal information data and ensuring that others with access to the personal information data adhere to their privacy policies and procedures. Further, such entities can subject themselves to evaluation by third parties to certify their adherence to widely accepted privacy policies and practices.

[0014] Despite the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and / or software elements can be provided to prevent or block access to such personal information data. For example, in the case of monitoring voice communications or monitoring actions performed by users within applications, the present technology can be configured to allow users to select to "opt in" or "opt out" of participation in the collection of personal information data during registration for services. In another example, users can select not to provide location information for targeted content delivery services. In yet another example, users can select to not provide precise location information, but permit the transfer of location zone information.

[0015] Therefore, although the present disclosure broadly covers use of personal information data to implement one or more various disclosed embodiments, the present disclosure also contemplates that the various embodiments can also be implemented without the need for accessing such personal information data. That is, the various embodiments of the present technology are not rendered inoperable due to the lack of all or a portion of such personal information data. For example, content can be selected and delivered to users by inferring preferences based on non-personal information data or a bare minimum amount of personal information, such as the content being requested by the device associated with a user, other non-personal information available to the content delivery services, or publically available information.

[0016] Note that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in the specification are not all inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] For a better understanding of the various described embodiments, reference should be made to the Description of Embodiments section below, in conjunction with the following drawings in which like reference numerals refer to corresponding parts throughout the drawings. Figure 1A is a high-level block diagram of a computing device with a touch-sensitive display, in accordance with some embodiments. Figure 1B is a block diagram of example components for event handling, in accordance with some embodiments. Figure 1C is a schematic of a portable multifunction device having a touch-sensitive display, in accordance with some embodiments. Figure 1D is a schematic used to illustrate a computing device with a touch-sensitive surface that is separate from the display, in accordance with some embodiments. Figure 2 is a schematic of a touch screen used to illustrate a user interface for a menu of applications, in accordance with some embodiments. Figures 3A-3B are block diagrams illustrating data structures for storing application usage data, in accordance with some embodiments. Figures 4A-4B are block diagrams illustrating data structures for storing trigger conditions, in accordance with some embodiments. Figure 5 is a block diagram illustrating an example of a trigger condition establishing system, in accordance with some embodiments. Figures 6A-6B are a flowchart representation of a method of proactively identifying and surfacing (e.g., surfacing for user selection) relevant content on an electronic device with a touch-sensitive display, in accordance with some embodiments. Figures 7A-7B are schematics of a touch-sensitive display used to illustrate user interfaces for proactively identifying and surfacing relevant content, in accordance with some embodiments. Figures 8A-8B are a flowchart representation of a method of proactively identifying and surfacing (e.g., surfacing for user selection) relevant content on an electronic device with a touch-sensitive display, in accordance with some embodiments. Figures 9A-9D are schematics of a touch-sensitive display used to illustrate user interfaces for proactively identifying and surfacing relevant content, in accordance with some embodiments. Figures 10A-10C are a flowchart representation of a method of proactively suggesting search queries based on content currently being displayed on an electronic device with a touch-sensitive display, in accordance with some embodiments. Figures 11A-11J are schematics of a touch-sensitive display used to illustrate user interfaces for proactively suggesting search queries based on content currently being displayed on the touch-sensitive display, in accordance with some embodiments. Figure 12 is a flowchart representation of a method of entering a search mode based on heuristics, in accordance with some embodiments. Figures 13A-13B are schematics of a touch-sensitive display used to illustrate user interfaces for entering a search mode based on heuristics, in accordance with some embodiments. Figure 14 is a flowchart representation of a method of proactively providing vehicle location on an electronic device with a touch-sensitive display, in accordance with some embodiments. Figures 15A-15B are schematics of a touch-sensitive display used to illustrate user interfaces for proactively providing vehicle location, in accordance with some embodiments. Figures 16A-16B are a flowchart representation of a method of proactively providing nearby point of interest (POI) information for search queries, in accordance with some embodiments. Figures 17A-17E are schematics of a touch-sensitive display used to illustrate user interfaces for proactively providing nearby point of interest (POI) information for search queries, in accordance with some embodiments. Figures 18A-18B are a flowchart representation of a method of extracting a content item from a voice communication and interacting with the extracted content item, in accordance with some embodiments. Figures 19A-19F are schematics of a touch-sensitive display used to illustrate user interfaces for displaying and interacting with content items that have been extracted from voice communications, in accordance with some embodiments. Figure 20 is a flowchart representation of a method of determining that a voice communication includes speech that identifies a physical location and populating an application with information about the physical location, in accordance with some embodiments. Figures 21A-21B are schematics of a touch-sensitive display used to illustrate user interfaces for determining that a voice communication includes speech that identifies a physical location and populating an application with information about the physical location, in accordance with some embodiments. Figures 22A-22B are a flowchart representation of a method of proactively suggesting physical locations for use in a messaging application, in accordance with some embodiments. Figure 22C is a flowchart representation of a method of proactively suggesting information that relates to locations, events, or contacts, in accordance with some embodiments. Figures 23A-23O are schematics of a touch-sensitive display used to illustrate user interfaces for proactively suggesting information that relates to locations, events, or contacts (e.g., for easy selection by a user and inclusion in a messaging application), in accordance with some embodiments. Figures 24A-24B are a flowchart representation of a method of proactively populating an application with information that was previously viewed by a user in a different application, in accordance with some embodiments. Figures 25A-25J are schematics of a touch-sensitive display used to illustrate user interfaces for proactively populating an application with information that was previously viewed by a user in a different application (e.g., populating a ride-sharing application with information about locations viewed by the user in a reviewing application), in accordance with some embodiments. Figures 26A-26B are a flowchart representation of a method of proactively suggesting information that was previously viewed by a user in a first application for use in a second application, in accordance with some embodiments. Figures 27 is a flowchart representation of a method of proactively suggesting a physical location for use as a destination for route guidance in a vehicle, in accordance with some embodiments. Figure 28 is a schematic of a touch-sensitive display used to illustrate a user interface for proactively suggesting a physical location for use as a destination for route guidance in a vehicle, in accordance with some embodiments. Figures 29 is a flowchart representation of a method of proactively suggesting a paste action, in accordance with some embodiments. Figures 30A-30D are schematics of a touch-sensitive display used to illustrate user interfaces for proactively suggesting a paste action, in accordance with some embodiments. Figure 31_1 illustrates a mobile device configured to perform dynamic adjustment of the mobile device, in accordance with some embodiments. Figure 31_2 illustrates an example process for invoking heuristic processes, in accordance with some embodiments. Figure 31_3 illustrates a process for adjusting the settings of a mobile device using a heuristic process, in accordance with some embodiments. Figure 31_4 illustrates an example system for performing background fetch updating of applications, in accordance with some embodiments. Figure 31_5 illustrates peer forecasting for determining user invocation probabilities for applications on mobile device 100, in accordance with some embodiments. Figure 31_6 is a flow diagram of an example process for predictively launching applications to perform background updates, in accordance with some embodiments. Figure 31_7 is a flow diagram of an example process for determining when to launch applications on a mobile device, in accordance with some embodiments. Figure 31_8 is a flow diagram illustrating state transitions for an entry in a trending table, in accordance with some embodiments. Figure 31_9 is a block diagram illustrating a system for providing push notifications to a mobile device, in accordance with some embodiments. Figure 31_10 is a flow diagram of an example process for performing non-waking pushes at a push notification server, in accordance with some embodiments. Figure 31_11 is a flow diagram of an example process for performing background updating of an application in response to a low priority push notification, in accordance with some embodiments. Figure 31_12 is a flow diagram of an example process for performing background updating of an application in response to a high priority push notification, in accordance with some embodiments. Figure 31_13 is a block diagram an example system for performing background downloading and / or uploading of data on a mobile device, in accordance with some embodiments. Figure 31_14 is flow diagram of an example process for performing background downloads and uploads, in accordance with some embodiments. Figure 31_15 illustrates an example graphical user interface (GUI) for enabling and / or disabling background updates for applications on a mobile device, in accordance with some embodiments. Figure 31_16 illustrates an example system for sharing data between peer devices, in accordance with some embodiments. Figure 31_17 illustrates an example process for sharing data between peer devices, in accordance with some embodiments. Figure 32_1 is a block diagram of one embodiment of a system that returns search results based on input query prefixes, in accordance with some embodiments. Figure 32_2 is flowchart of one embodiment of a process to determine query completions and relevant results based on an input query prefix, in accordance with some embodiments. Figure 32_3 is a block diagram of one embodiment of an aggregator and multiple search domains, in accordance with some embodiments. Figure 32_4 is an illustration of one embodiment to a query completion search domain, in accordance with some embodiments. Figure 32_5 is an illustration of one embodiment of a maps search domain. Figure 32_6 is a flow chart of one embodiment of a process to determine query completions from multiple search domains, in accordance with some embodiments. Figure 32_7 is a flow chart of one embodiment of a process to determine relevant results over multiple search domains from a determined query completion, in accordance with some embodiments. Figure 32_8 is a block diagram of one embodiment of a system that incorporates user feedback into a feedback search index, in accordance with some embodiments. Figure 32_9 is a flow chart of one embodiment of a process to incorporate user feedback into a citation search index, in accordance with some embodiments. Figure 32_10 is a flow chart of one embodiment of a process to collect user feedback during a user search session, in accordance with some embodiments. Figure 32_11 is a flow chart of one embodiment of a process to incorporate user feedback during into a feedback index, in accordance with some embodiments. Figure 32_12 is a flow chart of one embodiment of a process to use the user feedback to update a results cache, in accordance with some embodiments. Figure 32_13 is a block diagram of one embodiment of a federator that performs a multi- domain search using a characterized query completion, in accordance with some embodiments. Figure 32_14 is a flow chart of one embodiment of a process to determine relevant results using a vocabulary service, in accordance with some embodiments. Figure 32_15 is a flow chart of one embodiment of a process to characterize a query completion, in accordance with some embodiments. Figure 32_16 is a block diagram of one embodiment of a completion module to determine query completions from multiple search domains, in accordance with some embodiments. Figure 32_17 is a block diagram of one embodiment of a results module to determine relevant results over multiple search domains from a determined query completion, in accordance with some embodiments. Figure 32_18 is a block diagram of one embodiment of a collect feedback module to collect user feedback during a user search session, in accordance with some embodiments. Figure 32_19 is a block diagram of one embodiment of a process feedback module to incorporate user feedback during into a feedback index, in accordance with some embodiments. Figure 32_20 is a block diagram of one embodiment of an update query results module to use the user feedback to update a results cache, in accordance with some embodiments. Figure 32_21 is a block diagram of one embodiment of a process feedback module to incorporate user feedback during into a feedback index, in accordance with some embodiments. Figure 32_22 is a block diagram of one embodiment of an update query results module to use the user feedback to update a results cache, in accordance with some embodiments. Figure 33_1 illustrates, in block diagram form, a local search subsystem and a remote search subsystem on a computing device as is known in the prior art. Figure 33_2 illustrates, in block diagram form, a local search subsystem having local learning capability that can be used to improve the results returned from a remote search application on a computing device, in accordance with some embodiments. Figure 33_3 illustrates, in block diagram form, a method of locally learning a query feature utilizing local search queries, local results and local feedback based on the local results. , in accordance with some embodiments Figure 33_4 illustrates, in block diagram form, a method of locally learning a query feature utilizing search results returned from both local search queries and remote search queries, and local feedback on both local and remote search query results, in accordance with some embodiments. Figure 33_5 illustrates, in block diagram form, a method of locally learning a query feature passed to a local device by a remote service in response to a query sent to the remote service, in accordance with some embodiments. Figure 33_6 illustrates, in block diagram form, a method of receiving or determining a new feature, locally training on the feature, and utilizing the feature, in accordance with some embodiments. Figure 33_7 illustrates an exemplary embodiment of a software stack usable in some embodiments of the invention, in accordance with some embodiments. FIG. 34_5A illustrates a block diagram of an exemplary data architecture for suggested contacts in accordance with some embodiments. FIG. 34_5B illustrates a block diagram of an exemplary data architecture for suggested calendar events in accordance with some embodiments. FIGS. 34_6A-34_6G illustrate exemplary user interfaces for providing suggested contacts and calendar events in accordance with some embodiments. FIGS. 1A-1B, 2, and 3 provide a description of exemplary devices for performing the techniques for suggesting contact and event information described in this section. FIGS. 34_6A-34_6G illustrate exemplary user interfaces for suggesting contact and event information, and the user interfaces in these figures are also used to illustrate the processes described below, including the processes in FIGS. 34_7A-34_13. FIGS. 34_7A and 34_7B illustrate a flow diagram of an exemplary process for generating a suggested contact in accordance with some embodiments. FIGS. 34_8A and 34_8B illustrate a flow diagram of an exemplary process for updating an existing contact with a suggested item of contact information in accordance with some embodiments. FIGS. 34_9A and 34_9B illustrate a flow diagram of an exemplary process for displaying a contact with suggested contact information in accordance with some embodiments. FIG. 34_10 illustrates a flow diagram of an exemplary process for displaying suggested contact information with a message in accordance with some embodiments. FIGS. 34_11A and 34_11B illustrate a flow diagram of an exemplary process for generating a suggested calendar event in accordance with some embodiments. FIG. 34_12 illustrates a flow diagram of an exemplary process for displaying suggested event information with a message in accordance with some embodiments. FIG. 34_13 illustrates a flow diagram of an exemplary process for displaying multiple suggested contact or event information with a message in accordance with some embodiments. FIG. 34_14 is a functional block diagram of an electronic device in accordance with some embodiments. FIG. 34_15 is a functional block diagram of an electronic device in accordance with some embodiments. FIG. 35_1 is a flow chart of a method 35_100 for suggesting an application based upon a detected event according to some embodiments. FIG. 35_2 shows a segmentation process 35_200 according to some embodiments. FIG. 35_3 shows a decision tree 35_300 that may be generated according to some embodiments. FIG. 35_4 is a flowchart of a method 35_400 for suggesting an application to a user of a computing device based on an event according to some embodiments. FIGS. 35_5A-35_5D shows plots of example binomial distributions for various correct numbers and incorrect numbers according to some embodiments. FIGS. 35_6A and 35_6B show a parent model and a sub-model resulting from a segmentation according to some embodiments. FIG. 35_7 shows an example architecture 35_700 for providing a user interface to the user for interacting with the one or more applications, in accordance with some embodiments. FIG. 36_1 is a flowchart of a method for identifying an application based upon a triggering event according to some embodiments. FIG. 36_2 shows a block diagram of a system for determining a triggering event according to some embodiments. FIG. 36_3 shows a block diagram of a system for identifying an application for a user based on a triggering event according to some embodiments. FIG. 36_4 shows a block diagram of a system for identifying an application with multiple prediction models according to some embodiments. FIG. 36_5 is a flowchart of a method of identifying an application based on a triggering event with a device according to some embodiments. FIG. 36_6 is a simplified diagram of a device having a user interface for a music application according to some embodiments. FIGS. 36_7A and 36_7B are flowcharts of methods for removing an identified application from a user interface according to some embodiments. FIG. 37_1 is a flow chart of a method 100 for suggesting a recipient to contact based upon a detected event according to some embodiments. FIG. 37_2 shows a block diagram of a system for determining a triggering event according to some embodiments. FIG. 37_3 shows a block diagram of a system for identifying recipients to contact based on a triggering event according to some embodiments. FIG. 37_4 shows an example of suggesting recipients to contact in a user interface for an email application according to some embodiments. FIG. 37_5 shows an example of suggesting recipients to contact in a user interface for a search application according to some embodiments. FIG. 37_6 is a flowchart of a method 37_600 for suggesting recipients to a user of a computing device based on an event according to some embodiments. FIG. 37_7 shows an example data flow for suggesting recipients to contact according to some embodiments. FIG. 37_8 shows a block diagram of an interaction module according to some embodiments. FIG. 37_9 shows an example architecture 37_900 for providing a user interface to the user for suggesting recipients to contact according to some embodiments. FIG. 38_1 illustrates a block diagram of different components of a mobile computing device configured to implement the various techniques described herein, according to some embodiments. FIG. 38_2 illustrates a method that is implemented by the application prediction engine of FIG. 38_1, according to some embodiments. FIG. 38_3 illustrates a method that is implemented by the search application of FIG. 1, according to some embodiments. FIG. 38_4 illustrates a conceptual diagram of an example user interface of the search application of FIG. 38_1, according to some embodiments. FIG. 39_1 illustrates a block diagram of different components of a mobile computing device configured to implement the various techniques described herein, according to some embodiments. FIG. 39_2 illustrates a block diagram of a more detailed view of particular components of the mobile computing device illustrated in FIG. 39_1 (or Figure 1A), according to some embodiments. FIG. 39_3A illustrates a method for a high-level initialization and operation of a prediction engine, according to some embodiments. FIG. 39_3B illustrates a method for synchronously providing a prediction at a prediction engine, according to some embodiments. FIG. 39_3C illustrates a method for asynchronously providing a prediction at a prediction engine, according to some embodiments. FIG. 39_4A illustrates a method for a consumer application requesting to synchronously receive a prediction, according to some embodiments. FIG. 39_4B illustrates a method for a consumer application registering to asynchronously receive predictions, according to some embodiments. FIG. 39_5A illustrates a method for managing prediction engine registrations at a prediction engine center, according to some embodiments. FIG. 39_5B illustrates a method for synchronously providing predictions to consumer applications at a prediction engine center, according to some embodiments. FIG. 39_5C illustrates a method for asynchronously providing predictions to consumer applications at a prediction engine center, according to some embodiments. FIG. 40_1 is a block diagram of an example system for monitoring, predicting, and notifying context clients of changes in the current context of a computing device, in accordance with some embodiments. FIG. 40_2A illustrates an example of context items that can make up the current context, in accordance with some embodiments. FIG. 40_2B illustrates an example of a new context item being added to the current context, in accordance with some embodiments. FIG. 40_3 illustrates an example callback predicate database, in accordance with some embodiments. FIG. 40_4 is a graph that illustrates example state changes associated with context items over time, in accordance with some embodiments. FIG. 40_5 is a graph that illustrates example event streams associated with context items, in accordance with some embodiments. FIG. 40_6 illustrates an example historical event stream database, in accordance with some embodiments. FIG. 40_7 is a block diagram of an example system for providing a context callback notification to a requesting client, in accordance with some embodiments. FIG. 40_8A is a block diagram of an example system illustrating restarting a requesting client that has been terminated, in accordance with some embodiments. FIG. 40_8B is a block diagram of an example system illustrating restarting a requesting client that has been terminated, in accordance with some embodiments. FIG. 40_9A is a block diagram of an example system illustrating restarting a context daemon that has been terminated, in accordance with some embodiments. FIG. 40_9B is a block diagram of an example system illustrating restarting a context daemon that has been terminated, in accordance with some embodiments. FIG. 40_10A is a block diagram of an example system illustrating restarting a context daemon and a requesting client that have been terminated, in accordance with some embodiments. FIG. 40_10B is a block diagram of an example system illustrating restarting a context daemon and a requesting client that have been terminated, in accordance with some embodiments. FIG. 40_11 is a block diagram of an example system configured to restart a client and / or a context daemon based on device state information received by a launch daemon, in accordance with some embodiments. FIG. 40_12A is a block diagram of an example system illustrating restarting a context daemon using a launch daemon, in accordance with some embodiments. FIG. 40_12B is a block diagram of an example system illustrating restarting a context daemon using a launch daemon, in accordance with some embodiments. FIG. 40_13A is a block diagram of an example system illustrating restarting a requesting client using a launch daemon, in accordance with some embodiments. FIG. 40_13B is a block diagram of an example system illustrating restarting a requesting client using a launch daemon, in accordance with some embodiments. FIG. 40_14 is a graph that illustrates an example of slot-wise averaging to predict future events, in accordance with some embodiments. FIG. 40_15 depicts example graphs illustrating slot weighting, in accordance with some embodiments. FIG. 40_16A is a graph illustrating an example method for predicting a future context, in accordance with some embodiments. FIG. 40_16B is a graph illustrating an example method for converting slot-wise probabilities into a probability curve, in accordance with some embodiments. FIG. 40_17 illustrates an example event stream that includes a predicted future event, in accordance with some embodiments. FIG. 40_18 is a flow diagram of an example process for notifying clients of context changes on a computing device, in accordance with some embodiments. FIG. 40_19 is a flow diagram of an example process for restarting a context daemon to service a callback request, in accordance with some embodiments. FIG. 40_20 is a flow diagram of an example process for restarting a callback client to receive a callback notification, in accordance with some embodiments. FIG. 40_21 is a flow diagram of an example process for predicting future events based on historical context information, in accordance with some embodiments. FIG. 40_22 is a flow diagram of an example process for servicing a sleep context callback request, in accordance with some embodiments. Figure 41_1 is a block diagram of one embodiment of a system that indexes application states for use in a local device search index. Figure 41_2 is a block diagram of one embodiment of a system that searches application states using an on-device application state search index. Figure 41_3 is a block diagram of embodiments of user interfaces that display an application state query results among other query results. Figure 41_4A is a flow diagram of one embodiment of a process to index application states received from multiple different applications on a device. Figure 41_4B is a flow diagram of one embodiment of a process to determine query results for a query using an application state index. Figure 41_5 is a flow diagram of one embodiment of a process to receive and present an application state as part of a query result. Figure 41_6 is a block diagram of one embodiment of a system that indexes application states for use in a remote search index. Figure 41_7 is a block diagram of one embodiment of a system that searches application states using a remote application state search index. Figure 41_8 is a flow diagram of one embodiment of a process to add an application state to an application state index. Figure 41_9 is a flow diagram of one embodiment of a process to export an application state to an application state indexing service. Figure 41_10 is a flow chart of one embodiment of a process to perform a query search using an application state index. Figure 41_11 is a flow diagram of one embodiment of a process to receive and present an application state as part of a query result. Figure 41_12 is a block diagram of one embodiment of a system that indexes application state views for use in a remote search index. Figure 41_13 is a block diagram of one embodiment of an application view. Figure 41_14 is a flow chart of one embodiment of a process to generate an application state view using an application state. Figure 41_15 is a flow chart of one embodiment of a process to receive and present an application state that includes an application state view as part of a query result. Figures 42-55 are functional block diagrams of an electronic device, in accordance with some embodiments. DESCRIPTION OF EMBODIMENTS

[0018] As discussed above and in more detail below, there is a need for electronic devices with faster, more efficient methods and interfaces for quickly accessing applications and desired functions within those applications. In particular, there is a need for devices that help users to avoid repetitive tasks and provide proactive assistance by identifying and providing relevant information before a user explicitly requests it. Additionally, there is a need for quickly accessing applications and desired functions within those applications at particular periods of time (e.g., accessing a calendar application after waking up each morning), at particular places (e.g., accessing a music application at the gym), etc. Disclosed herein are novel methods and interfaces to address these needs and provide users with ways to quickly access applications and functions within those applications at these particular places, periods of time, etc. Such methods and interfaces optionally complement or replace conventional methods for accessing applications. Such methods and interfaces reduce the cognitive burden on a user and produce a more efficient human-machine interface. For battery-operated devices, such methods and interfaces conserve power and increase the time between battery charges. Moreover, such methods and interfaces help to extend the life of the touch-sensitive display by requiring a fewer number of touch inputs (e.g., instead of having to continuously and aimlessly tap on a touch-sensitive display to located a desired piece of information, the methods and interfaces disclosed herein proactively provide that piece of information without requiring user input).

[0019] Below, Figures 1A-1E and 2 provide a description of example devices. Figures 10 and 11 provide functional block diagrams of example electronic devices. Figures 3A-3B and Figures 4A-4B are block diagrams of example data structures that are used to proactively identify and surface relevant content (these data structures are used in the method described in reference to Figures 6A-6B and in the method described with reference to Figures 8A-8B). Figure 5 is a block diagram illustrating an example system for establishing trigger conditions that are used to proactively identify and surface relevant content (the example system is used in the method described in reference to Figures 6A-6B and in the method described with reference to Figures 8A-8B). Figures 6A-6B are a flowchart depicting a method of proactively identifying and surfacing relevant content. Figures 7A-7B are schematics of a touch-sensitive display used to illustrate example user interfaces and gestures for proactively identifying and surfacing relevant content. Figures 8A-8B are a flowchart depicting a method of proactively identifying and surfacing relevant content. Figures 9A-9D are schematics of a touch-sensitive display used to illustrate additional user interfaces for proactively identifying and surfacing relevant content. Figures 3A-3B, 4A-4B, 5, and 7A-7B are used to illustrate the methods and / or processes of Figures 6A-6B. Figures 3A-3B, 4A-4B, 5, and 9A-9D are used to illustrate the methods and / or processes of Figures 8A-8B.

[0020] Figures 10A-10C are a flowchart depicting a method of proactively suggesting search queries based on content currently being displayed on an electronic device with a touch-sensitive display. Figures 11A-11J are schematics of a touch-sensitive display used to illustrate user interfaces for proactively suggesting search queries based on content currently being displayed on the touch-sensitive display. Figures 11A-11J are used to illustrate the methods and / or processes of Figures 10A-10C.

[0021] Figure 12 is a flowchart representation of a method of entering a search mode based on heuristics. Figures 13A-13B are schematics of a touch-sensitive display used to illustrate user interfaces for entering a search mode based on heuristics. Figures 13A-13B are used to illustrate the methods and / or processes of Figure 12.

[0022] Figure 14 is a flowchart representation of a method of proactively providing vehicle location on an electronic device with a touch-sensitive display, in accordance with some embodiments. Figures 15A-15B are schematics of a touch-sensitive display used to illustrate user interfaces for proactively providing vehicle location, in accordance with some embodiments. Figures 15A-15B are used to illustrate the methods and / or processes of Figure 14.

[0023] Figures 16A-16B are a flowchart representation of a method of proactively providing nearby point of interest (POI) information for search queries, in accordance with some embodiments. Figures 17A-17E are schematics of a touch-sensitive display used to illustrate user interfaces for proactively providing nearby point of interest (POI) information for search queries, in accordance with some embodiments. Figures 16A-16B are used to illustrate the methods and / or processes of Figures 17A-17E.

[0024] Figures 18A-18B are a flowchart representation of a method of extracting a content item from a voice communication and interacting with the extracted content item, in accordance with some embodiments. Figures 19A-19F are schematics of a touch-sensitive display used to illustrate user interfaces for displaying and interacting with content items that have been extracted from voice communications, in accordance with some embodiments. Figures 19A-19F are used to illustrate the methods and / or processes of Figures 18A-18B.

[0025] Figure 20 is a flowchart representation of a method of determining that a voice communication includes speech that identifies a physical location and populating an application with information about the physical location, in accordance with some embodiments. Figures 21A-21B are schematics of a touch-sensitive display used to illustrate user interfaces for determining that a voice communication includes speech that identifies a physical location and populating an application with information about the physical location, in accordance with some embodiments. Figures 19A-19F and Figures 21A-21B are used to illustrate the methods and / or processes of Figure 20.

[0026] Figures 22A-22B are a flowchart representation of a method of proactively suggesting physical locations for use in a messaging application, in accordance with some embodiments. Figures 23A-23O are schematics of a touch-sensitive display used to illustrate user interfaces for proactively suggesting information that relates to locations, events, or contacts (e.g., for easy selection by a user and inclusion in a messaging application), in accordance with some embodiments. Figures 23A-23O are used to illustrate the methods and / or processes of Figures 22A-22B.

[0027] Figure 22C is a flowchart representation of a method of proactively suggesting information that relates to locations, events, or contacts, in accordance with some embodiments. Figures 23A-23O are used to illustrate the methods and / or processes of Figure 22C.

[0028] Figures 24A-24B are a flowchart representation of a method of proactively populating an application with information that was previously viewed by a user in a different application, in accordance with some embodiments. Figures 25A-25J are schematics of a touch-sensitive display used to illustrate user interfaces for proactively populating an application with information that was previously viewed by a user in a different application (e.g., populating a ride-sharing application with information about locations viewed by the user in a reviewing application), in accordance with some embodiments. Figures 25A-25J are used to illustrate the methods and / or processes of Figures 24A-24B.

[0029] Figures 26A-26B are a flowchart representation of a method of proactively suggesting information that was previously viewed by a user in a first application for use in a second application, in accordance with some embodiments. Figures 25A-25J are used to illustrate the methods and / or processes of Figures 26A-26B.

[0030] Figures 27 is a flowchart representation of a method of proactively suggesting a physical location for use as a destination for route guidance in a vehicle, in accordance with some embodiments. Figure 28 is a schematic of a touch-sensitive display used to illustrate a user interface for proactively suggesting a physical location for use as a destination for route guidance in a vehicle, in accordance with some embodiments. Figure 28 is used to illustrate the methods and / or processes of Figure 27.

[0031] Figures 29 is a flowchart representation of a method of proactively suggesting a paste action, in accordance with some embodiments. Figures 30A-30D are schematics of a touch-sensitive display used to illustrate user interfaces for proactively suggesting a paste action, in accordance with some embodiments. Figures 30A-30D is used to illustrate the methods and / or processes of Figure 29.

[0032] Sections 1-11 in the "Additional Descriptions of Embodiments" section describe additional details that supplement those provided in reference to Figures 1A-30D.

[0033] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0034] It will also be understood that, although the terms first, second, etc. are, in some instances, used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact.

[0035] The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms "a", "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "includes," "including," "comprises," and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0036] As used herein, the term "if" is, optionally, construed to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" is, optionally, construed to mean "upon determining" or "in response to determining" or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]," depending on the context.

[0037] The disclosure herein interchangeably refers to detecting a touch input on, at, over, on top of, or substantially within a particular user interface element or a particular portion of a touch-sensitive display. As used herein, a touch input that is detected "at" a particular user interface element could also be detected "on," "over," "on top of," or "substantially within" that same user interface element, depending on the context. In some embodiments and as discussed in more detail below, desired sensitivity levels for detecting touch inputs are configured by a user of an electronic device (e.g., the user could decide (and configure the electronic device to operate) that a touch input should only be detected when the touch input is completely within a user interface element).

[0038] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communications device, such as a mobile telephone, that also contains other functions, such as PDA and / or music player functions. Example embodiments of portable multifunction devices include, without limitation, the IPHONE ®< , IPOD TOUCH ®< , and IPAD ®< devices from APPLE Inc. of Cupertino, California. Other portable electronic devices, such as laptops or tablet computers with touch-sensitive surfaces (e.g., touch-sensitive displays and / or touch pads), are, optionally, used. It should also be understood that, in some embodiments, the device is not a portable communications device, but is a desktop computer with a touch-sensitive surface (e.g., a touch-sensitive display and / or a touch pad).

[0039] In the discussion that follows, an electronic device that includes a display and a touch-sensitive surface is described. It should be understood, however, that the electronic device optionally includes one or more other physical user-interface devices, such as a physical keyboard, a mouse and / or a joystick.

[0040] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an e-mail application, an instant messaging application, a health / fitness application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0041] The various applications that are executed on the device optionally use at least one common physical user-interface device, such as the touch-sensitive surface. One or more functions of the touch-sensitive surface as well as corresponding information displayed on the device are, optionally, adjusted and / or varied from one application to the next and / or within a respective application. In this way, a common physical architecture (such as the touch-sensitive surface) of the device optionally supports the variety of applications with user interfaces that are intuitive and transparent to the user.

[0042] Attention is now directed toward embodiments of portable electronic devices with touch-sensitive displays. FIG. 1A is a block diagram illustrating portable multifunction device 100 (also referred to interchangeably herein as electronic device 100 or device 100) with touch-sensitive display 112 in accordance with some embodiments. Touch-sensitive display 112 is sometimes called a "touch screen" for convenience, and is sometimes known as or called a touch-sensitive display system. Device 100 includes memory 102 (which optionally includes one or more computer-readable storage mediums), controller 120, one or more processing units (CPU's) 122, peripherals interface 118, RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, input / output (I / O) subsystem 106, other input or control devices 116, and external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more intensity sensors 165 for detecting intensity of contacts on device 100 (e.g., a touch- sensitive surface such as touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile outputs on device 100 (e.g., generating tactile outputs on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or a touchpad of device 100). These components optionally communicate over one or more communication buses or signal lines 103.

[0043] As used in the specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch sensitive surface, or to a substitute (proxy) for the force or pressure of a contact on the touch sensitive surface. The intensity of a contact has a range of values that includes at least four distinct values and more typically includes hundreds of distinct values (e.g., at least 256). Intensity of a contact is, optionally, determined (or measured) using various approaches and various sensors or combinations of sensors. For example, one or more force sensors underneath or adjacent to the touch-sensitive surface are, optionally, used to measure force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., a weighted average) to determine an estimated force of a contact. Similarly, a pressure-sensitive tip of a stylus is, optionally, used to determine a pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch- sensitive surface and / or changes thereto, the capacitance of the touch-sensitive surface proximate to the contact and / or changes thereto, and / or the resistance of the touch-sensitive surface proximate to the contact and / or changes thereto are, optionally, used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some implementations, the substitute measurements for contact force or pressure are used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurements). In some implementations, the substitute measurements for contact force or pressure are converted to an estimated force or pressure and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure).

[0044] As used in the specification and claims, the term "tactile output" refers to physical displacement of a device relative to a previous position of the device, physical displacement of a component (e.g., a touch-sensitive surface) of a device relative to another component (e.g., housing) of the device, or displacement of the component relative to a center of mass of the device that will be detected by a user with the user's sense of touch. For example, in situations where the device or the component of the device is in contact with a surface of a user that is sensitive to touch (e.g., a finger, palm, or other part of a user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in physical characteristics of the device or the component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is, optionally, interpreted by the user as a "down click" or "up click" of a physical actuator button. In some cases, a user will feel a tactile sensation such as a "down click" or "up click" even when there is no movement of a physical actuator button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movements. As another example, movement of the touch-sensitive surface is, optionally, interpreted or sensed by the user as "roughness" of the touch-sensitive surface, even when there is no change in smoothness of the touch-sensitive surface. While such interpretations of touch by a user will be subject to the individualized sensory perceptions of the user, there are many sensory perceptions of touch that are common to a large majority of users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., an "up click," a "down click," "roughness"), unless otherwise stated, the generated tactile output corresponds to physical displacement of the device or a component thereof that will generate the described sensory perception for a typical (or average) user.

[0045] It should be appreciated that device 100 is only one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. The various components shown in Figure 1A are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0046] Memory 102 optionally includes high-speed random access memory (e.g., DRAM, SRAM, DDR RAM or other random access solid state memory devices) and optionally also includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory 102 optionally includes one or more storage devices remotely located from processor(s) 122. Access to memory 102 by other components of device 100, such as CPU 122 and the peripherals interface 118, is, optionally, controlled by controller 120.

[0047] Peripherals interface 118 can be used to couple input and output peripherals of the device to CPU 122 and memory 102. The one or more processors 122 run or execute various software programs and / or sets of instructions stored in memory 102 to perform various functions for device 100 and to process data.

[0048] In some embodiments, peripherals interface 118, CPU 122, and controller 120 are, optionally, implemented on a single chip, such as chip 104. In some other embodiments, they are, optionally, implemented on separate chips.

[0049] RF (radio frequency) circuitry 108 receives and sends RF signals, also called electromagnetic signals. RF circuitry 108 converts electrical signals to / from electromagnetic signals and communicates with communications networks and other communications devices via the electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, and so forth. RF circuitry 108 optionally communicates with networks, such as the Internet, also referred to as the World Wide Web (WWW), an intranet and / or a wireless network, such as a cellular telephone network, a wireless local area network (LAN) and / or a metropolitan area network (MAN), and other devices by wireless communication. The wireless communication optionally uses any of a plurality of communications standards, protocols and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), near field communication (NFC), wideband code division multiple access (W-CDMA), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, and / or Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.1l a, IEEE 802.1 lb, IEEE 802.1l g and / or IEEE 802.1 ln).

[0050] Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between a user and device 100. Audio circuitry 110 receives audio data from peripherals interface 118, converts the audio data to an electrical signal, and transmits the electrical signal to speaker 111. Speaker 111 converts the electrical signal to human-audible sound waves. Audio circuitry 110 also receives electrical signals converted by microphone 113 from sound waves. Audio circuitry 110 converts the electrical signal to audio data and transmits the audio data to peripherals interface 118 for processing. Audio data is, optionally, retrieved from and / or transmitted to memory 102 and / or RF circuitry 108 by peripherals interface 118. In some embodiments, audio circuitry 110 also includes a headset jack. The headset jack provides an interface between audio circuitry 110 and removable audio input / output peripherals, such as output-only headphones or a headset with both output (e.g., a headphone for one or both ears) and input (e.g., a microphone).

[0051] I / O subsystem 106 connects input / output peripherals on device 100, such as touch screen 112 and other input control devices 116, to peripherals interface 118. I / O subsystem 106 optionally includes display controller 156, optical sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to other input or control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and so forth. In some alternate embodiments, input controller(s) 160 are, optionally, coupled to any (or none) of the following: a keyboard, infrared port, USB port, and a pointer device such as a mouse. The one or more buttons optionally include an up / down button for volume control of speaker 111 and / or microphone 113. The one or more buttons optionally include a push button.

[0052] Touch-sensitive display 112 provides an input interface and an output interface between the device and a user. Display controller 156 receives and / or sends electrical signals from / to touch screen 112. Touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively termed "graphics"). In some embodiments, some or all of the visual output corresponds to user-interface objects.

[0053] Touch screen 112 has a touch-sensitive surface, a sensor or a set of sensors that accepts input from the user based on haptic and / or tactile contact. Touch screen 112 and display controller 156 (along with any associated modules and / or sets of instructions in memory 102) detect contact (and any movement or breaking of the contact) on touch screen 112 and convert the detected contact into interaction with user-interface objects (e.g., one or more soft keys, icons, web pages or images) that are displayed on touch screen 112. In an example embodiment, a point of contact between touch screen 112 and the user corresponds to an area under a finger of the user.

[0054] Touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, or OLED (organic light emitting diode) technology, although other display technologies are used in other embodiments. Touch screen 112 and display controller 156 optionally detect contact and any movement or breaking thereof using any of a plurality of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch screen 112. In an example embodiment, projected mutual capacitance sensing technology is used, such as that found in the IPHONE ®< , IPOD TOUCH ®< , and IPAD ®< from APPLE Inc. of Cupertino, California.

[0055] Touch screen 112 optionally has a video resolution in excess of 400 dpi. In some embodiments, touch screen 112 has a video resolution of at least 600 dpi. In other embodiments, touch screen 112 has a video resolution of at least 1000 dpi. The user optionally makes contact with touch screen 112 using any suitable object or digit, such as a stylus or a finger. In some embodiments, the user interface is designed to work primarily with finger-based contacts and gestures. In some embodiments, the device translates the finger-based input into a precise pointer / cursor position or command for performing the actions desired by the user.

[0056] In some embodiments, in addition to the touch screen, device 100 optionally includes a touchpad (not shown) for activating or deactivating particular functions. In some embodiments, the touchpad is a touch-sensitive area of the device that, unlike the touch screen, does not display visual output. The touchpad is, optionally, a touch-sensitive surface that is separate from touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0057] Device 100 also includes power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with the generation, management and distribution of power in portable devices.

[0058] Device 100 optionally also includes one or more optical sensors 164. Figure 1A shows an optical sensor coupled to optical sensor controller 158 in I / O subsystem 106. Optical sensor 164 optionally includes charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors. Optical sensor 164 receives light from the environment, projected through one or more lenses, and converts the light to data representing an image. In conjunction with imaging module 143 (also called a camera module), optical sensor 164 optionally captures still images or video. In some embodiments, an optical sensor is located on the back of device 100, opposite touch screen 112 on the front of the device, so that the touch-sensitive display is enabled for use as a viewfinder for still and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device so that the user's image is, optionally, obtained for videoconferencing while the user views the other video conference participants on the touch-sensitive display.

[0059] Device 100 optionally also includes one or more contact intensity sensors 165. Figure 1A shows a contact intensity sensor coupled to intensity sensor controller 159 in I / O subsystem 106. Contact intensity sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of a contact on a touch- sensitive surface). Contact intensity sensor 165 receives contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is collocated with, or proximate to, a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on the back of device 100, opposite touch screen 112 which is located on the front of device 100.

[0060] Device 100 optionally also includes one or more proximity sensors 166. Figure 1A shows proximity sensor 166 coupled to peripherals interface 118. Alternately, proximity sensor 166 is coupled to input controller 160 in 1 / 0 subsystem 106. In some embodiments, the proximity sensor turns off and disables touch screen 112 when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).

[0061] Device 100 optionally also includes one or more tactile output generators 167. Figure 1A shows a tactile output generator coupled to haptic feedback controller 161 in I / O subsystem 106. Tactile output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components and / or electromechanical devices that convert energy into linear motion such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component that converts electrical signals into tactile outputs on the device). Contact intensity sensor 165 receives tactile feedback generation instructions from haptic feedback module 133 and generates tactile outputs on device 100 that are capable of being sensed by a user of device 100. In some embodiments, at least one tactile output generator is collocated with, or proximate to, a touch-sensitive surface (e.g., touch- sensitive display system 112) and, optionally, generates a tactile output by moving the touch-sensitive surface vertically (e.g., in / out of a surface of device 100) or laterally (e.g., back and forth in the same plane as a surface of device 100). In some embodiments, at least one tactile output generator sensor is located on the back of device 100, opposite touch-sensitive display 112 which is located on the front of device 100.

[0062] Device 100 optionally also includes one or more accelerometers 168. Figure 1A shows accelerometer 168 coupled to peripherals interface 118. Alternately, accelerometer 168 is, optionally, coupled to an input controller 160 in I / O subsystem 106. In some embodiments, information is displayed on the touch-sensitive display in a portrait view or a landscape view based on an analysis of data received from the one or more accelerometers. Device 100 optionally includes, in addition to accelerometer(s) 168, a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information concerning the location and orientation (e.g., portrait or landscape) of device 100.

[0063] In some embodiments, the software components stored in memory 102 include operating system 126, proactive module 163 (optionally including one or more of application usage data tables 335, trigger condition tables 402, trigger establishing module 163-1, and / or usage data collecting module 163-2), communication module (or set of instructions) 128, contact / motion module (or set of instructions) 130, graphics module (or set of instructions) 132, text input module (or set of instructions) 134, Global Positioning System (GPS) module (or set of instructions) 135, and applications (or sets of instructions) 136. Furthermore, in some embodiments memory 102 stores device / global internal state 157, as shown in Figure 1A. Device / global internal state 157 includes one or more of: active application state, indicating which applications, if any, are currently active; display state, indicating what applications, views or other information occupy various regions of touch-sensitive display 112; sensor state, including information obtained from the device's various sensors and input control devices 116; and location information concerning the device's location and / or attitude (e.g., orientation of the device).

[0064] Operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0065] Communication module 128 facilitates communication with other devices over one or more external ports 124 and also includes various software components for handling data received by RF circuitry 108 and / or external port 124. External port 124 (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) is adapted for coupling directly to other devices or indirectly over a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, or similar to and / or compatible with the 30-pin connector used on some embodiments of IPOD devices from APPLE Inc. In other embodiments, the external port is a multi-pin (e.g., 8-pin) connector that is the same as, or similar to and / or compatible with the 8-pin connector used in LIGHTNING connectors from APPLE Inc.

[0066] Contact / motion module 130 optionally detects contact with touch screen 112 (in conjunction with display controller 156) and other touch sensitive devices (e.g., a touchpad or physical click wheel). Contact / motion module 130 includes various software components for performing various operations related to detection of contact, such as determining if contact has occurred (e.g., detecting a finger-down event), determining an intensity of the contact (e.g., the force or pressure of the contact or a substitute for the force or pressure of the contact), determining if there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger-dragging events), and determining if the contact has ceased (e.g., detecting a finger-up event or a break in contact). Contact / motion module 130 receives contact data from the touch-sensitive surface. Determining movement of the point of contact, which is represented by a series of contact data, optionally includes determining speed (magnitude), velocity (magnitude and direction), and / or an acceleration (a change in magnitude and / or direction) of the point of contact. These operations are, optionally, applied to single contacts (e.g., one finger contacts) or to multiple simultaneous contacts (e.g., "multitouch" / multiple finger contacts). In some embodiments, contact / motion module 130 and display controller 156 detect contact on a touchpad.

[0067] In some embodiments, contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has selected or "clicked" on an affordance). In some embodiments at least a subset of the intensity thresholds are determined in accordance with software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of particular physical actuators and can be adjusted without changing the physical hardware of device 100). For example, a mouse "click" threshold of a trackpad or touch-sensitive display can be set to any of a large range of predefined thresholds values without changing the trackpad or touch-sensitive display hardware. Additionally, in some implementations a user of the device is provided with software settings for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting a plurality of intensity thresholds at once with a system-level click "intensity" parameter).

[0068] Contact / motion module 130 optionally detects a gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of detected contacts). Thus, a gesture is, optionally, detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger-down event followed by detecting a finger-up (liftoff) event at the same position (or substantially the same position) as the finger-down event (e.g., at the position of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger-down event followed by detecting one or more finger-dragging events, and, in some embodiments, subsequently followed by detecting a finger-up (liftoff) event.

[0069] Graphics module 132 includes various known software components for rendering and displaying graphics on touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual property) of graphics that are displayed. As used herein, the term "graphics" includes any object that can be displayed to a user, including without limitation text, web pages, icons (such as user-interface objects including soft keys), digital images, videos, animations and the like.

[0070] In some embodiments, graphics module 132 stores data representing graphics to be used. Each graphic is, optionally, assigned a corresponding code. Graphics module 132 receives, from applications etc., one or more codes specifying graphics to be displayed along with, if necessary, coordinating data and other graphic property data, and then generates screen image data to output to display controller 156.

[0071] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator(s) 167 to produce tactile outputs at one or more locations on device 100 in response to user interactions with device 100.

[0072] Text input module 134, which is, optionally, a component of graphics module 132, provides soft keyboards for entering text in various applications (e.g., contacts module 137, e-mail client module 140, IM module 141, browser module 147, and any other application that needs text input).

[0073] GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to telephone 138 for use in location-based dialing, to camera 143 as picture / video metadata, and to applications that provide location-based services such as weather widgets, local yellow page widgets, and map / navigation widgets).

[0074] Applications ("apps") 136 optionally include the following modules (or sets of instructions), or a subset or superset thereof: contacts module 137 (sometimes called an address book or contact list); telephone module 138; video conferencing module 139; e-mail client module 140; instant messaging (IM) module 141; health module 142; camera module 143 for still and / or video images; image management module 144; browser module 147; calendar module 148; widget modules 149, which optionally include one or more of: weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widgets 149-6; search module 151; video and music player module 152, which is, optionally, made up of a video player module and a music player module; notes module 153; map module 154; and / or online video module 155.

[0075] Examples of other applications 136 that are, optionally, stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, website creation applications, disk authoring applications, spreadsheet applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, widget creator module for making user-created widgets 149-6, and voice replication.

[0076] In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, contacts module 137 is, optionally, used to manage an address book or contact list (e.g., stored in contacts module 137 in memory 102 or memory 370), including: adding name(s) to the address book; deleting name(s) from the address book; associating telephone number(s), e-mail address(es), physical address(es) or other information with a name; associating an image with a name; categorizing and sorting names; providing telephone numbers or e-mail addresses to initiate and / or facilitate communications by telephone module 138, video conference module 139, e-mail client module 140, or IM module 141; and so forth.

[0077] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, telephone module 138 is, optionally, used to enter a sequence of characters corresponding to a telephone number, access one or more telephone numbers in address book 137, modify a telephone number that has been entered, dial a respective telephone number, conduct a conversation and disconnect or hang up when the conversation is completed. As noted above, the wireless communication optionally uses any of a plurality of communications standards, protocols and technologies.

[0078] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, videoconferencing module 139 includes executable instructions to initiate, conduct, and terminate a video conference between a user and one or more other participants in accordance with user instructions.

[0079] In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, e-mail client module 140 includes executable instructions to create, send, receive, and manage e-mail in response to user instructions. In conjunction with image management module 144, e-mail client module 140 makes it very easy to create and send e-mails with still or video images taken with camera module 143.

[0080] In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the instant messaging module 141 includes executable instructions to enter a sequence of characters corresponding to an instant message, to modify previously entered characters, to transmit a respective instant message (for example, using a Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for telephony-based instant messages or using XMPP, SIMPLE, or IMPS for Internet-based instant messages), to receive instant messages and to view received instant messages. In some embodiments, transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments as are supported in an MMS and / or an Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both telephony-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0081] In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and video and music player module 146, health module 142 includes executable instructions to create workouts (e.g., with time, distance, and / or calorie burning goals), communicate with workout sensors (sports devices such as a watch or a pedometer), receive workout sensor data, calibrate sensors used to monitor a workout, select and play music for a workout, and display, store and transmit workout data.

[0082] In conjunction with touch screen 112, display controller 156, optical sensor(s) 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions to capture still images or video (including a video stream) and store them into memory 102, modify characteristics of a still image or video, or delete a still image or video from memory 102.

[0083] In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions to arrange, modify (e.g., edit), or otherwise manipulate, label, delete, present (e.g., in a digital slide show or album), and store still and / or video images.

[0084] In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions to browse the Internet in accordance with user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0085] In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, e-mail client module 140, and browser module 147, calendar module 148 includes executable instructions to create, display, modify, and store calendars and data associated with calendars (e.g., calendar entries, to do lists, etc.) in accordance with user instructions.

[0086] In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, widget modules 149 are mini-applications that are, optionally, downloaded and used by a user (e.g., weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or created by the user (e.g., user-created widget 149-6). In some embodiments, a widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, a widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widgets).

[0087] In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, a widget creator module (not pictured) is, optionally, used by a user to create widgets (e.g., turning a user-specified portion of a web page into a widget).

[0088] In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, search module 151 includes executable instructions to search for text, music, sound, image, video, and / or other files in memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions. In some embodiments, search module 151 further includes executable instructions for displaying a search entry portion and a predictions portion (e.g., search entry portion 920 and predictions portion 930, Figure 9B, and discussed in more detail below in reference to Figures 6A-9C). In some embodiments, the search module 151, in conjunction with proactive module 163, also populates, prior to receiving any user input at the search entry portion, the predictions portion with affordances for suggested or predicted people, actions within applications, applications, nearby places, and / or news articles (as discussed in more detail below in reference to Figures 3A-9C).

[0089] In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow the user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions to display, present or otherwise play back videos (e.g., on touch screen 112 or on an external, connected display via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an IPOD from APPLE Inc.

[0090] In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, notes module 153 includes executable instructions to create and manage notes, to do lists, and the like in accordance with user instructions.

[0091] In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 is, optionally, used to receive, display, modify, and store maps and data associated with maps (e.g., driving directions; data on stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0092] In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, e-mail client module 140, and browser module 147, online video module 155 includes instructions that allow the user to access, browse, receive (e.g., by streaming and / or download), play back (e.g., on the touch screen or on an external, connected display via external port 124), send an e-mail with a link to a particular online video, and otherwise manage online videos in one or more file formats, such as H.264. In some embodiments, instant messaging module 141, rather than e-mail client module 140, is used to send a link to a particular online video.

[0093] As pictured in FIG. 1A, portable multifunction device 100 also includes a proactive module 163 for proactively identifying and surfacing relevant content (e.g., surfacing a user interface object corresponding to an action within an application (e.g., a UI object for playing a playlist within a music app) to a lock screen or within a search interface). Proactive module 163 optionally includes the following modules (or sets of instructions), or a subset or superset thereof: application usage tables 335; trigger condition tables 402; trigger establishing module 163-1; usage data collecting module 163-2; proactive suggestions module 163-3; and (voice communication) content extraction module 163-4.

[0094] In conjunction with applications 136, GPS module 135, operating system 126, I / O subsystem 106, RF circuitry 108, external portion 124, proximity sensor 166, audio circuitry 110, accelerometers 168, speaker 111, microphone 113, and peripherals interface 118, the application usage tables 335 and usage data collecting module 163-2 receive (e.g., from the components of device 100 identified above, Figure 1A) and / or store application usage data. In some embodiments, the application usage is reported to the usage data collecting module 163-2 and then stored in the application usage tables 335. In some embodiments, application usage data includes all (or the most important, relevant, or predictive) contextual usage information corresponding to a user's use of a particular application 136. In some embodiments, each particular application stores usage data while the user is interacting with the application and that usage data is then reported to the application usage data tables 335 for storage (e.g., usage data 193 for a particular application 136-1, Figure 3B, includes all sensor readings, in-application actions performed, device coupling info, etc., and this usage data 193 gets sent to an application usage table 335 for storage as a record within the table). For example, while the user interacts with browser module 147, application usage data receives and stores all contextual usage information, including current GPS coordinates of the device 100 (e.g., as determined by GPS module 135), motion data (e.g., as determined by accelerometers 168), ambient light data (e.g., as determined by optical sensor 164), and in-application actions performed by the user within the browser module 147 (e.g., URLs visited, amount of time spent visiting each page), among other sensor data and other contextual usage information received and stored by the application usage tables 335. Additional information regarding application usage tables 335 is provided below in reference to Figures 3A-3B. As discussed below in reference to Figure 5, the application usage data, in some embodiments, is stored remotely (e.g., at one or more servers 502, Figure 5).

[0095] Trigger condition tables 402 and trigger establishing module 163-1 receive and / or store trigger conditions that are established based on the usage data stored in application usage tables 335. In some embodiments, trigger establishing module 163-1 mines and analyzes the data stored in the application usage tables 335 in order to identify patterns. For example, if the application usage data indicates that the user always launches a music application between 3:00 PM - 4:00 PM daily, then the trigger establishing module 163-1 creates and stores a trigger condition in the trigger condition tables 402 that, when satisfied (e.g., when a current time of day is within a predetermined amount of time of 3:00 PM - 4:00 PM), causes the device 100 to launch the music application (or at least provide an indication to the user that the music application is available (e.g., display a UI object on the lock screen, the UI object allowing the user to easily access the music application). Additional information regarding trigger condition tables 402 is provided below in reference to Figures 4A-4B. As discussed below in reference to Figure 5, in some embodiments, the identification of patterns and establishing of trigger conditions based on the identified patterns is done at a remote server (e.g., at one or more servers 502, Figure 5).

[0096] The proactive suggestions module 163-3 works in conjunction with other components of the device 100 to proactively provide content to the user for use in a variety of different applications available on the electronic device. For example, the proactive suggestions module 163-3 provides suggested search queries and other suggested content for inclusion in a search interface (e.g. as discussed below in reference to Figures 10A-10C), provides information that helps users to locate their parked vehicles (e.g., as discussed below in reference to Figure 14), provides information about nearby points of interest (e.g., as discussed below in reference to Figures 16A-16B), provides content items that have been extracted from speech provided during voice communications (e.g., as discussed below in reference to Figures 18A-18B), and helps to provide numerous other suggestions (e.g., as discussed below in reference to Figures 20, 21A-21B, 24A-24B, 26A-26B, 27, and 29) that help users to efficiently located desired content with a minimal number of inputs (e.g., without having to search for that content, the proactive suggestions module 163-3 helps to ensure that the content is provided at an appropriate time for selection by the user).

[0097] The (voice communication) content extraction module 163-4 works in conjunction with other components of device 100 to identify speech that relates to a new content item and to extract new content items from voice communications (e.g., contact information, information about events, and information about locations, as discussed in more detail below in reference to Figures 18A-18B and 20).

[0098] Each of the above-identified modules and applications correspond to a set of executable instructions for performing one or more functions described above and the methods described in this application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these modules are, optionally, combined or otherwise re-arranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the modules and data structures identified above. Furthermore, memory 102 optionally stores additional modules and data structures not described above.

[0099] In some embodiments, device 100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touchpad. By using a touch screen and / or a touchpad as the primary input control device for operation of device 100, the number of physical input control devices (such as push buttons, dials, and the like) on device 100 is, optionally, reduced.

[0100] The predefined set of functions that are performed exclusively through a touch screen and / or a touchpad optionally include navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates device 100 to a main, home, or root menu from any user interface that is displayed on device 100. In such embodiments, a "menu button" is implemented using a touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device instead of a touchpad.

[0101] FIG. 1B is a block diagram illustrating example components for event handling in accordance with some embodiments. In some embodiments, memory 102 (in FIG. 1A) includes event sorter 170 (e.g., in operating system 126) and a respective application 136-1 selected from among the applications 136 of portable multifunction device 100 (FIG. 1A) (e.g., any of the aforementioned applications stored in memory 102 with applications 136).

[0102] Event sorter 170 receives event information and determines the application 136-1 and application view 191 of application 136-1 to which to deliver the event information. Event sorter 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates the current application view(s) displayed on touch sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event sorter 170 to determine which application(s) is (are) currently active, and application internal state 192 is used by event sorter 170 to determine application views 191 to which to deliver event information.

[0103] In some embodiments, application internal state 192 includes additional information, such as one or more of: resume information to be used when application 136-1 resumes execution, user interface state information that indicates information being displayed or that is ready for display by application 136-1, a state queue for enabling the user to go back to a prior state or view of application 136-1, and a redo / undo queue of previous actions taken by the user.

[0104] Event monitor 171 receives event information from peripherals interface 118. Event information includes information about a sub-event (e.g., a user touch on touch-sensitive display 112, as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or a sensor, such as proximity sensor 166, accelerometer(s) 168, and / or microphone 113 (through audio circuitry 110). Information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

[0105] In some embodiments, event monitor 171 sends requests to the peripherals interface 118 at predetermined intervals. In response, peripherals interface 118 transmits event information. In other embodiments, peripherals interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or for more than a predetermined duration).

[0106] In some embodiments, event sorter 170 also includes a hit view determination module 172 and / or an active event recognizer determination module 173.

[0107] Hit view determination module 172 provides software procedures for determining where a sub-event has taken place within one or more views, when touch sensitive display 112 displays more than one view. Views are made up of controls and other elements that a user can see on the display.

[0108] Another aspect of the user interface associated with an application is a set of views, sometimes herein called application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of a respective application) in which a touch is detected optionally correspond to programmatic levels within a programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is, optionally, called the hit view, and the set of events that are recognized as proper inputs are, optionally, determined based, at least in part, on the hit view of the initial touch that begins a touch-based gesture.

[0109] Hit view determination module 172 receives information related to sub-events of a touch-based gesture. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies a hit view as the lowest view in the hierarchy which should handle the sub-event. In most circumstances, the hit view is the lowest level view in which an initiating sub-event occurs (e.g., the first sub-event in the sequence of sub-events that form an event or potential event). Once the hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0110] Active event recognizer determination module 173 determines which view or views within a view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of a sub-event are actively involved views, and therefore determines that all actively involved views should receive a particular sequence of sub-events. In other embodiments, even if touch sub-events were entirely confined to the area associated with one particular view, views higher in the hierarchy would still remain as actively involved views.

[0111] Event dispatcher module 174 dispatches the event information to an event recognizer (e.g., event recognizer 180). In embodiments including active event recognizer determination module 173, event dispatcher module 174 delivers the event information to an event recognizer determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores in an event queue the event information, which is retrieved by a respective event receiver 182.

[0112] In some embodiments, operating system 126 includes event sorter 170. Alternatively, application 136-1 includes event sorter 170. In yet other embodiments, event sorter 170 is a stand-alone module, or a part of another module stored in memory 102, such as contact / motion module 130.

[0113] In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a respective view of the application's user interface. Each application view 191 of the application 136-1 includes one or more event recognizers 180. Typically, a respective application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of event recognizers 180 are part of a separate module, such as a user interface kit (not shown) or a higher level object from which application 136-1 inherits methods and other properties. In some embodiments, a respective event handler 190 includes one or more of: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event sorter 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177 or GUI updater 178 to update the application internal state 192. Alternatively, one or more of the application views 191 includes one or more respective event handlers 190. Also, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in a respective application view 191.

[0114] A respective event recognizer 180 receives event information (e.g., event data 179) from event sorter 170, and identifies an event from the event information. Event recognizer 180 includes event receiver 182 and event comparator 184. In some embodiments, event recognizer 180 also includes at least a subset of: metadata 183, and event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0115] Event receiver 182 receives event information from event sorter 170. The event information includes information about a sub-event, for example, a touch or a touch movement. Depending on the sub-event, the event information also includes additional information, such as location of the sub-event. When the sub-event concerns motion of a touch, the event information optionally also includes speed and direction of the sub-event. In some embodiments, events include rotation of the device from one orientation to another (e.g., from portrait to landscape, or vice versa), and the event information includes corresponding information about the current orientation (also called device attitude) of the device.

[0116] Event comparator 184 compares the event information to predefined event or sub-event definitions and, based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definitions 186. Event definitions 186 contain definitions of events (e.g., predefined sequences of sub-events), for example, event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in an event 187 include, for example, touch begin, touch end, touch movement, touch cancellation, and multiple touching. In one example, the definition for event 1 (187-1) is a double tap on a displayed object. The double tap, for example, comprises a first touch (touch begin) on the displayed object for a predetermined phase, a first lift-off (touch end) for a predetermined phase, a second touch (touch begin) on the displayed object for a predetermined phase, and a second lift-off (touch end) for a predetermined phase. In another example, the definition for event 2 (187-2) is a dragging on a displayed object. The dragging, for example, comprises a touch (or contact) on the displayed object for a predetermined phase, a movement of the touch across touch-sensitive display 112, and lift-off of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0117] In some embodiments, event definition 186 includes a definition of an event for a respective user-interface object. In some embodiments, event comparator 184 performs a hit test to determine which user-interface object is associated with a sub-event. For example, in an application view in which three user-interface objects are displayed on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user-interface objects is associated with the touch (sub-event). If each displayed object is associated with a respective event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with the sub-event and the object triggering the hit test.

[0118] In some embodiments, the definition for a respective event 187 also includes delayed actions that delay delivery of the event information until after it has been determined whether the sequence of sub-events does or does not correspond to the event recognizer's event type.

[0119] When a respective event recognizer 180 determines that the series of sub-events do not match any of the events in event definitions 186, the respective event recognizer 180 enters an event impossible, event failed, or event ended state, after which it disregards subsequent sub-events of the touch-based gesture. In this situation, other event recognizers, if any remain active for the hit view, continue to track and process sub-events of an ongoing touch-based gesture.

[0120] In some embodiments, a respective event recognizer 180 includes metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively involved event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact, or are enabled to interact, with one another. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to varying levels in the view or programmatic hierarchy.

[0121] In some embodiments, a respective event recognizer 180 activates event handler 190 associated with an event when one or more particular sub-events of an event are recognized. In some embodiments, a respective event recognizer 180 delivers event information associated with the event to event handler 190. Activating an event handler 190 is distinct from sending (and deferred sending) sub-events to a respective hit view. In some embodiments, event recognizer 180 throws a flag associated with the recognized event, and event handler 190 associated with the flag catches the flag and performs a predefined process.

[0122] In some embodiments, event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver event information to event handlers associated with the series of sub-events or to actively involved views. Event handlers associated with the series of sub-events or with actively involved views receive the event information and perform a predetermined process.

[0123] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates the telephone number used in contacts module 137, or stores a video file used in video and music player module 145. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 176 creates a new user-interface object or updates the position of a user-interface object. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends it to graphics module 132 for display on a touch-sensitive display.

[0124] In some embodiments, event handler(s) 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of a respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0125] In some embodiments, each particular application 136-1 stores usage data while the user is interacting with the application and that usage data is then reported to the application usage data tables 335 for storage (e.g., usage data 193 for a particular application 136-1, Figure 3B, includes all sensor readings, in-application actions performed, device coupling info, etc., and this usage data 193 gets sent to a respective application usage table 335 for the particular application for storage as a record within the table). In some embodiments, usage data 193 stores data as reported by usage data collecting module 163-2 while the particular application 136-1 is in use (e.g., the user is actively interactive with the particular application 136-1).

[0126] It shall be understood that the foregoing discussion regarding event handling of user touches on touch-sensitive displays also applies to other forms of user inputs to operate multifunction devices 100 with input-devices, not all of which are initiated on touch screens. For example, mouse movement and mouse button presses, optionally coordinated with single or multiple keyboard presses or holds; contact movements such as taps, drags, scrolls, etc., on touch-pads; pen stylus inputs; movement of the device; oral instructions; detected eye movements; biometric inputs; and / or any combination thereof is optionally utilized as inputs corresponding to sub-events which define an event to be recognized.

[0127] Figure 1C is a schematic of a portable multifunction device (e.g., portable multifunction device 100) having a touch-sensitive display (e.g., touch screen 112) in accordance with some embodiments. In this embodiment, as well as others described below, a user can select one or more of the graphics by making a gesture on the screen, for example, with one or more fingers or one or more styluses. In some embodiments, selection of one or more graphics occurs when the user breaks contact with the one or more graphics (e.g., by lifting a finger off of the screen). In some embodiments, the gesture optionally includes one or more tap gestures (e.g., a sequence of touches on the screen followed by liftoffs), one or more swipe gestures (continuous contact during the gesture along the surface of the screen, e.g., from left to right, right to left, upward and / or downward), and / or a rolling of a finger (e.g., from right to left, left to right, upward and / or downward) that has made contact with device 100. In some implementations or circumstances, inadvertent contact with a graphic does not select the graphic. For example, a swipe gesture that sweeps over an application affordance (e.g., an icon) optionally does not launch (e.g., open) the corresponding application when the gesture for launching the application is a tap gesture.

[0128] Device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As described previously, menu button 204 is, optionally, used to navigate to any application 136 in a set of applications that are, optionally executed on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on touch screen 112.

[0129] In one embodiment, device 100 includes touch screen 112, menu button 204, push button 206 for powering the device on / off and locking the device, volume adjustment button(s) 208, Subscriber Identity Module (SIM) card slot 210, head set jack 212, and docking / charging external port 124. Push button 206 is, optionally, used to turn the power on / off on the device by depressing the button and holding the button in the depressed state for a predefined time interval; to lock the device by depressing the button and releasing the button before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlock process. In an alternative embodiment, device 100 also accepts verbal input for activation or deactivation of some functions through microphone 113. Device 100 also, optionally, includes one or more contact intensity sensors 165 for detecting intensity of contacts on touch screen 112 and / or one or more tactile output generators 167 for generating tactile outputs for a user of device 100.

[0130] Figure 1D is a schematic used to illustrate a user interface on a device (e.g., device 100, FIG. 1A) with a touch-sensitive surface 195 (e.g., a tablet or touchpad) that is separate from the display 194 (e.g., touch screen 112). In some embodiments, touch-sensitive surface 195 includes one or more contact intensity sensors (e.g., one or more of contact intensity sensor(s) 359) for detecting intensity of contacts on touch-sensitive surface 195 and / or one or more tactile output generator(s) 357 for generating tactile outputs for a user of touch-sensitive surface 195.

[0131] Although some of the examples which follow will be given with reference to inputs on touch screen 112 (where the touch sensitive surface and the display are combined), in some embodiments, the device detects inputs on a touch-sensitive surface that is separate from the display, as shown in FIG. 1D. In some embodiments the touch sensitive surface (e.g., 195 in FIG. 1D) has a primary axis (e.g., 199 in FIG. 1D) that corresponds to a primary axis (e.g., 198 in FIG. 1D) on the display (e.g., 194). In accordance with these embodiments, the device detects contacts (e.g., 197-1 and 197-2 in FIG. 1D) with the touch-sensitive surface 195 at locations that correspond to respective locations on the display (e.g., in FIG. 1D, 197-1 corresponds to 196-1 and 197-2 corresponds to 196-2). In this way, user inputs (e.g., contacts 197-1 and 197-2, and movements thereof) detected by the device on the touch-sensitive surface (e.g., 195 in FIG. 1D) are used by the device to manipulate the user interface on the display (e.g., 194 in FIG. 1D) of the multifunction device when the touch-sensitive surface is separate from the display. It should be understood that similar methods are, optionally, used for other user interfaces described herein.

[0132] Additionally, while the following examples are given primarily with reference to finger inputs (e.g., finger contacts, finger tap gestures, finger swipe gestures), it should be understood that, in some embodiments, one or more of the finger inputs are replaced with input from another input device (e.g., a mouse based input or stylus input). For example, a swipe gesture is, optionally, replaced with a mouse click (e.g., instead of a contact) followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is, optionally, replaced with a mouse click while the cursor is located over the location of the tap gesture (e.g., instead of detection of the contact followed by ceasing to detect the contact). Similarly, when multiple user inputs are simultaneously detected, it should be understood that multiple computer mice are, optionally, used simultaneously, or mouse and finger contacts are, optionally, used simultaneously.

[0133] As used herein, the term "focus selector" refers to an input element that indicates a current part of a user interface with which a user is interacting. In some implementations that include a cursor or other location marker, the cursor acts as a "focus selector," so that when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touch-sensitive surface 195 in FIG. 1D (touch-sensitive surface 195, in some embodiments, is a touchpad)) while the cursor is over a particular user interface element (e.g., a button, window, slider or other user interface element), the particular user interface element is adjusted in accordance with the detected input. In some implementations that include a touch-screen display (e.g., touch- sensitive display system 112 in FIG. 1A or touch screen 112) that enables direct interaction with user interface elements on the touch-screen display, a detected contact on the touch-screen acts as a "focus selector," so that when an input (e.g., a press input by the contact) is detected on the touch-screen display at a location of a particular user interface element (e.g., a button, window, slider or other user interface element), the particular user interface element is adjusted in accordance with the detected input. In some implementations focus is moved from one region of a user interface to another region of the user interface without corresponding movement of a cursor or movement of a contact on a touch-screen display (e.g., by using a tab key or arrow keys to move focus from one button to another button); in these implementations, the focus selector moves in accordance with movement of focus between different regions of the user interface. Without regard to the specific form taken by the focus selector, the focus selector is generally the user interface element (or contact on a touch-screen display) that is controlled by the user so as to communicate the user's intended interaction with the user interface (e.g., by indicating, to the device, the element of the user interface with which the user is intending to interact). For example, the location of a focus selector (e.g., a cursor, a contact or a selection box) over a respective button while a press input is detected on the touch-sensitive surface (e.g., a touchpad or touch-sensitive display) will indicate that the user is intending to activate the respective button (as opposed to other user interface elements shown on a display of the device).

[0134] Figure 4C illustrates example electronic devices that are in communication with display 194 and touch-sensitive surface 195. For at least a subset of the electronic devices, display 194 and / or touch-sensitive surface 195 is integrated into the electronic device in accordance with some embodiments. While the examples described in greater detail below are described with reference to a touch-sensitive surface 195 and a display 194 that are in communication with an electronic device (e.g., portable multifunction device 100 in Figures 1A-1B), it should be understood that in accordance with some embodiments, the touch-sensitive surface and / or the display are integrated with the electronic device, while in other embodiments one or more of the touch-sensitive surface and the display are separate from the electronic device. Additionally, in some embodiments the electronic device has an integrated display and / or an integrated touch-sensitive surface and is in communication with one or more additional displays and / or touch-sensitive surfaces that are separate from the electronic device.

[0135] In some embodiments, all of the operations described below with reference to Figures 6A-6B, 7A-7B, 8A-8B, 9A-9D, 10A-10C, 11A-11J, 12, 13A-13B, 14, 15A-15B, 16A-16B, 17A-17E, 18A-18B, 19A-19F, 20, 21A-21B, 22A-22C, 23A-23O, 24A-24B, 25A- 25J, 26A-26B, 27, 28, 29, 30A-30D are performed on a single electronic device with user interface navigation logic 480 (e.g., Computing Device A described below with reference to Figure 4C). However, it should be understood that frequently multiple different electronic devices are linked together to perform the operations described below with reference to Figures 6A-6B, 7A-7B, 8A-8B, 9A-9D, 10A-10C, 11A-11J, 12, 13A-13B, 14, 15A-15B, 16A-16B, 17A-17E, 18A-18B, 19A-19F, 20, 21A-21B, 22A-22C, 23A-23O, 24A-24B, 25A-25J, 26A-26B, 27, 28, 29, 30A-30D(e.g., an electronic device with user interface navigation logic 480 communicates with a separate electronic device with a display 194 and / or a separate electronic device with a touch-sensitive surface 195). In any of these embodiments, the electronic device that is described below with reference to Figures 6A-6B, 7A-7B, 8A-8B, 9A-9D, 10A-10C, 11A-11J, 12, 13A-13B, 14, 15A-15B, 16A-16B, 17A-17E, 18A-18B, 19A-19F, 20, 21A-21B, 22A-22C, 23A-23O, 24A-24B, 25A- 25J, 26A-26B, 27, 28, 29, 30A-30D is the electronic device (or devices) that contain(s) the user interface navigation logic 480. Additionally, it should be understood that the user interface navigation logic 480 could be divided between a plurality of distinct modules or electronic devices in various embodiments; however, for the purposes of the description herein, the user interface navigation logic 480 will be primarily referred to as residing in a single electronic device so as not to unnecessarily obscure other aspects of the embodiments.

[0136] In some embodiments, the user interface navigation logic 480 includes one or more modules (e.g., one or more event handlers 190, including one or more object updaters 177 and one or more GUI updaters 178 as described in greater detail above with reference to Figure 1C) that receive interpreted inputs and, in response to these interpreted inputs, generate instructions for updating a graphical user interface in accordance with the interpreted inputs which are subsequently used to update the graphical user interface on a display. In some embodiments, an interpreted input is an input that has been detected (e.g., by a contact motion 130 in Figures 1A-1B and 3), recognized (e.g., by an event recognizer 180 in Figure 1C) and / or prioritized (e.g., by event sorter 170 in Figure 1C). In some embodiments, the interpreted inputs are generated by modules at the electronic device (e.g., the electronic device receives raw contact input data so as to identify gestures from the raw contact input data). In some embodiments, some or all of the interpreted inputs are received by the electronic device as interpreted inputs (e.g., an electronic device that includes the touch-sensitive surface 195 processes raw contact input data so as to identify gestures from the raw contact input data and sends information indicative of the gestures to the electronic device that includes the user interface navigation logic 480).

[0137] In some embodiments, both the display 194 and the touch-sensitive surface 195 are integrated with the electronic device (e.g., Computing Device A in Figure 4C) that contains the user interface navigation logic 480. For example, the electronic device may be a desktop computer or laptop computer with an integrated display (e.g., 340 in Figure 3) and touchpad (e.g., 355 in Figure 3). As another example, the electronic device may be a portable multifunction device 100 (e.g., a smartphone, PDA, tablet computer, etc.) with a touch screen (e.g., 112 in Figure 2).

[0138] In some embodiments, the touch-sensitive surface 195 is integrated with the electronic device while the display 194 is not integrated with the electronic device (e.g., Computing Device B in Figure 4C) that contains the user interface navigation logic 480. For example, the electronic device may be a device (e.g., a desktop computer or laptop computer) with an integrated touchpad (e.g., 355 in Figure 3) connected (via wired or wireless connection) to a separate display (e.g., a computer monitor, television, etc.). As another example, the electronic device may be a portable multifunction device 100 (e.g., a smartphone, PDA, tablet computer, etc.) with a touch screen (e.g., 112 in Figure 2) connected (via wired or wireless connection) to a separate display (e.g., a computer monitor, television, etc.).

[0139] In some embodiments, the display 194 is integrated with the electronic device while the touch-sensitive surface 195 is not integrated with the electronic device (e.g., Computing Device C in Figure 4C) that contains the user interface navigation logic 480. For example, the electronic device may be a device (e.g., a desktop computer, laptop computer, television with integrated set-top box) with an integrated display (e.g., 340 in Figure 3) connected (via wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, a portable multifunction device, etc.). As another example, the electronic device may be a portable multifunction device 100 (e.g., a smartphone, PDA, tablet computer, etc.) with a touch screen (e.g., 112 in Figure 2) connected (via wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, another portable multifunction device with a touch screen serving as a remote touchpad, etc.).

[0140] In some embodiments, neither the display 194 nor the touch-sensitive surface 195 is integrated with the electronic device (e.g., Computing Device D in Figure 4C) that contains the user interface navigation logic 480. For example, the electronic device may be a stand-alone electronic device (e.g., a desktop computer, laptop computer, console, set-top box, etc.) connected (via wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, a portable multifunction device, etc.) and a separate display (e.g., a computer monitor, television, etc.). As another example, the electronic device may be a portable multifunction device 100 (e.g., a smartphone, PDA, tablet computer, etc.) with a touch screen (e.g., 112 in Figure 2) connected (via wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, another portable multifunction device with a touch screen serving as a remote touchpad, etc.).

[0141] In some embodiments, the computing device has an integrated audio system. In some embodiments, the computing device is in communication with an audio system that is separate from the computing device. In some embodiments, the audio system (e.g., an audio system integrated in a television unit) is integrated with a separate display 194. In some embodiments, the audio system (e.g., a stereo system) is a stand-alone system that is separate from the computing device and the display 194.

[0142] Attention is now directed towards user interface ("UI") embodiments and associated processes that may be implemented on an electronic device with a display and a touch-sensitive surface, such as device 100.

[0143] Figure 2 is a schematic of a touch screen used to illustrate a user interface for a menu of applications, in accordance with some embodiments. Similar user interfaces are, optionally, implemented on device 100 (FIG. 1A). In some embodiments, the user interface displayed on the touch screen 112 includes the following elements, or a subset or superset thereof: Signal strength indicator(s) 202 for wireless communication(s), such as cellular and Wi-Fi signals; Time 203; Bluetooth indicator 205; Battery status indicator 206; Tray 209 with icons for frequently used applications, such as: ∘ Icon 216 for telephone module 138, labeled "Phone," which optionally includes an indicator 214 of the number of missed calls or voicemail messages; ∘ Icon 218 for e-mail client module 140, labeled "Mail," which optionally includes an indicator 210 of the number of unread e-mails; ∘ Icon 220 for browser module 147, labeled "Browser;" and ∘ Icon 222 for video and music player module 152, also referred to as IPOD (trademark of APPLE Inc.) module 152, labeled "iPod;" and Icons for other applications, such as: ∘ Icon 224 for IM module 141, labeled "Messages;" ∘ Icon 226 for calendar module 148, labeled "Calendar;" ∘ Icon 228 for image management module 144, labeled "Photos;" ∘ Icon 230 for camera module 143, labeled "Camera;" ∘ Icon 232 for online video module 155, labeled "Online Video" ∘ Icon 234 for stocks widget 149-2, labeled "Stocks;" ∘ Icon 236 for map module 154, labeled "Maps;" ∘ Icon 238 for weather widget 149-1, labeled "Weather;" ∘ Icon 240 for alarm clock widget 149-4, labeled "Clock;" ∘ Icon 242 for health module 142, labeled "Health;" ∘ Icon 244 for notes module 153, labeled "Notes;" ∘ Icon 246 for a settings application or module, which provides access to settings for device 100 and its various applications; and ∘ Other icons for additional applications, such as App Store, iTunes, Voice Memos, and Utilities.

[0144] It should be noted that the icon labels illustrated in Figure 2 are merely examples. Other labels are, optionally, used for various application icons. For example, icon 242 for health module 142 is alternatively labeled "Fitness Support," "Workout," "Workout Support," "Exercise," "Exercise Support," or "Fitness." In some embodiments, a label for a respective application icon includes a name of an application corresponding to the respective application icon. In some embodiments, a label for a particular application icon is distinct from a name of an application corresponding to the particular application icon.

[0145] Figures 3A-3B are block diagrams illustrating data structures for storing application usage data, in accordance with some embodiments. As shown in Figure 3A, application usage data tables 335 include a collection of data structures 335, optionally implemented as a collection of tables for each application installed on the device 100, that each store usage data associated with a corresponding respective application installed on the electronic device (e.g., application 1 usage data table 335-1 stores usage data for application 1 and application usage data table 335-2 stores usage data for application 2). In some embodiments, each table (e.g., table 335-1, 335-2, 335-3 ... 335-N) in the collection of application usage data tables stores usage data for more than one application installed on the electronic device (e.g. table 335-1 stores usage data for related applications that are each provided by a common application developer or application vendor, for efficient storage of potentially related data).

[0146] In some embodiments, one or more application usage data tables 335 (e.g., application 1 usage data table 335-1) are used for storing usage data associated with applications installed on the device 100. As illustrated in Figure 3B, application 1 usage data table 335-1 contains a number of usage entries. In some embodiments, the usage entries are stored in individual records 340-1 through 340-z and, optionally, a header 340-0. Header 340-0, in some embodiments, contains a brief description of each field of information (e.g., each field associated with each of the records) stored within the table. For example, Header 340-0 indicates that each record 340-1 through 340-z includes an entry ID that uniquely identifies the usage entry. In some embodiments, application 1 usage data table 335-1 includes additional fields in addition to the entry ID field, such as a timestamp field that identifies when the usage entry was created and / or stored in the table 335-1 and a related usage entries field that identifies related usage entries that may be stored in other application usage data tables 335.

[0147] In some embodiments, each record within the application 1 usage data table 335-1 contains one or more usage entries containing usage data collected while a user interacts with application 1 (e.g., every time the user launches application 1, a new usage entry is created to store collected usage data). In some embodiments, each usage entry in the table stores the following information and data structures, or a subset or superset thereof: information identifying in-app actions performed (e.g., in-app actions performed 340-1(a)) by the user within the application (in some embodiments, these actions are reported to the device by the application), for example the application reports to the usage data collecting module 163-2 that the user played a particular song within a particular playlist; information identifying other actions performed (e.g., other actions performed 340-1(b)) by the user within other applications (e.g., system-level applications), such as providing verbal instructions to a virtual assistant application or conducting a search for an item of information within a search application (e.g., search module 151, Figure 1A); sensor data (e.g., usage data 340-1(c)) that includes data collected by the sensors on the device 100 while the user is interacting with the application associated with the usage entry, optionally including: ∘ time of day (e.g., time of day 340-1(d)) information; ∘ location data (e.g., location data 340-1(e)) identifying a current location at the time when the user launched the application and other locations visited by the user while executing the application (e.g., as reported by GPS module 132); ∘ other sensor data (e.g., other sensor data 340-1(f)) collected while the user is interacting with the application (such as ambient light data, altitude data, pressure readings, motion data, etc.); device coupling information (e.g., device coupling info 340-1(g)) identifying external devices coupled with the device 100 while the user is interacting with the application (e.g., an example external device could be a pair of headphones connected to the headphone jack or another example device could be a device connected via BLUETOOTH (e.g., speakers in a motor vehicle or a hands-free system associated with a motor vehicle)); and other information (e.g., other information 340-1(h)) collected while the user is interacting with the application (e.g., information about transactions completed, such as information about the user's use of APPLE PAY.

[0148] In some embodiments, the application each usage entry further includes information identifying an action type performed by a user, while in other embodiments, the information identifying the in-app actions performed is used to determine or derive action types.

[0149] In some embodiments, the application usage data tables 335 also store information about privacy settings associated with users of the device 100. For example, the users of device 100 are able to configure privacy settings associated with the collection of usage data for each application. In some embodiments, users are able to control data collection settings for all information contained within each usage entry (e.g., in-app actions performed, other actions performed, sensor data, device coupling info, and other information). For example, a user can configure a privacy setting so that the device 100 (or a component thereof, such as usage data collecting module 163-2) does not collect location data, but does collect information about in-app actions performed for the browser module 147. As another example, the user can configure a privacy setting so that the device 100 does not collect information about in-app actions performed, but does collect location data for the online video module 155. In this way, users are able to control the collection of usage data on the device 100 and configure appropriate privacy settings based on their personal preferences regarding the collection of usage data for each application available on the device 100.

[0150] Figures 4A-4B are block diagrams illustrating data structures for storing trigger conditions, in accordance with some embodiments. As shown in Figure 4A, proactive trigger condition tables 402 include a collection of data structures 402, optionally implemented as a collection of tables for each respective application installed on the device 100, that each store trigger conditions associated with the respective application (e.g., application 1 trigger conditions table 402-1 stores trigger conditions that are associated with application 1 (e.g., trigger conditions that, when satisfied, cause the device 100 to launch or use application 1)). In some embodiments, each table (e.g., table 335-1, 335-2, 335-3 ... 335-N) in the collection of application usage data tables stores trigger conditions associated with more than one application installed on the electronic device (e.g. table 335-1 stores trigger conditions for related applications that are each provided by a common application developer or application vendor, for efficient storage of potentially related data).

[0151] In some embodiments, one or more proactive trigger condition tables 402 (e.g., application 1 trigger conditions table 402-1) are used for storing trigger conditions associated with applications installed on the device 100. For example, as illustrated in Figure 4B, an application 1 trigger condition table 402-1 contains information identifying a number of prerequisite conditions and associated actions for each trigger condition that is associated with application 1. As shown in Figure 4B, the application 1 trigger condition table 402-1 contains records 414-1 through 414-z and, optionally, includes a header 414-0. Header 414-0, in some embodiments, contains a brief description of each field of information (e.g., each field associated with each of the records) stored within the table. Each record (e.g., record 414-1) includes information that allows the device 100 to determine the prerequisite conditions for satisfying each trigger condition. In some embodiments, prereqs 1 of record 414-1 contains or identifies a number of prerequisite conditions (e.g., sensor readings) that, when detected, cause the device 100 to perform the associated action (e.g., action 4).

[0152] As a specific example, prereqs 1 may indicate that if the time of day is between 4:00 PM - 4:30 PM; location data (e.g., as reported by GPS module 135) shows that the user is still near their office (e.g., within a predetermined distance of their work address); and accelerometer data shows that the user is moving (e.g., as reported by accelerometers 168), then the device 100 should detect the trigger condition associated with prereqs 1 and perform action 4 (e.g., action 4 is associated with instant messaging module 141 and causes the module 141 to send a message to the user's spouse (or present a dialog asking the user whether they would like to send the message) indicating he / she is headed back home from work). According to the invention, prerequisite conditions are identified based on a pattern of user behavior identified by the trigger establishing module 163-1 (Figure 1A). The trigger establishing module 163-1, in conjunction with usage data collecting module 163-2 and application usage data tables 335, mines data that is stored in the application usage data tables to identify the patterns of user behavior. Continuing the previous example, after observing on three separate days that the user has sent the message to their spouse between 4:00 PM - 4:30 PM, while the user is within the predetermined distance of their work and while the user is moving, then the trigger establishing module 163-1 creates a corresponding trigger condition to automatically send the message (or ask the user for permission to automatically send the message) when the prerequisite conditions are observed. In some embodiments, the trigger establishing module 163-1 analyzes or mines the application usage data tables 335 at predefined intervals (e.g., every hour, every four hours, every day, or when the device is connected to an external power source) and creates trigger conditions only at these predefined intervals. In some embodiments, the user confirms that the trigger condition should be created (e.g., the device 100 presents a dialog to the user that describes the prerequisite conditions and the associated action and the user then confirms or rejects the creation of the trigger condition). For example, an example dialog contains the text "I've noticed that you always text your wife that you are on your way home at this time of day. Would you like to send her a text saying: I'm heading home now?"

[0153] Figure 5 is a block diagram illustrating an example trigger condition establishing system, in accordance with some embodiments. As shown in Figure 5, a trigger condition establishing system 500 includes the portable multifunction device 100 and also includes one or more servers 502. The portable multifunction device 100 communicates with the one or more servers 502 over one or more networks. The one or more networks (e.g., network(s) 520) communicably connect each component of the trigger condition establishing system 500 with other components of the trigger condition establishing system 500. In some embodiments, the one or more networks 520 include public communication networks, private communication networks, or a combination of both public and private communication networks. For example, the one or more networks 520 can be any network (or combination of networks) such as the Internet, other wide area networks (WAN), local area networks (LAN), virtual private networks (VPN), metropolitan area networks (MAN), peer-to-peer networks, and / or ad-hoc connections.

[0154] In some embodiments, one or more proactive trigger condition tables 402 are stored on the portable multifunction device 100 and one or more other proactive trigger condition tables 402 are stored on the one or more servers 502. In some embodiments, the portable multifunction device 100 stores the proactive trigger condition tables 402, while in other embodiments, the one or more servers 502 store the proactive trigger condition tables 402. Similarly, in some embodiments, one or more application usage data tables 335 are stored on the portable multifunction device 100 and one or more other application usage data tables 335 are stored on the one or more servers 502. In some embodiments, the portable multifunction device 100 stores the application usage data tables 335, while in other embodiments, the one or more servers 502 store the application usage data tables 335.

[0155] In embodiments in which one or more proactive trigger condition tables 402 or one or more application usage data tables 335 are stored on the one or more servers 502, then some of functions performed by the trigger establishing module 163-1 and the usage data collecting module 163-2, respectively, are performed at the one or more servers 502. In these embodiments, information is exchanged between the one or more servers 502 and the device 100 over the networks 520. For example, if the one or more servers 502 store proactive trigger condition tables 402 for the online video module 155, then, in some embodiments, the device 100 sends one or more usage entries corresponding to the online video module 155 to the one or more servers 502. In some embodiments, the one or more servers 502 then mine the received usage data to identify usage patterns and create trigger conditions (as discussed above in reference to Figures 4A-4B) and sends the created trigger conditions to the device 100. In some embodiments, while receiving data associated with the online video module 155 (e.g., data for one or more video streams), the device 100 and the one or more servers 502 exchange usage data and trigger conditions. In some embodiments, the one or more servers 502 are able to detect the created trigger conditions as well (e.g., based on the usage data received during the exchange of the data for one or more video streams, the server can determine that the trigger conditions has been satisfied), such that the trigger conditions do not need to be sent to the device 100 at all. In some embodiments, the usage data that is sent to the one or more servers 502 is of limited scope, such that it contains only information pertaining to the user's use of the online video module 155 (as noted above, the user must also configure privacy settings that cover the collection of usage data and these privacy settings, in some embodiments, also allow the user to configure the exchange of usage data with one or more servers 502 (e.g., configure what type of data should be sent and what should not be sent)).

[0156] In some embodiments, data structures discussed below in reference to Sections 1-11 are also used to help implement and / or improve any of the methods discussed herein. For example, the predictions engines discussed below in reference to Figures 1-11 are used to help establish trigger conditions and / or other techniques discussed in Sections 1-11 are also used to help monitor application usage histories.

[0157] Figures 6A-6B illustrate a flowchart representation of a method 600 of proactively identifying and surfacing relevant content, in accordance with some embodiments. Figures 3A-3B, 4A-4B, 5, and 7A-7B are used to illustrate the methods and / or processes of Figures 6A-6B. Although some of the examples which follow will be given with reference to inputs on a touch-sensitive display (in which a touch-sensitive surface and a display are combined), in some embodiments, the device detects inputs on a touch-sensitive surface 195 that is separate from the display 194, as shown in Figure 1D.

[0158] In some embodiments, the method 600 is performed by an electronic device (e.g., portable multifunction device 100, Figure 1A) and / or one or more components of the electronic device (e.g., I / O subsystem 106, operating system 126, etc.). In some embodiments, the method 600 is governed by instructions that are stored in a non-transitory computer-readable storage medium and that are executed by one or more processors of a device, such as the one or more processors 122 of device 100 (Figure 1A). For ease of explanation, the following describes method 600 as performed by the device 100. In some embodiments, with reference to Figure 1A, the operations of method 600 are performed by or use, at least in part, a proactive module (e.g., proactive module 163), application usage data tables (e.g., application usage data tables 335), trigger condition tables (e.g., trigger condition tables 402), a trigger establishing module (e.g., trigger establishing module 163-1), a usage data collecting module (e.g., usage data collecting module 163-2), a proactive suggestions module (e.g., proactive suggestions module 163-3), a contact / motion module (e.g., contact / motion module 130), a graphics module (e.g., graphics module 132), one or more contact intensity sensors (e.g., contact intensity sensors 165), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 600 are, optionally, combined and / or the order of some operations is, optionally, changed.

[0159] As described below, the method 600 provides an intuitive way to proactively identify and surface relevant content on an electronic device with a touch-sensitive display. The method creates more efficient human-machine interfaces by requiring fewer touch inputs in order to perform various functions. For battery-operated electronic devices, proactively identifying and surfacing relevant content faster and more efficiently both conserves power and increases the time between battery charges.

[0160] As shown in Figure 6A, the device executes (602), on the electronic device, an application in response to an instruction from a user of the electronic device. In some embodiments, the instruction from the user is a touch input over an icon associated with the application or a voice command received from the user that instructs a virtual assistant application (e.g., a virtual assistant application managed by operating system 126, Figure 1A) to execute the application. While executing the application, the device (or a component thereof, such as usage data collecting module 163-2) collects (604) usage data that includes one or more actions performed by the user within the application. In some embodiments the usage data, in addition to or instead of including the one or more actions, also includes information identifying an action type associated with each of the one or more actions. For example, the usage data includes information identifying that, while interacting with the music player module 152, the user searched for a first playlist, navigated within the first playlist, selected a first track within the first playlist, then searched for a second playlist (e.g., the usage data includes each of the one or more actions performed by the user within the music player module 152). In this way, the usage data includes information about each of the individual actions performed (e.g., the user searched for and played the first track of the first playlist) and also includes information identifying the action types (search, navigate, select, etc.). In some embodiments, the usage data collection module 163-2 collects the one or more actions and then the trigger establishing module 163-1 later assigns an action type to each of the one or more actions.

[0161] In some embodiments the collected usage data is stored in a usage entry (as described above in reference to Figures 3A-3B) in an application usage data table that is associated with the application. In some embodiments, the collected usage data includes in-app actions performed by the user, other actions performed by the user (e.g., interactions with a virtual assistant application, interactions with a search interface (e.g., search module 151), and other interactions with applications that are managed by the operating system 126), information associated with calendar events, and additional data obtained from sensors on the device 100 (as explained above in reference to Figure 3B).

[0162] In some embodiments, the usage data includes (618) verbal instructions, from the user, provided to a virtual assistant application while continuing to execute the application, and the at least one trigger condition is further based on the verbal instructions provided to the virtual assistant application. In some embodiments, the verbal instructions comprise a request to create a reminder that corresponds to (e.g., references or requires recreation / re-execution of) a current state of the application, the current state corresponding to a state of the application when the verbal instructions were provided (e.g., one or more application views 191, Figure 1B). In some embodiments, the state of the application when the verbal instructions were provided is selected from the group consisting of: a page displayed within the application when the verbal instructions were provided, content playing within the application when the verbal instructions were provided (e.g., a currently playing audio track), a notification displayed within the application when the verbal instructions were provided (e.g., a notification from instant messaging module 141 that is displayed while the user is interacting with browser module 147), and an active portion of the page displayed within the application when the verbal instructions were provided (e.g., currently playing video content within a web page). As additional examples, the current state of the application might correspond also to (i) an identifier of the particular page (e.g., a URL for a currently displayed webpage) that the user is currently viewing within the application when the verbal instructions are provided or a history of actions that the user took before navigating to a current page within the application (e.g., URLs visited by the user prior to the currently displayed webpage).

[0163] In some embodiments, the verbal instructions include the term "this" or "that" in reference to the current state of the application. For example, the user provides the instruction "remind me of 'this'" to the virtual assistant application while a notification from instant messaging module 141 is displayed, and, in response, the virtual assistant application causes the device 100 to create a reminder corresponding to content displayed within the notification. As another example, the user provides the instruction "remind me to watch 'this'" to the virtual assistant application while the user is watching particular video content in the online video module 155 and, in response, the virtual assistant application causes the device 100 to create a reminder corresponding to the particular video content. In some embodiments, the device 100 receives information regarding the current state of the application when the verbal instructions were provided from the application itself (e.g., continuing with the previous example, the online video module 155 reports its current state back to the device 100, or to a component thereof such as proactive module 163 and, in this way, the proactive module 163 receives information identifying the particular video content)

[0164] The device then automatically and without human intervention, obtains (606) at least one trigger condition based on the collected usage data. In some embodiments, the at least one trigger condition is established on the device, while in other embodiments, the trigger condition is obtained (612) from a server (e.g., one or more servers 502, Figure 5) that established the trigger condition based on usage data that was sent from the device to the one or more servers 502 (as explained above in reference to Figure 5). In some embodiments, the at least one trigger condition, when satisfied, causes the device (or a component thereof, such as proactive module 163) to allow the user to easily perform (e.g., without any input or with only a single touch or verbal input from the user) an action that is associated with the at least one trigger condition. For example one example trigger might indicate that between 2:00 PM and 2:30 PM, while the accelerometer data (e.g., as reported by accelerometers 168) indicates that the user is walking between previously-visited GPS coordinates (e.g., between two often-visited buildings located near a work address for the user), the device should automatically (and without any input from the user) open a music application (e.g., music player 152, Figure 1A) and begin playing a specific playlist. In some embodiments, this example trigger was established (by the one or more servers 502 or by the device 100) after collecting usage data and determining that the collected usage data associated with the music player 152 indicates that the user opens the music player 152 and plays the specific playlist while walking between the previously-visited GPS coordinates every weekday between 2:00 PM - 2:30 PM. In this way, the device (or the server) identifies and recognizes a pattern based on the collected usage data. By performing the action (e.g., playing the specific playlist) automatically for the user, the user does not need to waste any time unlocking the device, searching for the music player 152, searching for the specific playlist, and then playing the specific playlist.

[0165] In some embodiments, the method also includes checking privacy settings associated with the user of the device prior to establishing or obtaining trigger conditions, in order to confirm that the user has permitted the device to collect certain usage data and / or to verify that the user has permitted the device to establish trigger conditions (e.g., the user may configure a setting to prohibit the device from establishing trigger conditions that cause the device to automatically send text messages).

[0166] The device (or a component thereof, such as trigger condition establishing module 163-1) also associates (608) the at least one trigger condition with a particular action (or with a particular action type that corresponds to the particular action) of the one or more actions performed by the user within the application (e.g., by storing the prerequisite conditions for satisfying the trigger condition together with the particular action in a proactive trigger condition table 402, Figures 4A-4B). Upon determining that the at least one trigger condition has been satisfied, the device provides (610) an indication to the user that the particular action (or that the particular action type) associated with the trigger condition is available. In some embodiments, providing the indication to the user includes surfacing a user interface object for launching the particular action (or for performing an action corresponding to the particular action type) (e.g., UI object 702, Figure 7A), surfacing an icon associated with the application that performs the particular action (e.g., application icon 710, as shown in the bottom left corner of touch screen 112, Figure 7A), or simply performing the particular action (as described in the example of the specific playlist above). In some embodiments, the device surfaces the user interface object and / or the icon, while also (automatically and without human intervention) simply performing the particular action (or an action that is of the same particular action type as the particular action).

[0167] In some embodiments, obtaining the at least one trigger condition includes (612) sending, to one or more servers that are remotely located from the electronic device (e.g., servers 502, Figure 5), the usage data and receiving, from the one or more servers the at least one trigger condition. For example, consistent with these embodiments, the electronic device sends (over networks 520) one or more usage entries (e.g., usage entry 1, Figure 3B) to the servers 502 and, based on the usage data, the servers 502 establish the at least one trigger condition. Continuing the example, the servers 502 then send (using networks 520) the at least one trigger condition (e.g., prerequisite conditions and associated actions, stored in a proactive trigger condition table 402-1) to the device 100.

[0168] In some embodiments, providing the indication includes (614) displaying, on a lock screen on the touch-sensitive display, a user interface object corresponding to the particular action associated with the trigger condition. In some embodiments, the user interface object is displayed in a predefined central portion of the lock screen (e.g., as pictured in Figure 7A, the UI object 702 is displayed substantially in the middle of the lock screen). For example, the device provides the indication by displaying UI object 702 on the lock screen (Figure 7A). As shown in Figure 7A, UI object 702 includes a predicted action 706. In some embodiments, the predicted action 706 is a description of an action associated with the at least one trigger condition (in other words, the user interface object includes a description of the particular action associated with the trigger condition (616)), such as "Swipe to Play Track 2 of Walking Playlist"). In some embodiments, the UI object 702 also optionally includes additional info 704 that provides information to the user as to why the UI object 702 is being displayed. In some embodiments, the additional info 704 includes a description of the usage data that was used to detect the trigger condition (e.g., sensor data 340-1(c)) and / or a description of the prerequisite conditions for the at least one trigger condition (e.g., prereqs 1 of record 414-1, Figure 4B). For example, the additional info 704 indicates that the predicted action 706 is being displayed because the user often listens to the walking playlist at this particular time of day and while the user is walking. In some embodiments, selecting the additional info 704 (e.g., tapping on top of the additional info 704) causes the device 100 to displays a user interface that allows the user to change privacy settings associated with the collection of usage data and the creation of trigger conditions.

[0169] In some embodiments, the UI object 702 also optionally includes (616) an application icon 710 that is associated with the predicted action 706. For example, the application icon 710 is the icon for music player 152 (as shown in Figure 7A). In some embodiments, the UI object 702 also includes an affordance 708 that, when selected, causes the device to perform the predicted action (e.g., causes the device to begin playing track 2 of the walking playlist). In some embodiments, the user interface object (e.g., user interface object 702) includes a description of the particular action associated with the trigger condition (e.g., predicted action 706, as explained above). In some embodiments, the user interface object 702 further includes an icon associated with the application (e.g., application icon 710 displayed within the UI object 702). In some embodiments, the user interface object 702 further includes a snooze button that, when selected, causes the device to cease displaying the UI object 702 and to re-display the UI object 702 after a period of time selected or pre-configured by the user. For example, the user selects to snooze the UI object 702 for two hours and, after the two hours, the device then re-displays the UI object 702. As another example, the user selects to snooze the UI object 702 until they are available and, in some embodiments, the device 100 searches the calendar module 148 to identify the next open slot in the user's schedule and re-displays the UI object 702 during the identified next open slot.

[0170] In some embodiments, the device detects (622) a first gesture at the user interface object. In response to detecting the first gesture, the device displays (624), on the touch-sensitive display, the application and, while displaying the application, performs the particular action associated with the trigger condition. In some embodiments, the first gesture is a swipe gesture over the user interface object. In some embodiments, in response to detecting the swipe gesture over the user interface object, the device also unlocks itself prior to displaying the application (in other embodiments, the application is displayed right on the lock screen). According to the invention, the first gesture is indicated by the text displayed within the UI object 702 (e.g., the text within predicted action 706 includes a description of the first gesture, e.g., "Swipe to ..."). For example and with references to Figure 7A, the user makes contact with the touch-sensitive surface on top of the UI object 702 and, without breaking contact with the touch-sensitive surface, the user moves the contact in a substantially horizontal direction across the UI object 702. In response to detecting this swipe gesture from the user over the UI object 702, the device displays the music player 152 and begins playing track 2 of the walking playlist.

[0171] According to the invention, the device detects (626) a second gesture (e.g., a gesture distinct from the first gesture discussed above, such as a single tap at a predefined area of the user interface object (e.g., a play button, such as the affordance 708)) at the user interface object. In response to detecting the second gesture and while continuing to display the lock screen on the touch-sensitive display, the device performs (628) the particular action associated with the trigger condition. In other words, the device performs the particular action right from the lock screen and continues to display the lock screen, without displaying the application.

[0172] In some embodiments, the first and second gestures discussed above in reference to operations 622-628 are the same gesture but they are performed over different objects displayed within the UI object 702. For example, the first gesture is a swipe gesture over the predicated action 706, while the second gesture is a swipe gesture over the affordance 708. As another example, the first gesture is a single tap over the predicted action 706 and the second gesture is a single tap over the affordance 708.

[0173] In some embodiments, providing the indication to the user that the particular action is available includes letting the user know that the particular action is available for execution. In some embodiments, providing the indication to the user that the particular action associated with the trigger condition is available includes performing the particular action. In some embodiments, the indication is provided to the user by virtue of the performance of the particular action (e.g., the user hearing that a desired playlist is now playing). In some embodiments, the UI object 702 is displayed on the lock screen and the particular action is also performed without receiving any user input (such as the first and second gestures discussed above).

[0174] In some embodiments, instead of (or in addition to) displaying the UI object 702, the device displays an icon associated with the application substantially in a corner of the lock screen (e.g., as pictured in Figure 7A, application icon 710 is displayed substantially in a lower left corner of the touch screen 112).

[0175] In some embodiments, the device receives an instruction from the user to unlock the electronic device (e.g., recognizes the user's fingerprint as valid after an extended contact over the home button 204). In response to receiving the instruction (e.g., after unlocking the device and ceasing to display the lock screen), the device displays (620), on the touch-sensitive display, a home screen of the device and provides, on the home screen, the indication to the user that the particular action associated with the trigger condition is available. As pictured in Figure 7B, the UI object 702 is displayed as overlaying a springboard section (or application launcher) of the home screen after receiving the instruction to unlock the device. In some embodiments, instead of or in addition to display the UI object 702 at the top of the home screen, the device also displays the application icon 710 in a bottom portion that overlays a dock section of the home screen. In some embodiments, the home screen includes: (i) a first portion including one or more user interface pages for launching a first set of applications available on the electronic device (e.g., the first portion consists of all the individual pages of the springboard section of the home screen) and (ii) a second portion, that is displayed adjacent (e.g., below) to the first portion, for launching a second set of applications available on the electronic device, the second portion being displayed on all user interface pages included in the first portion (e.g., the second portion is the dock section). In some embodiments, providing the indication on the home screen includes displaying the indication over the second portion (e.g., as shown in Figure 7B, the bottom portion that includes application icon 710 is displayed over the dock portion). In some embodiments, the second set of applications is distinct from and smaller than the first set of applications (e.g., the second set of applications that is displayed within the dock section is a selected set of icons corresponding to favorite applications for the user).

[0176] In some embodiments, determining that the at least one trigger condition has been satisfied includes determining that the electronic device has been coupled with a second device, distinct from the electronic device. For example, the second device is a pair of headphones that is coupled to the device via the headset jack 212 and the at least one trigger condition includes a prerequisite condition indicating that the pair of headphones has been coupled to the device (e.g., prior to executing a particular action that includes launching the user's favorite podcast within a podcast application that the user always launches after connecting headphones). As another example, the second device is a Bluetooth speaker or other hands-free device associated with the user's motor vehicle and the at least one trigger condition includes a prerequisite condition indicating that the motor vehicle's Bluetooth speaker has been coupled to the device (e.g., prior to executing a particular action that includes calling the user's mom if the time of day and the user's location match additional prerequisite conditions for the particular action of calling the user's mom). Additional details regarding the coupling of an external device and performing an action in response to the coupling are provided in Section 6 below (e.g., in reference to Figure 36_1 of Section 6).

[0177] In some embodiments, determining that the at least one trigger condition has been satisfied includes determining that the electronic device has arrived at a location corresponding to a home or a work location associated with the user. In some embodiments, the device monitors locations (e.g., specific GPS coordinates or street addresses associated with the locations) that are frequently visited by the user and uses this information to ascertain the home or the work location associated with the user. In some embodiments, the device determines addresses for these locations based on information received from or entered by the user (such as stored contacts). In some embodiments, determining that the electronic device has arrived at an address corresponding to the home or the work location associated with the user includes monitoring motion data from an accelerometer of the electronic device and determining, based on the monitored motion data, that the electronic device has not moved for more than a threshold amount of time (e.g., user has settled in at home and has not moved for 10 minutes). In this way, for example, the device ensures that the particular action associated with the at least one trigger condition is performed when the user has actually settled in to their house, instead of just when the user arrives at the driveway of their house.

[0178] In some embodiments of the method 600 described above, the method begins at the obtaining operation 606 and, optionally, includes the executing operation 602 and the collecting operation 604. In other words, in these embodiments, the method 600 includes: obtaining at least one trigger condition that is based on usage data associated with a user of the electronic device, the usage data including one or more actions performed by the user within an application while the application was executing on the electronic device; associating the at least one trigger condition with a particular action of the one or more actions performed by the user within the application; and, upon determining that the at least one trigger condition has been satisfied, providing an indication to the user that the particular action associated with the trigger condition is available.

[0179] It should be understood that the particular order in which the operations in Figures 6A-6B have been described is merely one example and is not intended to indicate that the described order is the only order in which the operations could be performed. One of ordinary skill in the art would recognize various ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein with respect to other methods described herein (e.g., method 800) are also applicable in an analogous manner to method 600 described above with respect to Figures 6A-6B. For example, the user interface objects described above with reference to method 600 optionally have one or more of the characteristics of the user interface objects described herein with reference to other methods described herein (e.g., method 800). In some embodiments, any relevant details from Sections 1-11 may be utilized for any suitable purpose in conjunction with method 600. For brevity, these details are not repeated here.

[0180] Figures 8A-8B illustrate a flowchart representation of a method 800 of proactively identifying and surfacing relevant content, in accordance with some embodiments. Figures 3A-3B, 4A-4B, 5, and 9A-9D are used to illustrate the methods and / or processes of Figures 8A-8B. In some embodiments, the user interfaces illustrated in Figures 9A-9D are referred to as a zero-keyword search. A zero-keyword search is a search that is conducted without any input from a user (e.g., the search entry box remains blank) and allows the user to, for example, view people, applications, actions within applications, nearby places, and / or news articles that the user is likely going to (or predicted to) search for next. Although some of the examples which follow will be given with reference to inputs on a touch-sensitive display (in which a touch-sensitive surface and a display are combined), in some embodiments, the device detects inputs on a touch-sensitive surface 195 that is separate from the display 194, as shown in Figure 1D.

[0181] In some embodiments, a method 800 is performed by an electronic device (e.g., portable multifunction device 100, Figure 1A) and / or one or more components of the electronic device (e.g., I / O subsystem 106, operating system 126, etc.). In some embodiments, the method 800 is governed by instructions that are stored in a non-transitory computer-readable storage medium and that are executed by one or more processors of a device, such as the one or more processors 122 of device 100 (Figure 1A). For ease of explanation, the following describes a method 800 as performed by the device 100. In some embodiments, with reference to Figure 1A, the operations of method 800 are performed by or use, at least in part, a proactive module (e.g., proactive module 163), application usage data tables (e.g., application usage data tables 335), trigger condition tables (e.g., trigger condition tables 402), a trigger establishing module (e.g., trigger establishing module 163-1), a usage data collecting module (e.g., usage data collecting module 163-2), a proactive suggestions module (e.g., proactive suggestions module 163-3), a search module (e.g., search module 151), a contact / motion module (e.g., contact / motion module 130), a graphics module (e.g., graphics module 132), one or more contact intensity sensors (e.g., contact intensity sensors 165), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 800 are, optionally, combined and / or the order of some operations is, optionally, changed.

[0182] As described below, the method 800 provides an automated method for proactively identifying and surfacing relevant content (before the user explicitly asks for the relevant content, e.g., before the user enters any text into a search entry portion of a search interface) on an electronic device with a touch-sensitive display. The method reduces the cognitive burden on a user when accessing applications, thereby creating a more efficient human-machine interface.

[0183] As shown in Figure 8A, the device detects (802) a search activation gesture on the touch-sensitive display. For example, as shown in Figure 9A, the device detects a search activation gesture 902-1 (e.g., a contact on the touch-sensitive display followed by continuous movement of the contact in a substantially vertical direction (e.g., downward)). As another example, as is also shown in Figure 9A, the device detects a search activation gesture 902-2 (e.g., a contact on the touch-sensitive surface followed by continuous movement of the contact in a substantially horizontal direction (e.g., rightward)). In some embodiments, the search activation gesture is available from at least two distinct user interfaces, and a first user interface of the at least two distinct user interfaces corresponds to displaying a respective home screen page of a sequence of home screen pages on the touch-sensitive display.

[0184] In some embodiments, when the respective home screen page is a first home screen page in the sequence of home screen pages (e.g., as shown in Figure 9A), the search activation gesture includes one of the following: (i) a gesture moving in a substantially downward direction relative to the user of the electronic device (e.g., gesture 902-1) or (ii) a continuous gesture moving in a substantially left-to-right direction relative to the user and substantially perpendicular to the downward direction (e.g., gesture 902-2). In some embodiments, when the respective home screen page is a second home screen page in the sequence of home screen pages (in other words, not the first home screen page), the search activation gesture is the continuous gesture moving in the substantially downward direction relative to the user of the electronic device (in other words, only the search activation gesture 902-1 is available and gesture 902-2 is not available).

[0185] In some embodiments, a second user interface of the at least two distinct user interfaces corresponds to displaying an application switching interface on the touch-sensitive display (e.g., in response to the user double tapping on the home button 204). In some embodiments, the search activation gesture comprises a contact, on the touch-sensitive display, at a predefined search activation portion of the application switching interface (e.g., the application switching interface includes a search entry portion that is the predefined search activation portion (similar to search entry portion 920 of Figure 9B) displayed within a top portion of the application switching interface).

[0186] In response to detecting the search activation gesture, the device displays (804) a search interface on the touch-sensitive display that includes (806): (a) a search entry portion (e.g., search entry portion 920 for receiving input from a user that will be used as a search query, Figure 9B) and (b) a predictions portion that is displayed before receiving any user input at the search entry portion (e.g., predictions portion 930, Figure 9B). The predictions portion is populated with one or more of: (a) at least one affordance for contacting a person of a plurality of previously-contacted people (e.g., the affordances displayed within suggested people 940 section, Figure 9B) and (b) at least one affordance for executing a predicted action within an application (e.g., a "deep link") of a plurality of applications available on the electronic device (e.g., suggested actions 950 section, Figure 9B). "Within" the application refers to the at least one affordance for executing the predicated action representing a link to a specific page, view, or state (e.g., one of application views 191, Figure 1B) within the application. In other words the at least one affordance for executing the predicted action, when selected, does not just launch the application and display default content or content from a previous interaction with the application, but instead displays the specific page, view, or state corresponding to the deep link.

[0187] In some embodiments, the person is automatically selected (e.g., by the device 100 or proactive module 163) from the plurality of previously-contacted people based at least in part on a current time. For example, every day around 5:30 PM, while the user is still at work (work location is determined as explained above with reference to Figures 6A-6B), the user sends a text to their roommate indicating that they are headed home, so the predictions portion includes an affordance that is associated with the roommate (e.g., P-1 is for the roommate).

[0188] In some embodiments, the predicted action is automatically selected (e.g., by the device 100 or proactive module 163) based at least in part on an application usage history associated with the user of the electronic device (e.g., the application usage history (as provided by one or more application usage tables 335, Figures 3A-3B) indicates that every day around 2:15 PM the user opens the search interface (by providing the search activation gesture, as discussed above), searches for "music," selects a particular music app search result, and then plays a "walking playlist," so, based on this application usage history, the predictions portion, before receiving any user input in the search entry portion, includes an affordance to start playing the playlist within the music app (e.g., as shown by the content displayed within the suggested actions 950 section, Figure 9B)). In some embodiments, the at least one affordance for executing the predicated action within the application is also selected (instead of or in addition to the application usage history) based at least in part on the current time (e.g., based on the user providing the search activation gesture at around the same time that the user typically performs the predicted action). In some embodiments (and as pictured in Figure 9B), the at least one affordance for executing a predicted action corresponds to the user interface object 702 and, thus, the details provided above (Figures 6A-6B and 7A-7B) regarding user interface object 702 apply as well to the suggested actions section 950 and the content displayed therein.

[0189] In some embodiments, the person is further selected based at least in part on location data corresponding to the electronic device (e.g., the user frequently contacts their significant other when they reach an address in the morning associated with their work). In some embodiments, the application usage history and contact information for the person are retrieved from a memory of the electronic device (e.g., memory 102 of device 100, Figure 1A). In some embodiments, the application usage history and contact information for the person are retrieved from a server that is remotely located from the electronic device (e.g., one or more servers 502, Figure 5).

[0190] In some embodiments, the predictions portion is further populated (808) with at least one affordance for executing a predicted application (e.g., suggested apps 955 section, Figure 9B). In some embodiments, the predicted application is automatically selected (by the device 100) based at least in part on the application usage history. For example, the application usage history (e.g., one or more records within one of the application usage data tables 335, Figures 3A-3B) indicates that the user opens the calendar module 143 (Figure 1A) every morning at around 9:00 AM when they are at their home address and, thus, the suggested apps 955 section includes an affordance for the calendar module 143 when the current time is around 9:00 AM and the location data indicates that the user is at their home address. As an additional example, the application usage history indicates that a weather application (e.g., weather widget 149-1, Figure 1A) is has been launched on three consecutive days at around 5:15 AM and it is now 5:17 AM (e.g., the current time is 5:17 AM when the user launches spotlight using the search activation gesture), so the electronic device populates the search interface with the weather application as one of the predicted applications in the predictions portion based at least in part on this application usage history. In some embodiments, the predicted applications and the prediction actions are displayed within a single section in which the predicted actions are displayed above the predicted applications. As noted in the preceding examples, in some embodiments, the at least one affordance for executing the predicated application is also selected (instead of or in addition to the application usage history) based at least in part on the current time (e.g., based on the user providing the search activation gesture at around the same time that the user typically uses the predicted application).

[0191] In some embodiments, in order to populate the suggested apps 955 section, the device 100 (or a component thereof such as proactive module 163) determines whether any of the prerequisite conditions for a trigger (e.g., prereqs stored in one of the trigger condition tables 402, Figures 4A-4B) are satisfied and, in accordance with a determination that a particular trigger is satisfied, the device 100 populates the suggested apps 955 section accordingly (e.g., adds an affordance corresponding to an application that is associated with the trigger, such as the calendar module 143 or the weather widget 149-1 in the preceding examples). In some embodiments, the other sections within the search interface (e.g., sections 940, 950, 955, 960, and 990) are populated using a similar determination process (for the sake of brevity, those details are not repeated herein).

[0192] In some embodiments, the predictions portion is further populated (808) with at least one affordance for a predicted category of nearby places (e.g., suggested places 960 section, Figure 9B), and the predicted category of places (e.g., nearby places) is automatically selected based at least in part on one or more of: the current time and location data corresponding to the device. For example, the current time of day is around 7:30 AM and the location data indicates that the device is near (within a predetermined distance of) popular coffee shops (popularity of the coffee shops is determined, in some embodiments, by crowd-sourcing usage data across numerous device 100 associated with numerous distinct users) and, thus, the device 100 populates the suggested places 960 section with an affordance for "Coffee Shops." In some embodiments, the suggested places 960 section is populated with (in addition to or instead of the predicted category of places) information corresponding to a predicted search for nearby places based on the current time. In other words, based on previous searches (e.g., searches within the search module 151 or the browser module 147) conducted by the user at around the current time, the device proactively predicts a search the user is likely to conduct again. For example, based on the user having searched for "Coffee" between 7:20 AM and 8:00 AM on four previous occasions (or some other threshold number of occasions), the device (e.g., the trigger establishing module 163-1), in response to detecting the search activation gesture, populates the suggested places 960 section with an affordance for "Coffee Shops." In other embodiments, the suggested categories are only based on the device's current location and not on time. For example, an affordance linking to nearby coffee shops is displayed. In this way, the user does not need to manually conduct the search for "Coffee" again and can instead simply select the "Coffee Shops" or "Food" affordance and quickly view a list of nearby coffee shops. In some embodiments, the previous search history is stored with one or more usage entries as other information (e.g., other information 340-1(h), Figure 3B) and / or as other actions performed (e.g., other actions performed 340-1(b), Figure 3B).

[0193] In some embodiments, the device detects user input to scroll the predictions portion (e.g., scroll gesture 970, figure 9B) and, in response to detecting the user input to scroll the predictions portion, the device scrolls the predictions portion in accordance with the user input (e.g., scrolls the search interface in a downward direction or scrolls only the predictions portion within the search interface). In response to the scrolling, the device reveals at least one affordance for a predicted news article in the predictions portion (e.g., suggested news articles 990 section, Figure 9C). In some embodiments, the predicted news article(s) is(are) automatically selected (by the device 100) based at least in part on location data corresponding to the electronic device. In some embodiments, the suggested news articles 990 section is displayed without requiring the scroll input. In some embodiments, the predicted news article is optionally selected (in addition to or instead of the location data) based at least in part on the current time (e.g., the user has read similar or related articles more than a threshold number of times (e.g., three times) at around the current time (e.g., the time at which the user provided the search activation gesture that caused the device to display the search interface with the predictions portion 930)), a previous search history corresponding to the user (e.g., the user has searched for articles that are similar or related more than a threshold number of times (e.g., three times) to the predicted news article), trending data associated with the news story through searches conducted by other users, the user's friends, in social media, such as Twitter or Facebook, etc.

[0194] In some embodiments, the particular order in which the sections 940, 950, 955, 960, and 990 are displayed within the predictions portion 930 is configurable, such that the user is able to choose a desired ordering for each of the sections. For example, the user can configure the ordering such that the suggested apps 955 section is displayed first, the suggested people 940 section is displayed second, the suggested actions 950 section is displayed third, the suggested news articles 990 section is displayed fourth, and the suggested places 960 section is displayed last. In some embodiments, the predictions portion 930 includes any two of the sections 940, 950, 955, 960, and 990. In other embodiments, the predictions portions 930 includes any three of the sections 940, 950, 9550, 960, and 990. In still other embodiments, the predictions portion 930 includes any four of the sections 940, 950, 955, 960, and 990. In yet other embodiments, the predictions portion 930 includes all of the sections, 940, 950, 955, 960, and 990. In some embodiments, the user configures a preference as to how many and which of the sections 940, 950, 955, 960, and 990 should be displayed within the predictions portion 930.

[0195] Additionally, the user, in some embodiments, is able to configure the weights given to the data (e.g., current time, application usage history, location data, other sensor data, etc.) that is used to populate each of the sections 940, 950, 955, 960, and 990. For example, the user configures a preference so that the current time is weighted more heavily than the location data when determining the affordances to display within the suggested people 940 section of the predictions portion 930.

[0196] Turning now to Figure 8B, in some embodiments, the affordances displayed within each of the aforementioned sections 940, 950, 955, 960, and 990 are selectable, so that a user is able to select one of: a suggested action, a suggested app, a suggested place, or a suggested news article, respectively (each is discussed in order below).

[0197] As to selection of the affordances displayed within the suggested people 940 section, in some embodiments, the device detects (810) a selection of the at least one affordance for contacting the person. In some embodiments, the device detects a single touch input over the at least one affordance (e.g., a single tap over the affordance corresponding to P-1 displayed within the suggested people 940 section). In some embodiments, in response to detecting the selection of the at least one affordance for contacting the person, the device contacts the person (or suggests different communication mediums, e.g., text, email, telephone, and the like, for contacting the person) using contact information for the person (e.g., contact information retrieved from the device or from one or more servers, as discussed above). For example, in response to detecting a single tap over the affordance corresponding to P-1, the device sends a text message to the user's roommate that reads "on my way home." In some embodiments, the device automatically contacts P-1, while in other embodiments, the device displays the instant messaging module 141 and pre-populates an interface within the module 141 with a message (e.g., "on my way home") and then awaits a request from the user before sending the message (e.g., a voice command or a selection of a send button by the user). In this way, the user of the device is able to conveniently and quickly contact the person (e.g., P-1) and also send a relevant (or desired) message without having to enter any text in the search entry portion (thus saving time and frustration if the user had to enter text and was unable to locate the person).

[0198] As to selection of the affordances displayed within the suggested actions 950 section, in some embodiments, the device detects (812) a selection of the at least one affordance for executing the predicted action. For example, the device detects a single touch input (e.g., a tap over the icon for music player 152 or a tap over the text "Tap to Play Track 2 of Walking Playlist") within the suggested actions 950 section. In some embodiments, in response to detecting the selection of the at least one affordance for executing the predicted action, the device displays the application on the touch-sensitive display and executes the predicted action within the displayed application. In other words, the device ceases to display the search interface (e.g., search module 151 with the search entry and predictions portions) and instead launches and displays the application, and executes the predicted action within the displayed application. For example, in response to detecting a single tap over the text "Tap to Play Track 2 of Walking Playlist," the device displays the music player module 152 and executes the predicted action by playing track 2 of the walking playlist. In this way, the user of the device is able to conveniently and quickly access a relevant (or desired) application (e.g., the music player module) and also execute a desired function within the desired application without having to enter any text in the search entry portion (thus saving time and frustration if the user had to enter text and was unable to locate the music player module).

[0199] As to selection of the affordances displayed within the suggested apps 955 section, in some embodiments, the device detects (814) a selection of the at least one affordance for executing the predicted application. In some embodiments, the device detects a single touch input over the at least one affordance (e.g., a single tap over the affordance for the icon for browser app 147). In some embodiments, in response to detecting the selection of the at least one affordance for executing the predicted application, the device displays the predicted application on the touch-sensitive display (e.g., the device ceases to display the search interface with the search entry portion and the predictions portion and instead opens and displays the predicted application on the touch-sensitive display). For example, in response to detecting a single tap over the affordance corresponding to the icon for browser app 147, the device displays the browser app 147 (e.g., browser module 147, Figure 1A). In this way, the user of the device is able to conveniently and quickly access a relevant (or desired) application (e.g., the browser application) without having to enter any text in the search entry portion (thus saving time and frustration if the user had to enter text and was unable to locate the browser application).

[0200] As to selection of the affordances displayed within the suggested places 960 section, in some embodiments, the device detects (816) a selection of the at least one affordance for the predicted category of places (e.g., nearby places). In some embodiments, the device detects a single touch input over the at least one affordance (e.g., a single tap over the affordance for the "Coffee Shops"). In some embodiments, in response to detecting the selection of the at least one affordance for executing the predicted category of places, the device: (i) receives data corresponding to at least one nearby place (e.g., address information or GPS coordinates for the at least one nearby place, as determined by map module 154) and (ii) displays, on the touch-sensitive display, the received data corresponding to the at least one nearby place (e.g., ceases to display the search interface, launches the maps module 154, displays the maps module 154 including a user interface element within a displayed map that corresponds to the received data, such as a dot representing the GPS coordinates for the at least one nearby place). In some embodiments, the receiving and displaying step are performed substantially in parallel. For example, in response to detecting a single tap over the affordance corresponding to "Coffee Shops," the device retrieves GPS coordinates for a nearby café that serves coffee and, in parallel, displays the maps module 154 and, after receiving the GPS coordinates, displays the dot representing the GPS coordinates for the café. In this way, the user of the device is able to conveniently and quickly locate a relevant (or desired) point of interest (e.g., the café discussed above) without having to enter any text in the search entry portion (thus saving time and frustration if the user had to enter text and was unable to locate the café or any coffee shop). In some embodiments, the receiving data operation discussed above is performed (or at least partially performed) before receiving the selection of the at least one affordance for the predicted category of places. In this way, data corresponding to the nearby places is pre-loaded and is quickly displayed on the map after receiving the selection of the at least one affordance for the predicted category of places.

[0201] As to selection of the affordances displayed within the suggested news articles 990 section, in some embodiments, the device detects (818) a selection of the at least one affordance for the predicted news article. In some embodiments, the device detects a single touch input over the at least one affordance (e.g., a single tap over the affordance for News 1, Figure 9C). In some embodiments, in response to detecting the selection of the at least one affordance for the predicted news article, the device displays the predicted news article on the touch-sensitive display (e.g., the device ceases to display the search interface with the search entry portion and the predictions portion and instead opens and displays the predicted news article within the browser module 147). For example, in response to detecting a single tap over the affordance corresponding to News 1, the device displays the news article corresponding to News 1 within the browser app 147 (e.g., browser module 147, Figure 1A). In this way, the user of the device is able to conveniently and quickly access a relevant (or desired) news article (e.g., the browser application) without having to enter any text in the search entry portion (thus saving time and frustration if the user had to enter text and was unable to locate the predicted news article).

[0202] In some embodiments, the predicted / suggested content items that are included in the search interface (e.g., in conjunction with methods 600 and 800, or any of the other methods discussed herein) are selected based on techniques that are discussed below in reference to Sections 1-11.

[0203] It should be understood that the particular order in which the operations in Figures 8A-8B have been described is merely one example and is not intended to indicate that the described order is the only order in which the operations could be performed. One of ordinary skill in the art would recognize various ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein with respect to other methods described herein (e.g., method 600) are also applicable in an analogous manner to method 800 described above with respect to Figures 8A-8B. For example, the user interface objects described above with reference to method 800 optionally have one or more of the characteristics of the user interface objects described herein with reference to other methods described herein (e.g., method 600). In some embodiments, any relevant details from Sections 1-11 may be utilized for any suitable purpose in conjunction with method 800. For brevity, these details are not repeated here.

[0204] Figures 10A-10C illustrate a flowchart representation of a method 1000 of proactively suggesting search queries based on content currently being displayed on an electronic device with a touch-sensitive display, in accordance with some embodiments. Figures 11A-11J are used to illustrate the methods and / or processes of Figures 10A-10C. Although some of the examples which follow will be given with reference to inputs on a touch-sensitive display (in which a touch-sensitive surface and a display are combined), in some embodiments, the device detects inputs on a touch-sensitive surface 195 that is separate from the display 194, as shown in Figure 1D.

[0205] In some embodiments, the method 1000 is performed by an electronic device (e.g., portable multifunction device 100, FIG. 1A, configured in accordance with any one of Computing Device A-D, Figure 1E) and / or one or more components of the electronic device (e.g., I / O subsystem 106, operating system 126, etc.). In some embodiments, the method 1000 is governed by instructions that are stored in a non-transitory computer-readable storage medium and that are executed by one or more processors of a device, such as the one or more processors 122 of device 100 (Figure 1A). For ease of explanation, the following describes method 1000 as performed by the device 100. In some embodiments, with reference to Figure 1A, the operations of method 1000 are performed by or use, at least in part, a proactive module (e.g., proactive module 163) and the components thereof, a contact / motion module (e.g., contact / motion module 130), a graphics module (e.g., graphics module 132), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 1000 are, optionally, combined and / or the order of some operations is, optionally, changed.

[0206] As described below, the method 1000 provides an intuitive way to proactively suggest relevant content (e.g., suggested search queries) on an electronic device with a touch-sensitive display. The method requires less touch inputs in order to perform a search on the electronic device (e.g., the user need only select a suggested search query and does not need to type any text), thereby creating a more efficient human-machine interface and allow users to quickly execute relevant searches. By providing suggested search queries, method 1000 also helps to ensure that users know that a proactive assistant is available on the device to assist with performing actions more quickly (thus, improving user satisfaction with their devices). For battery-operated electronic devices, the method 1000 both conserves power and increases the time between battery charges.

[0207] As shown in Figure 10A, the device displays (1002), on the display, content associated with an application that is executing on the electronic device. For example, as shown in Figure 11A, content associated with an email application that is executing on the electronic device 100 is displayed on the touch screen 112. The content at least includes the sender name and / or address of an email (e.g., "From: John Applecore"), the subject text (e.g., "Where to next?"), and a body of the email. In some embodiments, the body of the email may include image 1108 and / or may text 1110.

[0208] While displaying the application, the device detects (1004), via the touch-sensitive surface, a swipe gesture that, when detected, causes the electronic device to enter a search mode that is distinct from the application. In some embodiments, detecting the swipe gesture includes (1006) detecting the swipe gesture over at least a portion of the content that is currently displayed. In some embodiments, the swipe gesture is used to invoke a search interface over the application (e.g., such as that shown in Figure 11B). In some embodiments, the swipe gesture is a first swipe gesture that is received over the application and is not received within any user interface field that is included in the content associated with the application (e.g., the first swipe gesture is not a tap within a search box that might be displayed in the application). In some embodiments, the first swipe gesture causes the electronic device to enter the search mode of the electronic device that is distinct from the application, the search mode including display of a search interface (e.g., such as the search interface shown in Figures 11B and 11D, and 11F-11J and discussed in greater detail below).

[0209] In some embodiments, the first swipe gesture is available at any time by swiping in a downward direction (and travelling at least a threshold distance (e.g., 2, 3, 4 cm.)) over the touch-sensitive display (e.g., the downward swipe 1102-1 and 1102-3 as shown in Figures 11A and 11E, respectively). In some embodiments, the swipe gesture is detected (e.g., the first swipe gesture discussed above) while the application is currently displayed on the touch-sensitive display and the swipe gesture is detected on top of the content that is currently displayed for the application. For example, in Figures 11A and 11E, the downward swipe gestures 1102-1 and 1102-3 are detected on top of the email content while the email application is currently displayed.

[0210] In some embodiments, a second swipe gesture, that also causes the device to enter the search mode, is also available at a later time (e.g., after exiting the application). In some embodiments, before detecting the swipe gesture, the device detects (1008) an input that corresponds to a request to view a home screen of the electronic device, and in response to detecting the input, the device ceases to display the content associated with the application and displays a respective page of the home screen of the electronic device. In some embodiments, the respective page is an initial page in a sequence of home screen pages (e.g., a first page in a sequence of home screen pages), and the swipe gesture is detected (e.g., the second swipe gesture) while the initial page of the home screen is displayed on the display.

[0211] For example, as shown in Figures 11A and 11E, the user exits the application and switches to viewing a home screen as shown in Figure 11C by tapping 1106 physical home button 204 of the device while the application is displayed. In Figure 11C, a first page of the home screen is displayed as indicated by the highlighted first dot 1112-1 of home screen page indicator and not highlighting the remaining dots 1112-2 of the home screen. While viewing the first page of the home screen, the user is able to provide the second swipe gesture by swiping in a substantially horizontal direction (e.g., a left-to-right direction shown for swipe gesture 1104-1 in Figure 11E). In response to receiving the second swipe gesture, the electronic device enters the search mode, including displaying a search interface on the touch-sensitive display (as discussed in greater detail below with reference to Figure 11D).

[0212] In response to detecting the swipe gesture, the device enters (1010) the search mode, the search mode including a search interface that is displayed on the display. Example search interfaces are shown in Figures 11B and 11D. In some embodiments, the search interface is displayed (1012) as translucently overlaying the application, e.g., search interface 1115 in Figure 11B. In some embodiments, the search interface is displayed as a translucent overlay over the application (e.g., as shown for search interface 1115 in Figure 11B). In some embodiments, the search interface 1115 is gradually displayed such that an animation of the search interface 1115 is played, e.g., fading in and / or transitioning in from one side. In Figure 11B, the search interface 1115 is displayed as translucently overlying the email application such that the email application remains partially visible beneath the search interface 1115 on the touch-sensitive display 112. In some embodiments, the search interface is displayed as translucently overlaying the home screen as shown in Figures 11G-11J in response to the second swipe gesture discussed above.

[0213] In some embodiments, the search interface further includes (1014) one or more trending queries, e.g., one or more trending terms that have been performed by members of a social network that is associated with the user. In some embodiments, the one or more trending queries include one or more trending terms that are based on (i) popular news items, (ii) a current location of the electronic device (e.g., if the user is visiting a location other than their home (such as Tokyo)), and / or (iii) items that are known to be of interest to tourists, etc.). For example, trending searches 1160 is shown as optional in Figures 11B and 11D, and the one or more trending terms include, e.g., "Patagonia", "Ecuador", "Mt. Rainier" etc. In some embodiments, the search interface also includes trending GIFs (e.g., based on emotive phrases, such as "Congrats!," in the content that lead people to want to share a GIF).

[0214] In conjunction with entering the search mode, the device determines (1018) at least one suggested search query based at least in part on information associated with the content. In some embodiments, this determination is conducted as the animation of the search interface 1115 is played, e.g., as the search interface 1115 is gradually revealed. In other embodiments, this determination is conducted before the swipe gesture is even received.

[0215] In some embodiments, in accordance with a determination that the content includes textual content, the device determines (1022) the at least one suggested search query based at least in part on the textual content. In some embodiments, determining the at least one suggested search query based at least in part on the textual content includes (1024) analyzing the textual content to detect one or more predefined keywords that are used to determine the at least one suggested search query. In some embodiments, the one or more predefined keywords are stored in one or more data structures that are stored on the electronic device, including a first data structure with at least 150,000 entries for predefined keywords. In this way, the device includes a number of common terms that can be quickly detected in content and then provided to the user as suggested search queries, and this is all done without requiring any input from the user at the search interface. In some embodiments, a second data structure of the one or more data structures is associated with a context kit that leverages the second data structure to identify a context for the content and then identify the at least one suggested search query based at least in part on the identified context for the content. In some embodiments, the second data structure is an on-device index (such as a Wikipedia index that is specific to the electronic device). In some embodiments, suggested search queries are determined using both the first and second data structures and then the suggested search queries are aggregated and presented to the user (e.g., within the search interface and before receiving any user input). In some embodiments, leveraging both the first and second data structures also allows the electronic device to help distinguish between businesses with the same name, but with different addresses / phones.

[0216] For example, in Figure 11A, the content associated with the email application includes textual content, such as the sender and / or recipient information, the subject line, and the text in email body, "I love Ecuador!" etc. Based at least in part on the textual content, the device determines at least one suggested search query and displays the search results as shown in Figure 11B, e.g., Ecuador, John Applecore, Guide Service, Cayambe, Antisana etc. The term "Ecuador" may be a predefined keyword stored on the electronic device as part of entries in the first data structure, while other entries may be identified based on a context for the content and using the second data structure while leveraging the first data structure.

[0217] In some embodiments, determining the at least one suggested search query includes (1026) determining a plurality of suggested search queries, and populating the search interface includes populating the search interface with the plurality of suggested search queries. As shown in Figure 11B, one suggested search query "Ecuador" is displayed in the suggested searches 1150. Optionally, as indicated by the dotted line in Figure 11B, in the suggested searches 1150 section, a plurality of suggested search queries, e.g., "John Applecore", "Guide Service", "Cayambe", and "Antisana" etc. in addition to "Ecuador" are displayed.

[0218] In some embodiments, in conjunction with entering the search mode, the device obtains (1036) the information that is associated with the content (before and / or after displaying the search interface) by using one or more accessibility features that are available on the electronic device. In some embodiments, an operating system of the electronic device does not have direct access to (or knowledge) of the content that is currently displayed in some applications on the electronic device (e.g., third-party applications developed by companies other than a provider of the operating system). As such, the operating system obtains information about the content by using APIs (e.g., accessibility APIs) and other features that are available on the electronic device and allow the operating system to learn about the content that is displayed within the third-party applications.

[0219] In some embodiments, using the one or more accessibility features includes (1038) using the one or more accessibility features to generate the information that is associated with the content by: (i) applying a natural language processing algorithm to textual content that is currently displayed within the application; and (ii) using data obtained from the natural language processing algorithm to determine one or more keywords that describe the content, and wherein the at least one suggested search query is determined based on the one or more keywords. (e.g., a natural language processing algorithm that is used to provide functions such as VoiceOver, Dictation, and Speak Screen that are available as the one or more accessibility features on the electronic device). In some embodiments, the information that is associated with the content includes information that is extracted from the content that is currently displayed in the application, including names, addresses, telephone numbers, instant messaging handles, and email addresses (e.g., extracted using the natural language processing algorithm discussed above).

[0220] In some embodiments, determining the one or more keywords that describe the content also includes (1040): (i) retrieving metadata that corresponds to non-textual content that is currently displayed in the application; and (ii) using the retrieved metadata, in addition to the data obtained from the natural language processing algorithm, to determine the one or more keywords. An example of non-textual content is an image that is displayed within the application (e.g., image 1108 in Figure 11A and one or more images 1114 in Figure 11E). In some embodiments, one or more informational tags (such as HTML tags, CSS descriptors, and other similar metadata) are associated with the image and can be used to help the one or more accessibility features learn about the image (e.g., one of the informational tags could describe a type of the image and / or provide details about what is displayed in the image).

[0221] In some embodiments (in particular when only non-textual content is displayed in the application), the natural language processing algorithm is not utilized and instead only the retrieved metadata is used to determine the one or more keywords. In some embodiments, inputs from the user that were previously provided in the application are also used to help determine the one or more keywords. For example, the user searches for a particular restaurant name in order to locate an address and / or telephone number and the name of that restaurant may also be used (e.g., even if the restaurant name is not currently displayed in the application and was only used as an earlier input or search query) to help determine the one or more keywords that describe the content.

[0222] Turning to Figure 10C, before receiving any user input at the search interface, the device populates (1020) the displayed search interface with the at least one suggested search query. In some embodiments, the search interface includes a search input portion (e.g., search entry portion 1120 at a top portion of the search interface 1115, Figures 11B, 11D, and 11F-11J) and a search results portion (e.g., search results portion 1130 directly below the search input portion 1120, Figures 11B, 11D, and 11F-11J) and the at least one suggested search query is displayed within the search results portion. For example, in Figure 11B, suggested searches 1150 include at least one suggested search query, e.g., "Ecuador", "John Applecore", "Guide Service", "Cayambe", "Antisana", and the at least one suggested query is displayed within the search results portion 1130.

[0223] In some embodiments, the first swipe gesture discussed above is available while any page of the home screen is displayed as well. For example, in addition to being able to use the first swipe gesture 1102-1 to enter the search mode over the application as shown in Figures 11A and 11B, the user may also use the first swipe gesture to enter the search mode over any page of the home screen. In Figure 11C, in response to swipe 1104-2 in a substantially vertical direction (e.g., downward), the device enters the search mode and displays the search interface 1105 as shown in Figure 11D. In this way, any time the user chooses to enter the search mode, the user is presented with relevant search queries that are related to content that was recently viewed in the application. Although Figure 11C illustrates detecting the swipe gesture 1104-2 over the first page of the home screen, as indicated by highlighting the first dot 1112-1 of home screen page indicator and not highlighting the remaining dots 1112-2 of the home screen page indicator, the swipe gesture 1104-2 can be detected over any page of the home screen, e.g., over a page over than the initial page of the home screen where one of the remaining dots 1112-2 is highlighted and the first dot 1112-1 is not highlighted.

[0224] In some embodiments, the device detects (1028), via the touch-sensitive surface, a new swipe gesture over new content that is currently displayed. In response to detecting the new swipe gesture, the device enters the search mode. In some embodiments, entering the search mode includes displaying the search interface on the display. In conjunction with entering the search mode and in accordance with a determination that the new content does not include textual content, in some embodiments, the device populates the search interface with suggested search queries that are based on a selected set of historical search queries from a user of the electronic device.

[0225] For example, after viewing the email content as shown in Figure 11A and exiting the search interface, the user viewed a picture of Mount Rainier 1114-2 (Figure 11E) along with a picture of Ecuador 1114-1 (Figure 11E). Both images do not include textual content. Subsequently, as shown in Figure 11E, a new swipe gesture 1102-3 is detected. In response to detecting the new swipe gesture 1102-3, the device enters the search mode and displays the search interface 1115 on the display as shown in Figure 11F. In Figure 11F, "Mount Rainier" is shown as a historical search query and displayed in recent searches 1155 section.

[0226] In some embodiments, the search interface is displayed (1130) with a point of interest based on location information provided by a second application that is distinct from the application. For example, continuing the above example, location information of Mt. Rainier is obtained by a second application, such as an imaging application, based on tags and / or metadata associated with the image. In response to the new swipe gesture 1102-3 (Figure 11E), the search interface 1115 is displayed with a point of interest, Mt. Rainier 1157-1 in suggested places section 1154, as shown in Figure 11F.

[0227] In some embodiments, the point of interest is displayed not just in response to a new swipe gesture over non-textual content. The point of interest can be displayed in response to a new swipe gesture over textual content. For example, in a scenario where the user was searching for restaurants in a first application (such as a YELP application), the user then switched to using a text messaging application (e.g., the application), the user then provided the swipe gesture over the text messaging application and, in response, the device pre-populates the search interface to include the point of interest (e.g., Best Sushi 1157-2, Figure 11F) as a suggested search query based on the user's earlier interactions with the first application.

[0228] In some embodiments, the search interface further includes (1032) one or more suggested applications. The suggested applications are applications that are predicted to be of interest to the user of the electronic device based on an application usage history associated with the user (application usage history is discussed above in reference to Figures 3A and 3B). In some embodiments, the set of historical search queries is selected based at least in part on frequency of recent search queries (e.g., based on when and how frequently each historical search query has been conducted by the user). For example, as shown in Figure 11D, based on the application usage history, applications, Health 242, Books 242, Maps 236 applications are suggested to the user in the suggested apps 1162 section. These application suggestions may be selected based at least in part on frequency of recent search queries. In some embodiments, an application that has not been installed on the electronic device is predicted to be of interest to the user. The name of the application 237 that has not been installed is displayed along with other suggested applications and a link to the application installation is provided.

[0229] In some embodiments, one or more suggested applications are displayed not just in response to a new swipe over non-textual content. For example, as shown in Figures 11D and 11E, in response to detecting the swipe gesture 1104-2 over the home screen (e.g., over any page of the home screen), suggested apps 1155 are optionally displayed in the search results portion 1130 of the search interface 1115.

[0230] Although Figures 11B, 11D, and 11F illustrate grouping the suggested search results into categories and displaying the suggested searches in different sections of the search interface 1115, other display formats are shown to the user. For example, the suggested search results can be blended. As shown in Figure 9D, point of interests, suggested places, recent searches, and suggested applications are displayed together in "My Location & Recently Viewed." In some embodiments, blending the suggested searches is performed in accordance with a set of predefined rules. For example, up to a number of search results spots (e.g., 8) can from each of the sources that contribute to the suggested searches. A predetermined order of precedence is used to determine the order of the suggested searches (e.g., connections, historical, then uninstalled hero assets). In another example, a predetermined set of rules includes: (i) for each type of suggested search results, it has a position and a maximum number of results it can contribute; (ii) for certain types of suggested search results, (e.g., applications that have not been installed), a maximum number of results can contribute to the blended results (e.g., each contribute 1); (iii) or for historical results, it is up to the user. For example, in some embodiments, the set of historical search queries is selected (1034) based at least in part on frequency of recent search queries. (e.g., based on when and how frequently each historical search query has been conducted by the user).

[0231] In some embodiments, only one or more suggested applications that are predicted to be the most of interest to the user are displayed in response to a search activation gesture. For example, in response to receiving a search activation gesture (e.g., swipe 1104 in Figure 11C), the device enters the search mode and displays a translucent search interface on the touch-sensitive display as shown in Figures 11G-11J. The search interface includes the search input portion 1120 and the search results portion 1130. For example, as shown in Figure 11G, suggesting applications are predicted to be of the most of interest to the user. Multiple applications are displayed in the search results portion 1130.

[0232] In some embodiments, the suggested application uses the location information to suggest content that is predicted to be of most of interest to the user. For example, in Figure 11H, "Find My Car" application is predicted to be the most of interest to the user. In the search results portion 1130, the user interface for "Find My Car" application is displayed. The application uses location information of the user to display a pin on the map and shows the relative position of the user to the car indicated by the dot. In another example, based on a user's location and / or other information described above (e.g., usage data, textual content, and / or non-textual content etc.), an application displaying nearby points of interest is predicted to be the most of interest to the user. In Figure 11I, the search results portion 1130 includes a point of interest, e.g., a restaurant within "Food" category named "Go Japanese Fusion". The "Food" category is highlighted as indicated in double circle and the nearby restaurant "Go Japanese Fusion" is located based on the user's location information and the location of the restaurant. In another example, as shown in Figure 11J, multiple points of interests within the "Food" category are predicted to be the most of interest to the user, and these points of interests, e.g., Caffe Macs, Out Steakhouse, and Chip Mexican Grill, within the food category are displayed and the "Food" category is highlighted.

[0233] It should be understood that the particular order in which the operations in Figures 10A-10C have been described is merely one example and is not intended to indicate that the described order is the only order in which the operations could be performed. One of ordinary skill in the art would recognize various ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein with respect to other methods described herein (e.g., methods 600 and 800) are also applicable in an analogous manner to method 1000 described above with respect to Figures 10A-10C. For example, the user interface objects (e.g., those displayed within the search interface) described above with reference to method 1000 optionally have one or more of the characteristics of the user interface objects described herein with reference to other methods described herein (e.g., methods 600 and 800). In some embodiments, aspects of method 1000 are optionally interchanged or supplemented by aspects of method 1200 discussed below (and vice versa). In some embodiments, any relevant details from Sections 1-11 may be utilized for any suitable purpose in conjunction with method 1000. For brevity, these details are not repeated here.

[0234] Figure 12 illustrates a flowchart representation of a method 1200 of entering a search mode, in accordance with some embodiments. Figures 13A-13B are used to illustrate the methods and / or processes of Figure 12. Although some of the examples which follow will be given with reference to inputs on a touch-sensitive display (in which a touch-sensitive surface and a display are combined), in some embodiments, the device detects inputs on a touch-sensitive surface 195 that is separate from the display 194, as shown in Figure 1D.

[0235] In some embodiments, the method 1200 is performed by an electronic device (e.g., portable multifunction device 100, FIG. 1A, configured in accordance with any one of Computing Device A-D, Figure 1E) and / or one or more components of the electronic device (e.g., I / O subsystem 106, operating system 126, etc.). In some embodiments, the method 1200 is governed by instructions that are stored in a non-transitory computer-readable storage medium and that are executed by one or more processors of a device, such as the one or more processors 122 of device 100 (Figure 1A). For ease of explanation, the following describes method 1200 as performed by the device 100. In some embodiments, with reference to Figure 1A, the operations of method 1200 are performed by or use, at least in part, a proactive module (e.g., proactive module 163) and the components thereof, a contact / motion module (e.g., contact / motion module 130), a graphics module (e.g., graphics module 132), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 1200 are, optionally, combined and / or the order of some operations is, optionally, changed.

[0236] As described below, the method 1200 provides an intuitive way to proactively suggest relevant content (e.g., suggested search queries or affordances with content relevant to a user's current location) on an electronic device in response to where a gesture is received. The method allows users to efficiently identify and select desired content with a minimal number of user inputs, thereby creating a more efficient human-machine interface (e.g., the device provides suggested search queries and content for nearby points of interest and the user need only select these, without having to search and locate them). For battery-operated electronic devices, proactively identifying and surfacing relevant content faster and more efficiently both conserves power and increases the time between battery charges.

[0237] As shown in Figure 12, the device detects (1202), via the touch-sensitive surface, a swipe gesture over a user interface. In some embodiments, the swipe gesture, when detected, causes the electronic device to enter a search mode. In response to detecting the swipe gesture, the device enters the search mode. In some embodiments, entering the search mode includes populating a search interface distinct from the user interface, before receiving any user input within the search interface (e.g., no text is entered into a search box within the search interface, no input is received within the search box (no tap within the search box), etc.), with a first content item.

[0238] In some embodiments, in accordance with a determination that the user interface includes content that is associated with an application that is distinct from a home screen that includes selectable icons for invoking applications (and, therefore, the swipe gesture was detected over the app-specific content), the device populates the search interface with the first content item includes populating the search interface with at least one suggested search query that is based at least in part on the content that is associated with the application. For example, as explained above with reference to Figures 11A-11B in response to the swipe gesture 1102 over email application with content of "John Applecore", Ecuador image, and / or "I love Ecuador" text (Figure 11A), the search interface 1115 is populated (Figure 11B). The search interface 1115 includes at least one suggested search query, e.g., "Ecuador", "John Applecore" based at least in part on the content associated with the email application. In another example, as explained above with reference to Figures 11E-11F, in response to the swipe gesture 1102 over image application with content of Ecuador and / or Mt. Rainier image (Figure 11E), the search interface 1115 is populated (Figure 11F). The search interface 1115 includes at least one suggested search query, e.g., "Ecuador", "Mount Rainier" based at least in part on the image content.

[0239] In some embodiments, in accordance with a determination that the user interface is associated with a page of the home screen (e.g., swipe gesture was over an initial home screen page, Figure 11C), populating the search interface with the first content item includes populating the search interface with an affordance that includes a selectable description of at least one point of interest that is within a threshold distance of a current location of the electronic device. For example, when the device is close to a mall with some restaurants, display information about those restaurants instead of suggested search queries, since the information about the restaurants is predicted to be of most of interest to the user based on the user's proximity to the mall. In the example explained above with reference to Figures 11I and 11J, in response to detecting the swipe gesture 1104 over the home screen (Figure 11C), instead of displaying the suggested search queries interface as shown in Figure 11D, at least one nearby point of interest is displayed in the search results portion 1130 of the search interface, e.g., "Go Japanese Fusion" restaurant (Figure 11I), "Caffe Macs", "Out Steakhouse", "Chip Mexican Grill" (Figure 11J). In Figure 11I and 11J, each point of interest includes an affordance and includes a selectable description, which upon selected, provides more information about the point of interest, e.g., selecting the icon and or the description of the point of interest provides more description, pricing, menu, and / or distance information.

[0240] In some embodiments, the decision as to whether to populate the search interface with suggested search queries or with an affordance for a nearby point of interest is additionally or alternatively based on whether a predetermined period of time has passed since displaying the content for the application. For example, in accordance with a determination that (i) the swipe gesture was detected over a home screen page (e.g., swipe gesture was note detected over the content) and (ii) a period of time since displaying the content that is associated with the application is below a threshold period of time, the search interface is still populated with the at least one suggested search query. Therefore, in such embodiments, the determination that the swipe gesture was not detected over the content includes a determination that the period of time since displaying the content meets or exceeds the threshold period of time (e.g., if the content was viewed too long ago, 2 minutes, 3 minutes ago) then the device determines that the user is not likely to be interested in suggested search queries based on that content and, instead, the search interface is populated with the affordance that includes the selectable description of the at least one point of interest. In this way, the user is still provided with suggested search queries if the device determines that the content that is associated with the application was recently displayed.

[0241] In some embodiments, populating the search interface with the affordance includes (1204) displaying a search entry portion of the search interface. In some embodiments, the device detects (1206) an input at the search entry portion; and in response to detecting the input (e.g., a tap within) the search entry portion, the electronic device ceases to display the affordance and display the at least one suggested search query within the search interface. For example, as shown in Figure 13A, the search interface includes a search entry portion 1120 and a search results portion 1130 with at least one affordance for nearby points of interest (e.g., nearby restaurants as shown in Figure 13A and selectable categories of interest for other nearby points of interest). While displaying the search interface with nearby point of interests, an input 1302 at the search entry portion 1120 is detected, e.g., the user taps within the search box with input 1302 as shown in Figure 13A. In response to detecting the input 1302, in Figure 13B, the device ceases to display the at least one affordance associated with the nearby points of interest and displays suggested search queries in the search results portion 1130, e.g., Ecuador, Mount Rainier, Best Sushi etc. Therefore, the device is able to quickly switch between suggested search queries and suggested points of interest (in this example, the users tap within the search box indicates that they are not interested in the suggested points of interest and, thus, the device attempts to provide a different type of suggested content, e.g., the suggested search queries based on content previously viewed in other applications).

[0242] Additional details regarding the selectable description of the at least one point of interest are provided below in reference to Figures 16A-16B and 17A-17E. Additional details regarding populating the search interface with the at least one suggested search query are provided above in reference to Figures 10A-10C and 11A-11J.

[0243] It should be understood that the particular order in which the operations in Figure 12 have been described is merely one example and is not intended to indicate that the described order is the only order in which the operations could be performed. One of ordinary skill in the art would recognize various ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein with respect to other methods described herein (e.g., methods 600, 800, 1000) are also applicable in an analogous manner to method 1200 described above with respect to Figure 12. For example, the user interface objects and / or operations described above with reference to method 1200 optionally have one or more of the characteristics of the user interface objects and / or operations described herein with reference to other methods described herein (e.g., methods 600, 800, and 1000). In some embodiments, any relevant details from Sections 1-11 may be utilized for any suitable purpose in conjunction with method 1200. For brevity, these details are not repeated here.

[0244] Figure 14 illustrates a flowchart representation of a method 1400 of proactively providing vehicle location information on an electronic device with a touch-sensitive display, in accordance with some embodiments. Figures 15A-15B are used to illustrate the methods and / or processes of Figure 14. Although some of the examples which follow will be given with reference to inputs on a touch-sensitive display (in which a touch-sensitive surface and a display are combined), in some embodiments, the device detects inputs on a touch-sensitive surface 195 that is separate from the display 194, as shown in Figure 1D.

[0245] In some embodiments, the method 1400 is performed by an electronic device (e.g., portable multifunction device 100, FIG. 1A, configured in accordance with any one of Computing Device A-D, Figure 1E) and / or one or more components of the electronic device (e.g., I / O subsystem 106, operating system 126, etc.). In some embodiments, the method 1400 is governed by instructions that are stored in a non-transitory computer-readable storage medium and that are executed by one or more processors of a device, such as the one or more processors 122 of device 100 (Figure 1A). For ease of explanation, the following describes method 1400 as performed by the device 100. In some embodiments, with reference to Figure 1A, the operations of method 1400 are performed by or use, at least in part, a proactive module (e.g., proactive module 163) and the components thereof, a contact / motion module (e.g., contact / motion module 130), a graphics module (e.g., graphics module 132), one or more location sensors (e.g., accelerometer(s) 168, a magnetometer and / or a GPS receiver), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 1400 are, optionally, combined and / or the order of some operations is, optionally, changed.

[0246] As described below, the method 1400 provides an intuitive way to proactively provide location information when users are in immediate need of that information. The method creates more efficient human-machine interfaces by proactively providing the vehicle location information without requiring users to attempt to locate the information themselves and by providing the information at a time when the user is determined to be returning to a parked vehicle. For battery-operated electronic devices, method 1400 both conserves power and increases the time between battery charges.

[0247] As shown in Figure 14, the device automatically, and without instructions from a user, performs (1402) steps 1404 and 1406 described below. In step 1404, the device determines that a user of the electronic device is in a vehicle that has come to rest at a geographic location.

[0248] In some embodiments, determining that the vehicle has come to rest at the geographic location includes determining that the electronic device has remained at the geographic location for more than a threshold period of time, e.g., the device is in one spot for approximately 2 minutes after having travelled above the threshold speed, so this gives an indication that the vehicle is now parked. In some embodiments, determining that the vehicle has come to rest at the geographic location includes determining that a communications link between the electronic device and the vehicle has been disconnected, e.g., the device losing Bluetooth connection with vehicle and / or the user removing a cable connecting the device with the vehicle, etc., thus providing an indication that the vehicle is stopped and / or the engine of the vehicle has been turned off. In some embodiments, determining that the vehicle has come to rest at the geographic location includes determining that the geographic location corresponds to a location within a parking lot, e.g., plug current GPS coordinates into (or send to) a maps application to make this determination and get back a determination as to whether the geographic location is in a parking lot.

[0249] In some embodiments, only one or more of the above determinations is conducted in order to determine whether the vehicle has come to rest at the geographic location, in other embodiments two or more of the determinations are conducted while, in still other embodiments, all three of the determinations are conducted in order to assess whether the vehicle has come to rest at the geographic location. For example, in some embodiments, determining that the user is in the vehicle that has come to rest at the geographic location includes (i) determining that the user is in the vehicle by determining that the electronic device is travelling above a threshold speed as described above, (ii) determining that the vehicle has come to rest at the geographic location by one or more of: (a) determining that the electronic device has remained at the geographic location for more than a threshold period of time as described above, (b) determining that a communications link between the electronic device and the vehicle has been disconnected as described above, and (c) determining that the geographic location corresponds to a location within a parking lot as described above.

[0250] In step 1406, the device further determines whether the user has left the vehicle. In some embodiments, the device makes the determination by determining that a current position of the device is more than a threshold distance away from the geographic location. In some embodiments, the device makes the determination by determining that the user has physically untethered the device from a connection with the vehicle or the user has broken a wireless connection between the device and the vehicle (e.g., Bluetooth or WiFi based connection). Additional details regarding determinations that are used to establish (with a high enough confidence) that the user has left the vehicle at the geographic location are provided below.

[0251] Upon determining that the user has left the vehicle at the geographic location, the device determines (1408) whether positioning information, retrieved from the location sensor to identify the geographic location, satisfies accuracy criteria. In some embodiments, the accuracy criteria include a criterion that is satisfied when accuracy of a GPS reading associated with the positioning information is above a threshold level of accuracy (e.g., 10 meters or less circular error probability).

[0252] Upon determining that the positioning information does not satisfy the accuracy criteria (1408 - No), the device provides (1410) a prompt to the user to input information about the geographic location, and in response to providing the prompt, the device receives information from the user about the geographic location and store the information as vehicle location information. In some embodiments, the prompt is an audio prompt provided by a virtual assistant that is available via the electronic device. When the prompt is an audio prompt, receiving the information from the user includes receiving a verbal description from the user that identifies the geographic location. In some embodiments, the prompt from the virtual assistant instructs the user to take a photo of the vehicle at the geographic location and / or to take a photo of the area surrounding the vehicle. In some embodiments, the user is instructed to provide a verbal description of the geographic location.

[0253] In some embodiments, upon determining that the positioning information satisfies the accuracy criteria (1408 - Yes), the device automatically, and without instructions from a user, stores (1412) the positioning information as the vehicle location information. In some embodiments, if the positioning information is accurate enough (e.g., satisfies the accuracy criteria), then no prompt is provided to the user. In other embodiments, even if the positioning information is accurate enough, the device still prompts the user to provide additional details regarding the geographic location (verbal, textual, or by taking a picture, as explained above in reference to operation 1410), in order to save these additional details and present them to the user if, for example, the device does not have a strong GPS signal at the time when the user is returning to their vehicle.

[0254] In some embodiments, the device further determines (1414) whether the user is heading towards the geographic location. In some embodiments, determining whether the user is heading towards the geographic location includes using new positioning information received from the location sensor to determine that the electronic device is moving towards the geographic location. In some embodiments, determining whether the user is heading towards the geographic location includes: (i) determining that the electronic device remained at a different geographic location for more than a threshold period of time (e.g., at a location / position associated with a shopping mall, a restaurant, a known home or work address for the user, etc.); and (ii) determining that the new positioning information indicates that the electronic device is moving away from the different geographic location and towards the geographic location. In some embodiments, the device additionally or alternatively compares a picture taken of the geographic location to an image of the user's current location in order to determine whether the user is heading towards the geographic location (e.g., by recognizing common or overlapping visual elements in the images). In some embodiments, the device additionally or alternatively detects that the user is accessing a settings user interface that allows the user to establish or search for a data connection with the vehicle and, in this way, the device has an indication that the user is heading towards the geographic location.

[0255] In some embodiments, in accordance with a determination that the user is heading towards the geographic location, the device displays a user interface object that includes the vehicle location information. In some embodiments, the user interface object is a maps object that includes an identifier for the user's current location and a separate identifier for the geographic location. For example, as shown in Figure 15A, the search user interface includes the search input portion 1120 and the search results portion 1130, which is a map object that includes the vehicle location information at the geographic location identified by a dot and the location label "Infinite Loop 2" and the user's current location separately identified by a pin.

[0256] In some embodiments, the user interface object is displayed on a lock screen of the electronic device. For example, as shown in Figure 15B, the map object is displayed on the lock screen. Thus, automatically, and without instructions from a user, the device predicts that finding the car is predicted to be of interest to the user based on relatively accurate location information and provides the map indicating the car location without the user unlocking the electronic device.

[0257] In some embodiments, the user interface object is displayed in response to a swipe gesture that causes the electronic device to enter a search mode. In some embodiments, determining whether the user is heading towards the geographic location is performed in response to receiving the same swipe gesture. Thus, the same swipe gesture causes the device to determine that the user is heading towards the geographic location and displays the user interface object based on relatively accurate location information.

[0258] In some embodiments, the search mode includes displaying a search interface that is pre-populated to include the user interface object, e.g., a maps object that includes an identifier that corresponds to the geographic location. In other words, before receiving any user input from the user within the search interface (e.g., before the user has enter any search queries), the search interface is populated to include the maps object, so that the user is provided with quick access to a visual reminder as to the geographic location at which they parked their vehicle (e.g., user interface object 1130 or user interface object 1525 or both, Figure 15A). In some embodiments, the swipe gesture is in a substantially left-to-right direction, and the swipe gesture is provided by the user while the electronic device is displaying an initial page of a home screen (e.g., 1104-1 in Figure 11C). In some circumstances, the swipe gesture is in a substantially downward direction and is provided by the user while viewing content that is associated with an application (e.g., 1102 in Figure 11A and 11E).

[0259] In some embodiments, in conjunction with determining that the user is heading towards to the geographic location (as discussed above in reference to operation 1414), the device also determines whether a current GPS signal associated with the location sensor of the electronic device is strong enough to allow the device to provide accurate directions back to the geographic location and, in accordance with a determination that the GPS signal is not strong enough, then the device provides both the positioning information and the additional details from the user, so that the user can rely on both pieces of information to help locate their parked vehicle.

[0260] In some embodiments, the prompt is an audio prompt provided by a virtual assistant that is available via the electronic device (as discussed above in reference to operation 1410), receiving the information from the user includes receiving a verbal description from the user that identifies the geographic location, and displaying the user interface object includes displaying a selectable affordance (e.g., affordance 1502, Figures 15A-15B) that, when selected, causes the device to playback the verbal description. In some embodiments, the prompt from the virtual assistant instructs the user to take a photo of the vehicle at the geographic location and / or to take one or more photos / videos of the area surrounding the vehicle and displaying the user interface object includes displaying a selectable affordance (e.g., the affordance 1502, Figure 15A-15B) that, when selected, causes the device to playback the recorded media. In some embodiments, the selectable affordance is displayed proximate to a maps object (as shown for affordance 1502), while in other embodiments, the selectable affordance is displayed by itself (in particular, in circumstances in which the positioning information did not satisfy the accuracy criteria, one example of this other display format is shown for affordance 1535, Figures 15A-15B). In some embodiments (depending on whether positioning information in addition to user-provided location information has been provided), one or both of the affordances 1130 and 1535 are displayed once it is determined that the user is heading towards their parked vehicle.

[0261] In some embodiments, the user interface object / affordance (e.g., 1130, 1535, or both) includes an estimated distance to reach the parked vehicle (e.g., the user interface object 1130 includes "0.3 mi" in the upper right corner).

[0262] In some embodiments, the prompt is displayed on the display of the electronic device, receiving the information from the user includes receiving a textual description from the user that identifies the geographic location, and displaying the user interface object includes displaying the textual description from the user. In other embodiments, a selectable affordance is displayed that allows the user to access the textual description. For example, in response to a selection of the affordance 1535 (Figures 15A-15B), the device opens up a notes application that includes the textual description from the user.

[0263] It should be understood that the particular order in which the operations in Figure 14 have been described is merely one example and is not intended to indicate that the described order is the only order in which the operations could be performed. One of ordinary skill in the art would recognize various ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein with respect to other methods described herein (e.g., methods 600, 800, 1000, and 1200) are also applicable in an analogous manner to method 1400 described above with respect to Figure 14. For example, the user interface objects described above with reference to method 1400 optionally have one or more of the characteristics of the user interface objects described herein with reference to other methods described herein (e.g., methods 600, 800, 1000, and 1200). Additionally, the details, operations, and data structures described below in reference to Sections 1-11 may also be utilized in conjunction with method 1400 (e.g., details discussed in reference to Section 6 may be used to help determine when to present user interface objects that include a location of a user's parked vehicle, details discussed in reference to Section 5 may be used to help identify and learn user patterns that relate to when a user typically parks their vehicle and then returns later, and details related to Section 10 may be utilized to help improve vehicle location information by relying on contextual information). In some embodiments, any other relevant details from Sections 1-11 may be utilized for any suitable purpose in conjunction with method 1400. For brevity, these details are not repeated here.

[0264] Figures 16A-16B illustrate a flowchart representation of a method 1600 of proactively providing information about nearby points of interest (POI), in accordance with some embodiments. Figures 17A-17E are used to illustrate the methods and / or processes of Figures 16A-16B. Although some of the examples which follow will be given with reference to inputs on a touch-sensitive display (in which a touch-sensitive surface and a display are combined), in some embodiments, the device detects inputs on a touch-sensitive surface 195 that is separate from the display 194, as shown in Figure 1D.

[0265] In some embodiments, the method 1600 is performed by an electronic device (e.g., portable multifunction device 100, FIG. 1A, configured in accordance with any one of Computing Device A-D, Figure 1E) and / or one or more components of the electronic device (e.g., I / O subsystem 106, operating system 126, etc.). In some embodiments, the method 1600 is governed by instructions that are stored in a non-transitory computer-readable storage medium and that are executed by one or more processors of a device, such as the one or more processors 122 of device 100 (Figure 1A). For ease of explanation, the following describes method 1600 as performed by the device 100. In some embodiments, with reference to Figure 1A, the operations of method 1600 are performed by or use, at least in part, a proactive module (e.g., proactive module 163) and the components thereof, a contact / motion module (e.g., contact / motion module 130), a graphics module (e.g., graphics module 132), one or more location sensors (e.g., accelerometer(s) 168, a magnetometer and / or a GPS receiver), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 1600 are, optionally, combined and / or the order of some operations is, optionally, changed.

[0266] As described below, the method 1600 proactively provides point-of-interest information on an electronic device without requiring the user to search for and locate that information themselves (and then surfaces that information when the user is within a certain distance of a particular POI). The method thus creates more efficient human-machine interfaces by requiring less touch inputs in order to perform a desired action (e.g., viewing information about nearby POIs). For battery-operated electronic devices, the method 1600 both conserves power and increases the time between battery charges.

[0267] As shown in Figure 16A, without receiving any instructions from a user of the electronic device, the device monitors (1602), using the location sensor, a geographic position of the electronic device. Also without receiving any instructions from the user of the electronic device, the device determines, based on the monitored geographic position, that the electronic device is within a threshold distance of a point of interest of a predetermined type (e.g., a point of interest for which activity suggestions are available, such as a restaurant, an amusement park, or a movie theatre).

[0268] In some embodiments, points of interest of the predetermined type are determined based on points of interest that the user frequently visits. In some embodiments, the points of interest also include points of interest that are predicted to be of interest to the user based on current text messages, emails, and / or other data associated with the user's social network.

[0269] Still without receiving any instructions from the user of the electronic device, in accordance with determining that the electronic device is within the threshold distance of the point of interest, the device identifies at least one activity that is currently popular at the point of interest, and retrieves information about the point of interest, including retrieving information about at least one activity that is currently popular at the point of interest (e.g., rides that are currently popular, menu items that are popular, movies that are popular, and the like). In some embodiments, popularity is assessed based on whether a threshold number (e.g., more than 5) or a threshold percentage (e.g., 5% or 10%) of individuals in the user's social network have posted something that is related to the at least one activity. In some embodiments, the device maintains a list of a predetermined number (e.g., 5, 10, or 20) of points of interest that the user often visits (and / or points of interest that are determined to be of interest right now based on text messages, emails, or activity within the user's social network, as discuss above) and the device retrieves information about current activities at those points of interest when the user is within the threshold distance (e.g., 1 mile, 1.5 miles, 2 miles) of any of them.

[0270] Still referring to Figure 16A, after retrieving the information about the point of interest, the device detects (1616), via the touch-sensitive surface, a first input that, when detected, causes the electronic device to enter a search mode. In some embodiments, the search mode is a system-level search mode that allows for conducting a search across the entire electronic device (e.g., across applications and content sources (both on-device and elsewhere), not just within a single application). In some embodiments, the first input corresponds to a swipe gesture (e.g., swipe gesture 1104-1 in a substantially left-to-right, Figure 11C) direction across the touch-sensitive surface that is received while the device is displaying an initial page of a home screen.

[0271] In some embodiments, in accordance with determining that the device is within the threshold distance of the point of interest, the device also displays an affordance, on a lock screen, the affordance indicating that information is available about current activities at the point of interest. In these embodiments, the first input corresponds to a request to view the available information about the current activities at the point of interest. For example, as shown in Figure 17D, the restaurant information object is displayed on the lock screen. The icon and / or description of the restaurant are selectable and indicate that more information, such as menu information is available about the restaurant. In response to a first input, e.g., a tap on the "View Menu" link, the menu is displayed (e.g., directly on the lock screen or by unlocking the device and opening an appropriate application for viewing of the menu). In some embodiments, any of the user interface objects / affordances shown in Figures 17A-17E (e.g., 1713 and 1715, and the content included therein) may be presented within the search interface or within the lock screen (or both).

[0272] Turning to Figure 16B, in response to detecting the first input, the device enters (1618) the search mode. In some embodiments, entering the search mode includes, before receiving any user input at the search interface (e.g., no search terms have been entered and no input has been received at a search box within the search interface), presenting, via the display, an affordance that includes (i) the information about the at least one activity and (ii) an indication that the at least one activity has been identified as currently popular at the point of interest, e.g., popular menu items at a nearby restaurant (e.g., affordance 1715 in Figures 17C-17D), ride wait times at a nearby amusement park (e.g., affordance 1713 in Figures 17A-17B), current show times at a nearby movie theatre, etc.

[0273] For example, as shown in Figure 17A, in some embodiments, the point of interest is (1604) an amusement park and the retrieved information includes current wait times for rides at the amusement park. In some embodiments and as shown in Figure 17A, the electronic device uses the retrieved information to present an average wait time (e.g., 1 hr) for all rides and the user is able to select a link in order to view wait times for each individual ride. As shown in Figure 17B, in some embodiments, the portion of the retrieved information includes (1606) information about wait times for rides that are located within a predefined distance of the electronic device, e.g., three rides / games are within a distance of approximately 100-150 feet from the electronic device and the wait time for each ride / game is displayed (after receiving an input from the user requesting to view the ride wait times, such as an input over the "View Wait Times" text shown in Figure 17A).

[0274] As another example, as show in Figure 17C, the point of interest is (1608) a restaurant and the retrieved information includes information about popular menu items at the restaurant. In some embodiments, the retrieved information is retrieved (1610) from a social network that is associated with the user of the electronic device. For example, in Figure 17C, popular menu item "Yakiniku Koji" at the restaurant "Go Japanese Fusion" is displayed within the affordance 1715, and the popular menu item may be determined based on information retrieved from a social network that is associated with the user of the electronic device.

[0275] As one additional example, the point of interest may be (1612) a movie theatre and the retrieved information includes information about show times for the movie theatre. In some embodiments, the retrieved information about the show times is retrieved (1614) from a social network that is associated with the user of the electronic device (e.g., based on information that has recently been posted by individuals in the user's social network).

[0276] In some embodiments, the device detects (1620) a second input, e.g., selection of a show more link that is displayed near (e.g., above) the affordance, such as the show more link shown for affordances 1713 and 1715 in Figures 17A-17D), and in response to detecting the second input, the device updates the affordance to include available information about current activities at a second point of interest, distinct from the point of interest. In some embodiments, the second point of interest is also within the threshold distance of the electronic device. For example, in response to a user selecting the show more link shown in Figure 17D, the device updates the affordance 1715 to include available information about restaurants and food at a different restaurant "Out Steakhouse" within 1 mile of the electronic device, as shown in Figure 17C. Stated another way, the affordance 1715 is initially presented with just the information about "Go Japanese Fusion" and, in response to the second input, the affordance 1715 is updated to include the information about the second point of interest (e.g., the information about "Out Steakhouse," shown within dotted lines in Figure 17C). In some embodiments, more than one points of interest distinct from the point of interest are displayed in response to detecting the second input, e.g., the device updates the restaurant information affordance to include available information about two or more new restaurants in addition to the point of interest. In some embodiments, the same functionality (i.e., the functionality allowing users to view information about additional points of interest in response to selection of the show more link) is also available for affordances presented on a lock screen (e.g., affordance 1715 shown on the lock screen, Figure 17D).

[0277] In some embodiments, the affordance further includes (1622) selectable categories of points of interest and the device detects (1624) a selection of a respective selectable category, and in response to detecting the selection, updates the affordance to include information about additional points of interest that are located within a second threshold distance of the device, e.g., the second threshold is greater than the threshold distance, in order to capture points of interest that might be of interest to the user, since they have not yet selected the closest points of interest. For example, the first threshold distance is 100 feet. The device displays "Go Japanese Fusion" as the point of interest as shown in Figures 17C and 17D when the electronic device is approximately 50 feet away from the point of interest. In response to detecting the selection of the "Food" category, as shown in Figure 17E, additional points of interest, e.g., "Out Steakhouse" and "Chip Mexican Grill" that are located more than 100 feet but within 1 mile of the device are displayed.

[0278] In some embodiments, after unlocking the electronic device, the user interface object is (1626) available in response to a swipe in a substantially horizontal direction (e.g., the left-to-right swipe 1104-1, Figure 11C) over an initial page of a home screen of the electronic device.

[0279] It should be understood that the particular order in which the operations in Figures 16A-16B have been described is merely one example and is not intended to indicate that the described order is the only order in which the operations could be performed. One of ordinary skill in the art would recognize various ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein with respect to other methods described herein (e.g., methods 600, 800, 1000, 1200, and 1400) are also applicable in an analogous manner to method 1600 described above with respect to Figure 16. For example, the user interface objects and / or operations described above with reference to method 1600 optionally have one or more of the characteristics of the user interface objects and / or operations described herein with reference to other methods described herein (e.g., methods 600, 800, 1000, 1200, and 1400). In some embodiments, any relevant details from Sections 1-11 may be utilized for any suitable purpose in conjunction with method 1600. For brevity, these details are not repeated here.

[0280] Figures 18A-18B are a flowchart representation of a method 1800 of extracting a content item from a voice communication and interacting with the extracted content item, in accordance with some embodiments. Figures 19A-19F are used to illustrate the methods and / or processes of Figures 18A-18B. Although some of the examples which follow will be given with reference to inputs on a touch-sensitive display (in which a touch-sensitive surface and a display are combined), in some embodiments, the device detects inputs on a touch-sensitive surface 195 that is separate from the display 194, as shown in Figure 1D.

[0281] In some embodiments, the method 1800 is performed by an electronic device (e.g., portable multifunction device 100, FIG. 1A, configured in accordance with any one of Computing Device A-D, Figure 1E) and / or one or more components of the electronic device (e.g., I / O subsystem 106, operating system 126, etc.). In some embodiments, the method 1800 is governed by instructions that are stored in a non-transitory computer-readable storage medium and that are executed by one or more processors of a device, such as the one or more processors 122 of device 100 (Figure 1A). For ease of explanation, the following describes method 1800 as performed by the device 100. In some embodiments, with reference to Figure 1A, the operations of method 1800 are performed by or use, at least in part, a proactive module (e.g., proactive module 163) and the components thereof, a contact / motion module (e.g., contact / motion module 130), a graphics module (e.g., graphics module 132), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 1800 are, optionally, combined and / or the order of some operations is, optionally, changed.

[0282] As described below, the method 1800 provides an intuitive way to extract content items from voice communications and present them to a user on an electronic device with a touch-sensitive display. The method reduces the number of inputs required from a user (e.g., the device automatically extracts relevant information for contacts, locations, and events and prepares that information for storage and use on the device), thereby creating a more efficient human-machine interface and assisting users with adding new content items based on what is discussed on voice communications. For battery-operated electronic devices, this method helps to both conserves power and increases the time between battery charges.

[0283] As shown in Figure 18A, the device receives (1801) at least a portion of a voice communication, the portion of the voice communication including speech provided by a remote user of a remote device that is distinct from a user of the electronic device. In some embodiments, the voice communication is a live phone call, a live video call (e.g., a FaceTime call), or a recorded voicemail (1803). In some embodiments, the voice communication is a live telephone call (or FaceTime call) between the user and the remote user and, thus, the voice communication includes speech provided by both the user and the remote user. In other embodiments, the voice communication is a recorded voicemail sent by the remote user to the user, the recorded voicemail is delivered from the remote device to the electronic device via a telecommunications network, and the recorded voicemail is then stored on the electronic device for later playback.

[0284] In some embodiments, the portion of the voice communication is identified based on an instruction from the user of the electronic device (1805). For example, the portion is flagged by the user of the electronic device for analysis based on the user's selection of a hardware button (e.g., the user taps the hardware button, just as a volume button, and in response, the device begins to analyze a predefined amount of the voice communication (e.g., a previous 10, 9, 8, or 7 seconds) to detect / extract content items. In some embodiments, the button may also be a button that is presented for user selection on the display of the electronic device (e.g., a button that is displayed on a user interface similar to that shown in Figure 21B during the voice communication that includes the text "tap here to analyze this voice communication for new content").

[0285] In some embodiments, the instruction from the user corresponds to a verbal command that includes the phrase "hey Siri" (e.g., "hey Siri, please save that," or "hey Siri, please remember that," or "hey Siri, please grab the event details that were just mentioned" or the like). In some embodiments, the verbal instruction from the user is any predefined phrase that causes the device to begin analyzing the voice communication to detect new content (e.g., the phrase could be in some other language besides English or the phrase could include different words, such as "Siri, please analyze this call" or "Siri, please begin analyzing" or something to that effect).

[0286] In some embodiments, the device does not record or maintain any portion of the voice communication in persistent memory, instead the device analyzes just the portion of the voice communication (e.g., 10 seconds at a time) and then immediately deletes all recorded data and only saves content items extracted based on the analysis (as discussed in more detail below). In this way, extracted content items are made available to users, but the actual content of the voice communication is not stored, thus helping to preserve user privacy.

[0287] In some embodiments, the device analyzes (1807) the portion of the voice communication (e.g., the portion flagged by the user of a recorded voicemail or a live phone call between the user of the device and another remotely located user of a different device, or a portion that is identified automatically by the device as including new content for extraction) to detect content of a predetermined type. In some embodiments, analyzing the voice communication includes (1809): converting the spe...

Claims

1. A method, comprising: at an electronic device with a touch-sensitive display: executing (602), on the electronic device, an application in response to an instruction from a user of the electronic device; while (604) executing the application, collecting usage data, wherein the usage data at least includes one or more actions performed by the user within the application; (i) automatically (606), without human intervention, obtaining at least one trigger condition based on the collected usage data, wherein obtaining the at least one trigger condition based on the collected usage data includes identifying a pattern of user behavior based on the collected usage data, the pattern of user behavior indicating that the user performs a particular action when the trigger condition is present at the electronic device; (ii) associating (608) the at least one trigger condition with the particular action of the one or more actions performed by the user within the application; and upon (610) determining that the at least one trigger condition has been satisfied, providing an indication to the user that the particular action associated with the trigger condition is available and will be performed in response to detecting a first gesture, wherein: the indication includes a description (706) of the first gesture; providing (614) the indication includes displaying, on a lock screen of the touch-sensitive display, a user interface object (702) corresponding to the particular action associated with the trigger condition; the first gesture is a single tap at a first predefined area (706) of the user interface object; and a single tap within a second predefined area (708) of the user interface object, distinct from the first predefined area (706), causes the electronic device to perform the particular action while continuing to display the lock screen.

2. The method of claim 1, wherein the user interface object further includes an icon associated with the application.

3. The method of any one of claims 1-2, further comprising: detecting the first gesture at the user interface object; and in response to detecting the first gesture, displaying, on the touch-sensitive display, the application; and while displaying the application, causing the electronic device to perform the particular action.

4. The method of any one of claims 1-3, wherein: collecting the usage data includes collecting time of day information (340-1(d)) associated with the user's interaction with the application; identifying a pattern based on the collected usage data includes identifying, based at least on the collected time of day information, a window of time during which the particular action is performed by the user within the application; associating (608) the at least one trigger condition with the particular action of the one or more actions performed by the user within the application includes associating the identified window of time with the particular action, and determining that the at least one trigger condition has been satisfied includes determining that a current time of day is within a predetermined amount of time of the identified window of time associated with the particular action.

5. The method of any one of claims 1-4, wherein determining that the at least one trigger condition has been satisfied includes determining that the electronic device has been coupled with a second device, distinct from the electronic device.

6. The method of any one of claims 1-5, further comprising: before collecting the usage data, receiving permission, from the user via the electronic device, to collect the usage data.

7. The method of any one of claims 1-6, wherein determining that the at least one trigger condition has been satisfied includes determining that the electronic device has arrived at a location corresponding to a home or a work location associated with the user.

8. The method of claim 7, wherein the address associated with the user is a previouslyvisited address corresponding to the user's home or work address.

9. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system that is in communication with a touch-sensitive display, the one or more programs including instructions for performing the method of any one of claims 1-8.

10. An electronic device, comprising: a touch-sensitive display; one or more processors; and memory storing one or more programs including instructions which, when executed by the one or more processors, cause the electronic device to perform the method of any one of claims 1-8.