System and method for proactively identifying and displaying related content on touch-sensitive device

Abstract

Systems and methods for actively identifying and displaying related content on an electronic device having a touch-sensitive display are disclosed herein. In one aspect, the method includes executing an application on the electronic device in response to an instruction from a user of the electronic device. While executing the application, the method further includes collecting usage data. The usage data includes at least one or more actions performed by the user within the application. The method further includes automatically acquiring, without human intervention, at least one trigger condition based on the collected usage data, and associating the at least one trigger condition with a particular action of the one or more actions performed by the user within the application. Upon determining that the at least one trigger condition has been satisfied, the method includes providing, to the user, an indication that the particular action associated with the trigger condition is available.

Description

[0001] This application is a divisional application of the invention patent application with the application date of May 27, 2016, application number 201680044146.3, and invention name "System and method for actively identifying and displaying related content on a touch-sensitive device". Technical Field

[0002] Embodiments disclosed herein relate generally to electronic devices with touch-sensitive displays and, more particularly, to systems and methods for actively identifying and displaying relevant content on an electronic device in communication with a display and a touch-sensitive surface. Background Art

[0003] Handheld electronic devices with touch-sensitive displays are ubiquitous. Users of these widely used handheld electronic devices can now install a large number of applications on their devices and use these applications to help them perform daily activities more efficiently. But in order to access these applications, users must typically unlock their devices, locate the desired application (e.g., by navigating a home screen to locate an icon associated with the desired application or by searching for the desired application within a search interface), and then locate the desired function within the desired application. As a result, users often spend a significant amount of time locating the desired applications and the desired functions within these applications without being able to immediately execute (e.g., via a single touch input) the desired application and / or the desired function.

[0004] Furthermore, the large number of installed applications overwhelms users with a continuous stream of information, making it impossible for them to review the information immediately and thoroughly. As a result, users often want to return at a later point in time to view a specific piece of information they noticed earlier or use a specific piece of information at a later point in time. However, users often cannot locate or do not remember how to locate a specific piece of information.

[0005] Therefore, it is desirable to provide an intuitive and easy-to-use system and method for actively identifying and displaying relevant content (e.g., a particular piece of information) on an electronic device in communication with a display and a touch-sensitive surface. Summary of the invention

[0006] Therefore, there is a need for electronic devices with faster and more efficient methods and interfaces for quickly accessing applications and the required functions within these applications. In addition, electronic devices are needed to help users manage the continuous information stream they receive every day by actively identifying and providing relevant information (e.g., contacts, nearby places, applications, news articles, addresses, and other available information on the device) before the user explicitly requests information. Such methods and interfaces optionally supplement or replace the conventional methods of accessing applications. Such methods and interfaces produce more efficient human-computer interfaces by requiring less input so that users can locate the required information. For battery-driven devices, this method and interface saves power and increases the battery charging interval (e.g., by requiring a smaller number of touch inputs to perform various functions). In addition, this method and interface helps to extend the life of the touch-sensitive display by requiring a smaller number of touch inputs (e.g., the method and interface disclosed herein can actively provide a required piece of information without user input, without having to continuously and purposelessly tap the touch-sensitive display to locate the piece of information).

[0007] The above-mentioned defects and other problems associated with the user interface of the electronic device with touch-sensitive surface are solved by the disclosed device. In some embodiments, the device is a desktop computer. In some embodiments, the device is portable (e.g., a laptop, a tablet computer or a handheld device). In some embodiments, the device has a touchpad. In some embodiments, the device has a touch-sensitive display (also referred to as a "touch screen" or a "touch screen display"). In some embodiments, the device has a graphical user interface (GUI), one or more processors, a memory and one or more modules, a program or an instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI mainly through stylus and / or finger contact and gestures on the touch-sensitive surface. In some embodiments, these functions optionally include image editing, drawing, presentation, word processing, web page creation, disk editing, spreadsheet production, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback and / or digital video playback. The executable instructions for performing these functions are optionally included in a non-transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0008] (A1) According to some embodiments, in a device having a touch-sensitive display (touch screen 112, Figure 1C ) of electronic devices (e.g. Figure 1A The portable multifunctional device 100, according to Figure 1EThe method is executed on a computing device A to D of any one configuration). The method includes: executing an application on the electronic device in response to an instruction from a user of the electronic device. When executing the application, the method also includes: collecting usage data. The usage data includes at least one or more actions (or action types) performed by the user within the application. The method also includes: (i) automatically obtaining at least one trigger condition based on the collected usage data without human intervention, and (ii) associating the at least one trigger condition with a specific action of the one or more actions performed by the user within the application. When it is determined that the at least one trigger condition has been met, the method includes: providing an indication to the user, indicating that the specific action associated with the trigger condition is available.

[0009] (A2) In some embodiments of the A1 method, obtaining the at least one trigger condition includes: sending usage data to one or more servers located away from the electronic device, and receiving the at least one trigger condition from the one or more servers.

[0010] (A3) In some embodiments of the method of any of A1 to A2, providing the indication includes displaying a user interface object on a lock screen on a touch-sensitive display, the user interface object corresponding to a specific action associated with the trigger condition.

[0011] (A4) In some embodiments of the A3 method, the user interface object includes a description of a specific action associated with the trigger condition.

[0012] (A5) In some embodiments of the A4 method, the user interface object also includes an icon associated with the application.

[0013] (A6) In some embodiments of the method of any one of A3 to A5, the method further includes: detecting a first gesture at the user interface object. In response to detecting the first gesture: (i) displaying the application on the touch-sensitive display, and (ii) while displaying the application, the method includes: performing a specific action associated with the trigger condition.

[0014] (A7) In some embodiments of the A6 method, the first gesture is a swipe gesture on the user interface object.

[0015] (A8) In some embodiments of the method of any one of A3 to A5, the method further includes: detecting a second gesture at the user interface object. In response to detecting the second gesture and while continuing to display the lock screen on the touch-sensitive display, performing a specific action associated with the trigger condition.

[0016] (A9) In some embodiments of the A8 method, the second gesture is a single tap on a predefined area of ​​the user interface object.

[0017] (A10) In some embodiments of the method of any one of A3 to A9, the user interface object is displayed in a predefined center portion of the lock screen.

[0018] (A11) In some embodiments of the A1 method, providing an indication to a user that a specific action associated with the trigger condition is available includes performing the specific action.

[0019] (A12) In some embodiments of the A3 method, the user interface object is an icon associated with an application, and the user interface object is generally displayed in a corner of a lock screen on a touch-sensitive display.

[0020] (A13) In some embodiments of the method of any one of A1 to A12, the method further includes: receiving an instruction from a user to unlock the electronic device. In response to receiving the instruction, the method includes: displaying a home screen of the electronic device on the touch-sensitive display. The method further includes: providing an indication to the user on the home screen indicating that a specific action associated with the trigger condition is available.

[0021] (A14) In some embodiments of the A13 method, the home screen includes: (i) a first portion including one or more user interface pages for launching a first set of applications available on the electronic device, and (ii) a second portion displayed adjacent to the first portion for launching a second set of applications available on the electronic device. The second portion is displayed on all user interface pages included in the first portion, and providing the indication on the home screen includes displaying the indication on the second portion.

[0022] (A15) In some embodiments of the A14 method, the second set of applications is different from and smaller than the first set of applications.

[0023] (A16) In some embodiments of the method of any one of A1 to A15, determining that the at least one trigger condition has been satisfied includes: determining that the electronic device has been coupled to a second device different from the electronic device.

[0024] (A17) In some embodiments of the method of any one of A1 to A16, determining that at least one of the trigger conditions has been met includes determining that the electronic device has arrived at a corresponding address of a home location or a work location associated with the user.

[0025] (A18) In some embodiments of the A17 method, determining that the electronic device has arrived at a corresponding address of a home location or a work location associated with the user includes: monitoring motion data from an accelerometer of the electronic device, and determining a time when the electronic device has not moved beyond a threshold amount based on the monitored motion data.

[0026] (A19) In some embodiments of the method of any one of A1 to A18, the usage data also includes a verbal instruction from the user, which is provided to the virtual assistant application while the application continues to execute. The at least one triggering condition is also based on the verbal instruction provided to the virtual assistant application.

[0027] (A20) In some embodiments of the A19 method, the verbal instruction includes a request to create a reminder that corresponds to a current state of the application, the current state corresponding to the state of the application when the verbal instruction is provided.

[0028] (A21) In some embodiments of the A20 method, the application state when the verbal instruction is provided is selected from the group consisting of: a page displayed within the application when the verbal instruction is provided, content played within the application when the verbal instruction is provided, a notification displayed within the application when the verbal instruction is provided, and an active portion of a page displayed within the application when the verbal instruction is provided.

[0029] (A22) In some embodiments of the A20 method, the verbal instruction includes the word "this" related to the current state of the application.

[0030] (A23) On the other hand, in one or more electronic devices (e.g., Figure 5 The portable multifunctional device 100, Figure 5 One or more servers 502) execute the method. The method includes: executing an application on a first electronic device of one or more electronic devices in response to an instruction from a user of the first electronic device. During the execution of the application, the method includes: automatically collecting usage data without human intervention, and the usage data includes at least one or more actions (or action types) performed by the user within the application. The method also includes: automatically establishing at least one trigger condition based on the collected usage data without human intervention. The method further includes: associating the at least one trigger condition with a specific action of the one or more actions performed by the user within the application. When it is determined that the at least one trigger condition has been met, the method includes: providing an indication to the user, indicating that the specific action associated with the trigger condition is available.

[0031] (A24) In another aspect, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive display, one or more processors, and a memory storing one or more programs, which, when executed by the one or more processors, causes the electronic device to perform any of the methods described in A1 to A22.

[0032] (A25) On the other hand, an electronic device is provided, comprising: a touch-sensitive display and a device for executing any one of the methods described in A1 to A22.

[0033] (A26) In another aspect, a non-transitory computer-readable storage medium is provided, which is used to store executable instructions, which, when executed by an electronic device having a touch-sensitive display, will cause the electronic device to perform any method described in A1 to A22.

[0034] (A27) On the other hand, a graphical user interface on an electronic device having a touch-sensitive display is provided. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of A1 to A22.

[0035] (A28) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.42 display unit 4201), a touch-sensitive surface unit (e.g., Fig.42 touch-sensitive surface unit 4203) and a processing unit (e.g., Fig.42 In some embodiments, according to Figure 1E Any of the computing devices shown (i.e., computing devices A to D) configures an electronic device. For ease of explanation, Fig.42 A display unit 4201 and a touch-sensitive surface unit 4203 are shown integrated with an electronic device 4200, however, in some embodiments, one or both of these units communicate with the electronic device even though these units remain physically separated from the electronic device. A processing unit is coupled to the touch-sensitive surface unit and the display unit. In some embodiments, the touch-sensitive surface unit and the display unit are integrated into a single touch-sensitive display unit (also referred to herein as a touch-sensitive display). The processing unit includes an execution unit (e.g., Fig.42 Execution unit 4207), collection unit (e.g., Fig.42 The collection unit 4209), the acquisition unit (for example, Fig.42 The acquisition unit 4211), the association unit (for example, Fig.42 4213), providing unit (e.g., Fig.42 The providing unit 4215), the sending unit (for example, Fig.42 The sending unit 4217), the receiving unit (for example, Fig.42 receiving unit 4219), a display unit (eg, Fig.42 display unit 4221), a detection unit (e.g., Fig.42 Detection unit 4223), execution unit (for example, Fig.42 Execution unit 4225), determination unit (for example, Fig.42 The determination unit 4227) and the monitoring unit (e.g., Fig.42 The processing unit (or one or more components thereof, such as units 4207 to 4229) is configured to: execute an application on the electronic device in response to an instruction from a user of the electronic device (e.g., by executing unit 4207); collect usage data (e.g., using collecting unit 4209) when executing the application, the usage data including at least one or more actions performed by the user within the application; automatically obtain at least one trigger condition based on the collected usage data without human intervention (e.g., by obtaining unit 4211); associate the at least one trigger condition with a specific action of the one or more actions performed by the user within the application (e.g., by associating unit 4213); and, upon determining that the at least one trigger condition has been met, provide an indication to the user (e.g., by providing unit 4215) indicating that a specific action associated with the trigger condition is available.

[0036] (A29) In some embodiments of the electronic device of A28, obtaining the at least one trigger condition includes: sending (for example, through a sending unit 4217) usage data to one or more servers located away from the electronic device, and receiving (for example, through a receiving unit 4219) the at least one trigger condition from one or more servers.

[0037] (A30) In some embodiments of an electronic device of any one of A28 to A29, providing an indication includes displaying on a lock screen on a touch-sensitive display unit (e.g., via display unit 4217 and / or display unit 4201) a user interface object corresponding to a specific action associated with the trigger condition.

[0038] (A31) In some embodiments of the electronic device of A30, the user interface object includes a description of a specific action associated with the trigger condition.

[0039] (A32) In some embodiments of the electronic device of A31, the user interface object also includes an icon associated with the application.

[0040] (A33) In some embodiments of the electronic device of any one of A30 to A32, the processing unit is further configured to: detect (e.g., by the detection unit 4223 and / or the touch-sensitive surface unit 4203) a first gesture at the user interface object. In response to detecting the first gesture: (i) displaying an application on the touch-sensitive display unit (e.g., by the display unit 4217 and / or the display unit 4201), and (ii) executing (e.g., by the execution unit 4225) a specific action associated with the trigger condition while displaying the application.

[0041] (A34) In some embodiments of the electronic device of A33, the first gesture is a swipe gesture on the user interface object.

[0042] (A35) In some embodiments of the electronic device of any one of A30 to A33, the processing unit is further configured to: detect (e.g., by detection unit 4223 and / or touch-sensitive surface unit 4203) a second gesture at the user interface object. In response to detecting the second gesture and while continuing to display the lock screen on the touch-sensitive display unit, the processing unit is configured to: execute (e.g., by execution unit 4225) a specific action associated with the trigger condition.

[0043] (A36) In some embodiments of the electronic device of A35, the second gesture is a single tap on a predefined area of ​​the user interface object.

[0044] (A37) In some embodiments of the electronic device of any one of A30 to A36, the user interface object is displayed in a predefined center portion of the lock screen.

[0045] (A38) In some embodiments of the electronic device of A28, providing an indication to a user that a specific action associated with the trigger condition is available includes performing (e.g., by executing unit 4225) the specific action.

[0046] (A39) In some embodiments of the electronic device of A30, the user interface object is an icon associated with an application, and the user interface object is generally displayed in a corner of a lock screen on the touch-sensitive display unit.

[0047] (A40) In some embodiments of the electronic device of any one of A28 to A39, the processing unit is further configured to: receive (e.g., via receiving unit 4219) an instruction from a user to unlock the electronic device. In response to receiving the instruction, the processing unit is configured to: display (e.g., via display unit 4217 and / or display unit 4201) a home screen of the electronic device on the touch-sensitive display unit. The processing unit is further configured to: provide (e.g., via providing unit 4215) an indication to the user on the home screen that a specific action associated with the trigger condition is available.

[0048] (A41) In some embodiments of the electronic device of A40, the home screen includes: (i) a first portion including one or more user interface pages for launching a first set of applications available on the electronic device, and (ii) a second portion displayed adjacent to the first portion for launching a second set of applications available on the electronic device. The second portion is displayed on all user interface pages included in the first portion, and providing an indication on the home screen includes displaying (e.g., via display unit 4217 and / or display unit 4201) an indication on the second portion.

[0049] (A42) In some embodiments of the electronic device of A41, the second group of applications is different from and smaller than the first group of applications.

[0050] (A43) In some embodiments of the electronic device of any one of A28 to A42, determining that at least one trigger condition has been met includes: determining (for example, by determining unit 4227) that the electronic device has been coupled to a second device different from the electronic device.

[0051] (A44) In some embodiments of the electronic device of any one of A28 to A43, determining that at least one trigger condition has been met includes: determining (for example, by determining unit 4227) that the electronic device has arrived at a corresponding address of a home location or a work location associated with the user.

[0052] (A45) In some embodiments of the electronic device of A44, determining that the electronic device has arrived at a corresponding address of a home location or a work location associated with the user includes: monitoring (e.g., by monitoring unit 4229) motion data from an accelerometer of the electronic device, and determining (e.g., by determining unit 4227) a time during which the electronic device has not moved beyond a threshold amount based on the monitored motion data.

[0053] (A46) In some embodiments of the electronic device of any one of A28 to A45, the usage data also includes a verbal instruction from the user, which is provided to the virtual assistant application while the application continues to execute. The at least one trigger condition is also based on the verbal instruction provided to the virtual assistant application.

[0054] (A47) In some embodiments of the electronic device of A46, the verbal instruction includes a request to create a reminder that corresponds to a current state of the application, the current state corresponding to the state of the application when the verbal instruction is provided.

[0055] (A48) In some embodiments of the electronic device of A47, the application state when the verbal instruction is provided is selected from the group consisting of: a page displayed within the application when the verbal instruction is provided, content played within the application when the verbal instruction is provided, a notification displayed within the application when the verbal instruction is provided, and an active portion of a page displayed within the application when the verbal instruction is provided.

[0056] (A49) In some embodiments of the electronic device of A46, the verbal instruction includes the term "this" related to the current state of the application.

[0057] (B1) According to some embodiments, on a device having a touch-sensitive display (touch screen 112, Figure 1C ) of electronic devices (e.g. Figure 1A The portable multifunctional device 100, according to Figure 1E The method is performed on a computing device A to D of any one of the configurations of the electronic device. The method includes: obtaining at least one trigger condition, the at least one trigger condition being based on usage data associated with a user of the electronic device, the usage data including one or more actions (or action types) performed by the user within the application when the application is executed on the electronic device. The method also includes: associating the at least one trigger condition with a specific action of the one or more actions performed by the user within the application. When it is determined that the at least one trigger condition has been met, the method includes: providing an indication to the user indicating that the specific action associated with the trigger condition is available.

[0058] (B2) In some embodiments of the B1 method, the method further comprises the method described in any one of A2 to A22.

[0059] (B3) In another aspect, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive display, one or more processors, and a memory storing one or more programs, which, when executed by the one or more processors, causes the electronic device to perform any of the methods described in B1 to B2.

[0060] (B4) On the other hand, an electronic device is provided, and the electronic device includes: a touch-sensitive display and a device for executing any one of the methods described in B1 to B2.

[0061] (B5) In another aspect, a non-transitory computer-readable storage medium is provided, which is used to store executable instructions, and when these instructions are executed by an electronic device having a touch-sensitive display, the electronic device will perform any method described in B1 to B2.

[0062] (B6) On the other hand, a graphical user interface on an electronic device having a touch-sensitive display is provided. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of B1 to B2.

[0063] (B7) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.42 display unit 4201), a touch-sensitive surface unit (e.g., Fig.42 touch-sensitive surface unit 4203) and a processing unit (e.g., Fig.42 In some embodiments, according to Figure 1E Any of the computing devices shown (i.e., computing devices A to D) configures an electronic device. For ease of explanation, Fig.42 The display unit 4201 and the touch-sensitive surface unit 4203 are shown integrated with the electronic device 4200, however, in some embodiments, one or both of these units communicate with the electronic device even though these units remain physically separated from the electronic device. The processing unit includes an execution unit (e.g., Fig.42 Execution unit 4207), collection unit (e.g., Fig.42 The collection unit 4209), the acquisition unit (for example, Fig.42 The acquisition unit 4211), the association unit (for example, Fig.42 4213), providing unit (e.g., Fig.42 The providing unit 4215), the sending unit (for example, Fig.42 The sending unit 4217), the receiving unit (for example, Fig.42 receiving unit 4219), a display unit (eg, Fig.42 display unit 4221), a detection unit (e.g., Fig.42 Detection unit 4223), execution unit (for example, Fig.42 Execution unit 4225), determination unit (for example, Fig.42 The determination unit 4227) and the monitoring unit (e.g., Fig.42The processing unit (or one or more components thereof, such as units 4207 to 4229) is configured to: obtain (e.g., by obtaining unit 4211) at least one trigger condition, the at least one trigger condition being based on usage data associated with a user of the electronic device, the usage data including one or more actions performed by the user within the application when the application is executed on the electronic device; associate the at least one trigger condition with a specific action of the one or more actions performed by the user within the application (e.g., by associating unit 4213); and upon determining that the at least one trigger condition has been met, provide an indication to the user (e.g., by providing unit 4215) indicating that the specific action associated with the trigger condition is available.

[0064] (B8) In some embodiments of the electronic device of B7, obtaining the at least one trigger condition includes: sending (for example, through sending unit 4217) usage data to one or more servers located away from the electronic device, and receiving (for example, through receiving unit 4219) the at least one trigger condition from one or more servers.

[0065] (B9) In some embodiments of an electronic device of any of B7 to B8, providing an indication includes displaying on a lock screen on a touch-sensitive display (e.g., via display unit 4217 and / or display unit 4201) a user interface object corresponding to a specific action associated with the trigger condition.

[0066] (B10) In some embodiments of the electronic device of B9, the user interface object includes a description of a specific action associated with the trigger condition.

[0067] (B11) In some embodiments of the electronic device of B10, the user interface object also includes an icon associated with the application.

[0068] (B12) In some embodiments of the electronic device of any one of B9 to B11, the processing unit is further configured to: detect (e.g., by detecting unit 4223 and / or touch-sensitive surface unit 4203) a first gesture at the user interface object. In response to detecting the first gesture: (i) displaying an application on the touch-sensitive display (e.g., by display unit 4217 and / or display unit 4201), and (ii) executing (e.g., by executing unit 4225) a specific action associated with the trigger condition while displaying the application.

[0069] (B13) In some embodiments of the electronic device of B12, the first gesture is a swipe gesture on the user interface object.

[0070] (B14) In some embodiments of the electronic device of any one of B9 to B12, the processing unit is further configured to: detect (e.g., by detection unit 4223 and / or touch-sensitive surface unit 4203) a second gesture at the user interface object. In response to detecting the second gesture and while continuing to display the lock screen on the touch-sensitive display, the processing unit is configured to: execute (e.g., by execution unit 4225) a specific action associated with the trigger condition.

[0071] (B15) In some embodiments of the electronic device of B14, the second gesture is a single tap on a predefined area of ​​the user interface object.

[0072] (B16) In some embodiments of the electronic device of any one of B9 to B15, the user interface object is displayed in a predefined center portion of the lock screen.

[0073] (B17) In some embodiments of the electronic device of B7, providing an indication to a user that a specific action associated with the trigger condition is available includes performing (e.g., by executing unit 4225) the specific action.

[0074] (B18) In some embodiments of the electronic device of B9, the user interface object is an icon associated with an application, and the user interface object is generally displayed in a corner of a lock screen on a touch-sensitive display.

[0075] (B19) In some embodiments of the electronic device of any one of B7 to B18, the processing unit is further configured to: receive (e.g., via receiving unit 4219) an instruction from a user to unlock the electronic device. In response to receiving the instruction, the processing unit is configured to: display (e.g., via display unit 4217 and / or display unit 4201) a home screen of the electronic device on the touch-sensitive display. The processing unit is further configured to: provide (e.g., via providing unit 4215) an indication to the user on the home screen that a specific action associated with the trigger condition is available.

[0076] (B20) In some embodiments of the electronic device of B19, the home screen includes: (i) a first portion including one or more user interface pages for launching a first set of applications available on the electronic device, and (ii) a second portion displayed adjacent to the first portion for launching a second set of applications available on the electronic device. The second portion is displayed on all user interface pages included in the first portion, and providing the indication on the home screen includes displaying (e.g., via display unit 4217 and / or display unit 4201) the indication on the second portion.

[0077] (B21) In some embodiments of the electronic device of B20, the second group of applications is different from and smaller than the first group of applications.

[0078] (B22) In some embodiments of the electronic device of any one of B7 to B21, determining that at least one trigger condition has been met includes: determining (for example, by determining unit 4227) that the electronic device has been coupled to a second device different from the electronic device.

[0079] (B23) In some embodiments of the electronic device of any one of B7 to B22, determining that at least one trigger condition has been met includes: determining (for example, by determining unit 4227) that the electronic device has arrived at a corresponding address of a home location or a work location associated with the user.

[0080] (B24) In some embodiments of the electronic device of B23, determining that the electronic device has arrived at a corresponding address of a home location or a work location associated with the user includes: monitoring (e.g., by monitoring unit 4229) motion data from an accelerometer of the electronic device, and determining (e.g., by determining unit 4227) a time during which the electronic device has not moved beyond a threshold amount based on the monitored motion data.

[0081] (B25) In some embodiments of the electronic device of any one of B7 to B24, the usage data also includes a verbal instruction from the user, which is provided to the virtual assistant application while the application continues to execute. The at least one triggering condition is also based on the verbal instruction provided to the virtual assistant application.

[0082] (B26) In some embodiments of the electronic device of B25, the verbal instruction includes a request to create a reminder that corresponds to a current state of the application, the current state corresponding to the state of the application when the verbal instruction is provided.

[0083] (B27) In some embodiments of the electronic device of B26, the application state when the verbal instruction is provided is selected from the group consisting of: a page displayed within the application when the verbal instruction is provided, content played within the application when the verbal instruction is provided, a notification displayed within the application when the verbal instruction is provided, and an active portion of a page displayed within the application when the verbal instruction is provided.

[0084] (B28) In some embodiments of the electronic device of B26, the verbal instruction includes the term "this" related to the current state of the application.

[0085] (C1) According to some embodiments, on a device having a touch-sensitive display (touch screen 112, Figure 1C ) of electronic devices (e.g. Figure 1A The portable multifunctional device 100, according to Figure 1E The method is performed on a computing device A to D of any one of the configurations of the electronic device. The method includes: detecting a search activation gesture from a user of the electronic device on a touch-sensitive display. In response to detecting the search activation gesture, the method includes: displaying a search interface on the touch-sensitive display, the search interface including: (i) a search entry portion and (ii) a prediction portion displayed before any user input is received at the search entry portion. The prediction portion is populated by one or more of the following: (a) at least one enabling representation for contacting a person from a plurality of previously contacted persons, the person being automatically selected from the plurality of previously contacted persons based at least in part on a current time, and (b) at least one enabling representation for performing a predicted action in a plurality of applications available on the electronic device, the predicted action being automatically selected based at least in part on an application usage history associated with the user of the electronic device.

[0086] (C2) In some embodiments of the C1 method, the person is further selected based at least in part on location data corresponding to the electronic device.

[0087] (C3) In some embodiments of the method of any one of C1 to C2, the person's application usage history and contact information are retrieved from the memory of the electronic device.

[0088] (C4) In some embodiments of the method of any one of C1 to C2, the person's application usage history and contact information are retrieved from a memory located remotely from the electronic device.

[0089] (C5) In some embodiments of the method of any one of items C1 to C4, the prediction portion is further populated by at least one enabling representation of executing a prediction application, which is automatically selected at least in part based on the application usage history.

[0090] (C6) In some embodiments of the method of any one of C1 to C5, the prediction portion is further populated by at least one enabling representation of a predicted place (or nearby place) category, and the predicted place category is automatically selected based at least in part on one or more items of current time and location data corresponding to the electronic device.

[0091] (C7) In some embodiments of the method of any one of C1 to C6, the method further comprises: detecting user input to scroll the prediction section. In response to detecting the user input to scroll the prediction section, the method comprises: scrolling the prediction section according to the user input. In response to the scrolling, the method comprises: visualizing at least one affordance of a predicted news article in the prediction section (e.g., the predicted news article is an article predicted to be of interest to the user).

[0092] (C8) In some embodiments of the C7 method, the predicted news article is automatically selected based at least in part on location data corresponding to the electronic device.

[0093] (C9) In some embodiments of the method of any one of C1 to C8, the method further includes: detecting a selection of at least one affordance to perform the predicted action within the application. In response to detecting the selection, the method includes: displaying the application on the touch-sensitive display and performing the predicted action within the application.

[0094] (C10) In some embodiments of the method of any one of C3 to C4, the method further includes: detecting a selection of the at least one affordance to contact the person. In response to detecting the selection, the method includes: contacting the person using the person's contact information.

[0095] (C11) In some embodiments of the C5 method, the method further includes: detecting a selection of the at least one affordance to execute the predicted application. In response to detecting the selection, the method includes: displaying the predicted application on the touch-sensitive display.

[0096] (C12) In some embodiments of the C6 method, the method further includes: detecting a selection of the at least one affordance for the predicted place category. In response to detecting the selection, the method further includes: (i) receiving data corresponding to at least one nearby place, and (ii) displaying the received data corresponding to the at least one nearby place on the touch-sensitive display.

[0097] (C13) In some embodiments of the C7 method, the method further includes: detecting a selection of the at least one affordance of the predicted news article. In response to detecting the selection, the method includes: displaying the predicted news article on the touch-sensitive display.

[0098] (C14) In some embodiments of the method of any one of items C1 to C13, the search activation gesture can be obtained from at least two different user interfaces, and a first user interface of the at least two different user interfaces corresponds to a corresponding home screen page of a sequence of home screen pages displayed on the touch-sensitive display.

[0099] (C15) In some embodiments of the C14 method, when the corresponding home screen page is the first home screen page in a sequence of home screen pages, the search activation gesture includes one of: (i) a gesture that moves generally in a downward direction relative to a user of the electronic device, or (ii) a continuous gesture that moves generally in a left to right direction relative to the user and generally perpendicular to the downward direction.

[0100] (C16) In some embodiments of the C15 method, when the corresponding home screen page is the second home screen page in a sequence of home screen pages, the search activation gesture includes a continuous gesture moving generally in a downward direction relative to a user of the electronic device.

[0101] (C17) In some embodiments of the C14 method, the second user interface of the at least two different user interfaces corresponds to displaying an application switching interface on a touch-sensitive display.

[0102] (C18) In some embodiments of the C17 method, the search activation gesture includes touching a predefined search activation portion of an application switching interface on the touch-sensitive display.

[0103] (C19) In another aspect, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive display, one or more processors, and a memory storing one or more programs, which, when executed by the one or more processors, causes the electronic device to perform any of the methods described in C1 to C18.

[0104] (C20) On the other hand, an electronic device is provided, and the electronic device includes: a touch-sensitive display and a device for executing any one of the methods described in C1 to C18.

[0105] (C21) In another aspect, a non-transitory computer-readable storage medium is provided, which is used to store executable instructions, which, when executed by an electronic device having a touch-sensitive display, will cause the electronic device to perform any method described in C1 to C18.

[0106] (C22) On the other hand, a graphical user interface on an electronic device with a touch-sensitive display is provided. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of C1 to C18.

[0107] (C23) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.43 display unit 4301), a touch-sensitive surface unit (e.g., Fig.43 touch-sensitive surface unit 4303) and a processing unit (e.g., Fig.43 In some embodiments, according to Figure 1E Any of the computing devices shown (i.e., computing devices A to D) configures an electronic device. For ease of explanation, Fig.43A display unit 4301 and a touch-sensitive surface unit 4303 are shown integrated with the electronic device 4300, however, in some embodiments, one or both of these units communicate with the electronic device even though these units remain physically separate from the electronic device. The processing unit includes a display unit (e.g., Fig.43 display unit 4309), a detection unit (e.g., Fig.43 Detection unit 4307), retrieval unit (e.g., Fig.43 retrieval unit 4311), filling unit (for example, Fig.43 Filling unit 4313), scrolling unit (for example, Fig.43 Scrolling unit 4315), display unit (for example, Fig.43 display unit 4317), a selection unit (e.g., Fig.43 The selection unit 4319), the contact unit (for example, Fig.43 The contact unit 4321), the receiving unit (for example, Fig.43 The receiving unit 4323) and the execution unit (for example, Fig.43 The processing unit (or one or more components thereof, such as units 4307 to 4225) is configured to: detect (e.g., by detection unit 4307 and / or touch-sensitive surface unit 4303) a search activation gesture on a touch-sensitive display from a user of the electronic device; in response to detecting the search activation gesture, display (e.g., by display unit 4309 and / or display unit 4301) a search interface on the touch-sensitive display, the search interface comprising: (i) a search entry portion; and (ii) a prediction portion displayed before any user input is received at the search entry portion, the prediction portion being populated with one or more of the following: (a) at least one affordance for contacting a person from a plurality of previously contacted persons, the person being automatically selected (e.g., by selection unit 4319) from the plurality of previously contacted persons based at least in part on a current time, and (b) at least one affordance for performing a predicted action in a certain application of a plurality of applications available on the electronic device, the predicted action being automatically selected (e.g., by selection unit 4319) based at least in part on an application usage history associated with the user of the electronic device.

[0108] (C24) In some embodiments of the electronic device of C23, the person is further selected (e.g., by selection unit 4319) based at least in part on location data corresponding to the electronic device.

[0109] (C25) In some embodiments of the electronic device of any one of C23 to C24, the person's application usage history and contact information are retrieved from the memory of the electronic device (e.g., by retrieval unit 4311).

[0110] (C26) In some embodiments of the electronic device of any one of C23 to C24, the person's application usage history and contact information are retrieved (e.g., by retrieval unit 4311) from a server located away from the electronic device.

[0111] (C27) In some embodiments of an electronic device of any one of C23 to C26, the prediction portion is further filled in (e.g., by filling unit 4313) by at least one enabling representation of executing a prediction application, which is automatically selected (e.g., by selection unit 4319) at least in part based on an application usage history.

[0112] (C28) In some embodiments of an electronic device of any one of C23 to C27, the prediction portion is further filled in by at least one enabling representation of a predicted place category (e.g., by filling unit 4313), and the predicted place category is automatically selected based at least in part on one or more items of current time and location data corresponding to the electronic device (e.g., by selection unit 4319).

[0113] (C29) In some embodiments of the electronic device of any one of C23 to C28, the processing unit is further configured to: detect (e.g., by detecting unit 4307 and / or touch-sensitive surface unit 4303) user input to scroll the prediction portion. In response to detecting the user input to scroll the prediction portion, the processing unit is configured to: scroll (e.g., by scrolling unit 4319) the prediction portion according to the user input. In response to the scrolling, the processing unit is configured to: display (e.g., by displaying unit 4317) at least one affordance of a predicted news article in the prediction portion (e.g., the predicted news article is an article predicted to be of interest to the user).

[0114] (C30) In some embodiments of the electronic device of C7, a predicted news article is automatically selected (e.g., by a selection unit 4319) based at least in part on location data corresponding to the electronic device.

[0115] (C31) In some embodiments of the electronic device of any one of C23 to C30, the processing unit is further configured to: detect (e.g., by detection unit 4307 and / or touch-sensitive surface unit 4303) a selection of at least one affordance to perform a predicted action within an application. In response to detecting the selection, the processing unit is configured to: display (e.g., by display unit 4309) the application on a touch-sensitive display (e.g., display unit 4301) and perform (e.g., by execution unit 4325) the predicted action within the application.

[0116] (C32) In some embodiments of the electronic device of any one of C25 to C26, the processing unit is further configured to: detect (e.g., via detection unit 4307 and / or touch-sensitive surface unit 4303) a selection of the at least one affordance to contact the person. In response to detecting the selection, the processing unit is configured to: contact the person using the contact information of the person (e.g., via contact unit 4321).

[0117] (C33) In some embodiments of the electronic device of C27, the processing unit is further configured to: detect (e.g., by detecting unit 4307 and / or touch-sensitive surface unit 4303) a selection of at least one affordance to execute the predicted application. In response to detecting the selection, the processing unit is configured to: display (e.g., by display unit 4307) the predicted application on the touch-sensitive display (e.g., display unit 4301).

[0118] (C34) In some embodiments of the electronic device of C28, the processing unit is further configured to: detect (e.g., by detecting unit 4307 and / or touch-sensitive surface unit 4303) a selection of the at least one affordance of the predicted place category. In response to detecting the selection, the processing unit is configured to: (i) receive (e.g., by receiving unit 4323) data corresponding to at least one nearby place, and (ii) display (e.g., by display unit 4307) the received data corresponding to the at least one nearby place on the touch-sensitive display (e.g., display unit 4301).

[0119] (C35) In some embodiments of the electronic device of C29, the processing unit is further configured to: detect (e.g., by detecting unit 4307 and / or touch-sensitive surface unit 4303) a selection of the at least one affordance of the predicted news article. In response to detecting the selection, the processing unit is configured to: display (e.g., by display unit 4307) the predicted news article on the touch-sensitive display (e.g., display unit 4301).

[0120] (C36) In some embodiments of an electronic device of any one of C23 to C35, the search activation gesture can be obtained from at least two different user interfaces, and a first user interface of the at least two different user interfaces corresponds to a corresponding home screen page of a sequence of home screen pages displayed on a touch-sensitive display.

[0121] (C37) In some embodiments of the electronic device of C36, when the corresponding home screen page is the first home screen page in a sequence of home screen pages, the search activation gesture includes one of: (i) a gesture that moves generally in a downward direction relative to a user of the electronic device, or (ii) a continuous gesture that moves generally in a left to right direction relative to the user and generally perpendicular to the downward direction.

[0122] (C38) In some embodiments of the electronic device of C37, when the corresponding home screen page is the second home screen page in a sequence of home screen pages, the search activation gesture includes a continuous gesture moving generally in a downward direction relative to a user of the electronic device.

[0123] (C39) In some embodiments of the electronic device of C36, the second user interface of the at least two different user interfaces corresponds to displaying an application switching interface on the touch-sensitive display.

[0124] (C40) In some embodiments of the electronic device of C39, the search activation gesture includes touching a predefined search activation portion of an application switching interface on the touch-sensitive display.

[0125] Thus, electronic devices having a display, a touch-sensitive surface, and optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface are provided with faster, more efficient methods and interfaces for actively accessing applications and actively executing functions within applications, thereby improving the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces can supplement or replace conventional methods for accessing applications and functions related thereto.

[0126] (D1) According to some embodiments, on a device having a touch-sensitive surface (e.g., touch-sensitive surface 195, Figure 1D ) and a display (e.g., display 194, Figure 1D ) of electronic devices (e.g. Figure 1A The portable multifunctional device 100, according to Figure 1EThe method is performed on a computing device A to D of any one configuration). The method includes: displaying content associated with an application being executed on the electronic device on a display. The method also includes: detecting a swipe gesture through a touch-sensitive surface, which, when detected, causes the electronic device to enter a search mode different from that of the application. The method also includes: entering a search mode in response to detecting the swipe gesture, the search mode including a search interface displayed on the display. In conjunction with entering the search mode, the method includes: determining at least one suggested search query based at least in part on information associated with the content. Before any user input is received at the search interface, the method includes: populating the displayed search interface with the at least one suggested search query. In this way, the device provides relevant suggestions to the user based on the application content viewed by the user, and the user only needs to select one of the suggestions without typing anything, thereby eliminating the need for the user to remember the information and re-enter the information in the search interface.

[0127] (D2) In some embodiments of the D1 method, detecting a swipe gesture includes detecting a swipe gesture on at least a portion of currently displayed content.

[0128] (D3) In some embodiments of the method of any one of D1 to D2, the method further includes: before detecting the swipe gesture, detecting an input corresponding to a request to view a home screen of the electronic device; and in response to detecting the input, stopping displaying the content associated with the application and displaying a corresponding page of the home screen of the electronic device. In some embodiments, the corresponding page is an initial page in a sequence of home screen pages, and the swipe gesture is detected when the initial page of the home screen is displayed on the display.

[0129] (D4) In some embodiments of the method of any one of D1 to D3, the search interface is displayed to semi-transparently cover the application.

[0130] (D5) In some embodiments of the method of any one of items D1 to D4, the method also includes: based on determining that the content includes text content, determining the at least one suggested search query based at least in part on the text content.

[0131] (D6) In some embodiments of the D5 method, determining the at least one suggested search query based at least in part on the text content includes: analyzing the text content to detect one or more predefined keywords used to determine the at least one suggested search query.

[0132] (D7) In some embodiments of the method of any one of D1 to D6, determining the at least one suggested search query includes determining a plurality of suggested search queries, and populating the search interface includes populating the search interface with the plurality of suggested search queries.

[0133] (D8) In some embodiments of the method of any one of D1 to D7, the method also includes: detecting, via the touch-sensitive surface, a new swipe gesture on new content currently displayed; and in response to detecting the new swipe gesture, entering a search mode, wherein entering the search mode includes displaying a search interface on the display; and in conjunction with entering the search mode and based on determining that the new content does not include text content, populating the search interface with suggested search queries, wherein the suggested search queries are based on a set of selected historical search queries from a user of the electronic device.

[0134] (D9) In some embodiments of the D8 method, the search interface displays points of interest based on location information provided by a second application that is different from the application.

[0135] (D10) In some embodiments of the method of any one of D8 to D9, the search interface also includes one or more suggested applications.

[0136] (D11) In some embodiments of the method of any one of D8 to D10, the set of historical search queries is selected based at least in part on a frequency of recent search queries.

[0137] (D12) In some embodiments of the method of any one of D1 to D11, the method further includes: in combination with entering a search mode, obtaining information associated with the content by using one or more auxiliary functions available on the electronic device.

[0138] (D13) In some embodiments of the D12 method, using one or more auxiliary functions includes using one or more auxiliary functions to generate information associated with the content by: (i) applying a natural language processing algorithm to text content currently displayed in an application, and (ii) using data obtained from the natural language processing algorithm to determine one or more keywords that describe the content, and determining the at least one suggested search query based on the one or more keywords.

[0139] (D14) In some embodiments of the D13 method, determining one or more keywords describing the content also includes: (i) retrieving metadata corresponding to non-text content currently displayed in the application, and (ii) determining one or more keywords using the retrieved metadata in addition to data obtained from the natural language processing algorithm.

[0140] (D15) In some embodiments of the method of any one of D1 to D14, the search interface also includes one or more trend queries.

[0141] (D16) In some embodiments of the D15 method, the search interface also includes one or more applications predicted to be of interest to the user of the electronic device.

[0142] (D17) In another aspect, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive surface and a display, one or more processors, and a memory storing one or more programs, which, when executed by the one or more processors, causes the electronic device to perform the method described in any one of D1 to D16.

[0143] (D18) On the other hand, an electronic device is provided, and the electronic device includes: a touch-sensitive surface and a display, and a device for executing any one of the methods described in D1 to D16.

[0144] (D19) In another aspect, a non-transitory computer-readable storage medium is provided, which is used to store executable instructions, which, when executed by an electronic device having a touch-sensitive surface and a display, will cause the electronic device to perform the method described in any one of D1 to D16.

[0145] (D20) On the other hand, a graphical user interface on an electronic device having a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of D1 to D16. On the other hand, an information processing device used in an electronic device including a touch-sensitive surface and a display is provided. The information processing device includes: a device for performing the method described in any one of D1 to D16.

[0146] (D21) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.44 display unit 4401), a touch-sensitive surface unit (e.g., Fig.44 touch-sensitive surface unit 4403) and a processing unit (e.g., Fig.44 The processing unit 4405 is coupled to the touch-sensitive surface unit and the display unit. Figure 1E Any of the computing devices shown (i.e., computing devices A to D) configures an electronic device. For ease of explanation, Fig.44 A display unit 4401 and a touch-sensitive surface unit 4403 are shown integrated with an electronic device 4400, however, in some embodiments, one or both of these units communicate with the electronic device even though these units remain physically separated from the electronic device. In some embodiments, the touch-sensitive surface unit and the display unit are integrated into a single touch-sensitive display unit (also referred to herein as a touch-sensitive display). The processing unit includes a detection unit (e.g., Fig.44 Detection unit 4407), display unit (for example, Fig.44 Display unit 4409), retrieval unit (for example, Fig.44retrieval unit 4411), search mode entry unit (for example, Fig.44 Search mode entry unit 4412), fill unit (for example, Fig.44 Filling unit 4413), acquisition unit (for example, Fig.44 The acquisition unit 4415), the determination unit (for example, Fig.44 A determination unit 4417) and a selection unit (eg, Fig.44 The processing unit (or one or more components thereof, such as units 1007 to 1029) is configured to: display (e.g., by display unit 4407) content associated with an application being executed on the electronic device on a display unit (e.g., display unit 4407); detect (e.g., by detection unit 4407) a swipe gesture by a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4403), which, when detected, causes the electronic device to enter a search mode different from that of the application; in response to detecting the swipe gesture, enter a search mode (e.g., by search mode entry unit 4412), the search mode including a search interface displayed on the display unit (e.g., display unit 4407); in conjunction with entering the search mode, determine (e.g., by determination unit 4417) at least one suggested search query based at least in part on information associated with the content; and populate (e.g., by populating unit 4413) the displayed search interface with the at least one suggested search query before any user input is received at the search interface.

[0147] (D22) In some embodiments of the electronic device of D21, detecting a swipe gesture includes detecting (e.g., by detection unit 4407) a swipe gesture on at least a portion of the currently displayed content.

[0148] (D23) In some embodiments of the electronic device of any one of D21 to D22, the processing unit is further configured to: before detecting a swipe gesture, detect (for example, by detection unit 4407) an input corresponding to a request to view the home screen of the electronic device; and in response to detecting (for example, by detection unit 4407) the input, stop displaying content associated with the application and display a corresponding page of the home screen of the electronic device (for example, by display unit 4409), wherein: the corresponding page is an initial page in a sequence of home screen pages; and detect (for example, by detection unit 4407) a swipe gesture when the initial page of the home screen is displayed on the display unit.

[0149] (D24) In some embodiments of the electronic device of any one of D21 to D23, the search interface is displayed (e.g., by display unit 4409 and / or display unit 4401) as a semi-transparent overlay of the application.

[0150] (D25) In some embodiments of an electronic device of any one of D21 to D24, the processing unit is further configured to: based on determining that the content includes text content, determine (e.g., by determining unit 4417) at least one suggested search query based at least in part on the text content.

[0151] (D26) In some embodiments of the electronic device of D25, determining the at least one suggested search query based at least in part on the text content includes: analyzing the text content to detect (e.g., by detection unit 4407) one or more predefined keywords for determining (e.g., by determination unit 4417) the at least one suggested search query.

[0152] (D27) In some embodiments of the electronic device of any one of D21 to D26, determining the at least one suggested search query includes determining (e.g., by determining unit 4417) multiple suggested search queries, and populating the search interface includes populating (e.g., by populating unit 4413) the search interface using the multiple suggested search queries.

[0153] (D28) In some embodiments of the electronic device of any one of D21 to D27, the processing unit is further configured to: detect (e.g., by detection unit 4407) a new swipe gesture on new content currently displayed via a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4403); and in response to detecting the new swipe gesture, enter a search mode (e.g., by search mode entry unit 4412), and entering the search mode includes displaying a search interface on the display unit (e.g., by display unit 4409); and in conjunction with entering the search mode and based on determining that the new content does not include text content, populate (e.g., by populating unit 4413) the search interface with suggested search queries, wherein the suggested search queries are based on a set of selected historical search queries from a user of the electronic device.

[0154] (D29) In some embodiments of the electronic device of D28, the search interface is displayed together with points of interest (e.g., display unit 4409), and the points of interest are based on location information provided by a second application that is different from the application.

[0155] (D30) In some embodiments of the electronic device of any one of D28 to D29, the search interface also includes one or more suggested applications.

[0156] (D31) In some embodiments of an electronic device of any one of D28 to D30, the group of historical search queries is selected (e.g., by selection unit 4419) at least in part based on the frequency of recent search queries.

[0157] (D32) In some embodiments of the electronic device of any one of D21 to D31, the processing unit is further configured to: in conjunction with entering the search mode, obtain (for example, through acquisition unit 4415) information associated with the content by using one or more auxiliary functions available on the electronic device.

[0158] (D33) In some embodiments of the electronic device of D32, using one or more auxiliary functions includes using one or more auxiliary functions to generate information associated with content by: (i) applying a natural language processing algorithm to text content currently displayed in an application, and (ii) determining (e.g., by determining unit 4417) one or more keywords describing the content using data obtained from the natural language processing algorithm (e.g., by obtaining unit 4415), wherein at least one suggested search query is determined based on the one or more keywords (e.g., by determining unit 4417).

[0159] (D34) In some embodiments of the electronic device of D33, determining one or more keywords describing the content also includes: (i) retrieving (e.g., by retrieval unit 4411) metadata corresponding to the non-text content currently displayed in the application, and (ii) determining (e.g., by determination unit 4417) one or more keywords using the retrieved metadata in addition to data obtained from the natural language processing algorithm.

[0160] (D35) In some embodiments of the electronic device of any one of D21 to D34, the search interface also includes one or more trend queries.

[0161] (D36) In some embodiments of the electronic device of D35, the search interface also includes one or more applications predicted to be of interest to the user of the electronic device.

[0162] (E1) According to some embodiments, on a device having a touch-sensitive surface (e.g., touch-sensitive surface 195, Figure 1D ) and a display (e.g., display 194, Figure 1D ) of electronic devices (e.g. Figure 1A The portable multifunctional device 100, according to Figure 1EThe method is performed on a computing device A to D of any one configuration). The method includes: detecting a swipe gesture on a user interface through a touch-sensitive surface, and the swipe gesture, when detected, causes the electronic device to enter a search mode. The method also includes: entering the search mode in response to detecting the swipe gesture, and entering the search mode includes: populating a search interface different from the user interface with a first content item before receiving any user input in the search interface. In some embodiments, based on determining that the user interface includes content associated with an application, wherein the user interface is different from a home screen that includes an optional icon for invoking the application, populating the search interface with the first content item includes populating the search interface with the at least one suggested search query, wherein the search query is based at least in part on the content associated with the application; and based on determining that the user interface is associated with a page of the home screen, populating the search interface with the first content item includes populating the search interface with an enable representation that includes an optional description of at least one point of interest within a threshold distance of the current location of the electronic device.

[0163] (E2) In some embodiments of the E1 method, filling the search interface with the enable representation includes displaying a search entry portion of the search interface on a touch-sensitive surface; and the method also includes: detecting input into the search entry portion; and in response to detecting input into the search entry portion, stopping displaying the enable representation and displaying the at least one suggested search query within the search interface.

[0164] (E3) In another aspect, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive surface, a display, one or more processors, and a memory storing one or more programs, which, when executed by the one or more processors, causes the electronic device to perform the method described in any one of E1 to E2.

[0165] (E4) On the other hand, an electronic device is provided, and the electronic device includes: a touch-sensitive surface, a display, and a device for performing any one of the methods described in E1 to E2.

[0166] (E5) In another aspect, a non-transitory computer-readable storage medium is provided, which is used to store executable instructions, which, when executed by an electronic device having a touch-sensitive surface and a display, will cause the electronic device to perform any of the methods described in E1 to E2.

[0167] (E6) On the other hand, a graphical user interface on an electronic device having a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of E1 to E2. On the other hand, an information processing device used in an electronic device including a touch-sensitive surface and a display is provided. The information processing device includes: a device for performing the method described in any one of E1 to E2.

[0168] (E7) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.45 display unit 4501), a touch-sensitive surface unit (e.g., Fig.45 touch-sensitive surface unit 4503) and a processing unit (e.g., Fig.45 The processing unit 4505 is coupled to the touch-sensitive surface unit and the display unit. Figure 1E Any of the computing devices shown (i.e., computing devices A to D) configures an electronic device. For ease of explanation, Fig.45 A display unit 4501 and a touch-sensitive surface unit 4503 are shown integrated with an electronic device 4500, however, in some embodiments, one or both of these units communicate with the electronic device, although these units remain physically separated from the electronic device. In some embodiments, the touch-sensitive surface unit and the display unit are integrated into a single touch-sensitive display unit (also referred to herein as a touch-sensitive display). The processing unit includes a detection unit (e.g., Fig.45 Detection unit 4507), display unit (e.g., Fig.45 Display unit 4509), filling unit (for example, Fig.45 Fill cell 4511) and search pattern entry cell (e.g., Fig.45The processing unit (or one or more components thereof, such as units 4507 to 4513) is configured to: detect (e.g., by detecting unit 4507) a swipe gesture on the user interface via a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4503), wherein when the swipe gesture is detected, the electronic device enters a search mode; and in response to detecting the swipe gesture, enter a search mode (e.g., by search mode entry unit 4513), wherein entering the search mode includes: before receiving any user input in the search interface, populating (e.g., by populating unit 4511) a search interface different from the user interface with the first content item. In some embodiments, based on determining that the user interface includes content associated with an application, wherein the user interface is different from a home screen that includes an optional icon for invoking the application, populating the search interface with the first content item includes populating (e.g., via populating unit 4511) the search interface with the at least one suggested search query, wherein the search query is based at least in part on the content associated with the application; and based on determining that the user interface is associated with a page of the home screen, populating the search interface with the first content item includes populating (e.g., via populating unit 4511) the search interface with an enable representation that includes an optional description of at least one point of interest within a threshold distance of a current location of the electronic device.

[0169] (E8) In some embodiments of the electronic device of E7, filling the search interface with the enable representation includes displaying (for example, through display unit 4507 and / or display unit 4501) a search entry portion of the search interface; and the processing unit is further configured to: detect (for example, through detection unit 4507) input in the search entry portion; and in response to detecting input in the search entry portion, stop displaying (for example, through display unit 4507 and / or display unit 4501) the enable representation and display (for example, through display unit 4507 and / or display unit 4501) the at least one suggested search query within the search interface.

[0170] (F1) According to some embodiments, in a device having a position sensor, a touch-sensitive surface (e.g., touch-sensitive surface 195, Figure 1D ) and a display (e.g., display 194, Figure 1D ) of electronic devices (e.g. Figure 1A The portable multifunctional device 100, according to Figure 1EThe method is executed on a computing device A to D of any one configuration). The method includes: automatically determining that a user of an electronic device is located in a vehicle that is parked at a certain geographic location without instructions from the user; when it is determined that the user has left the vehicle at the geographic location, determining whether the positioning information retrieved from the position sensor for identifying the geographic location meets the accuracy standard. The method also includes: when it is determined that the positioning information does not meet the accuracy standard, providing a prompt to the user to input information about the geographic location. The method also includes: in response to providing the prompt, receiving information about the geographic location from the user and storing the information as vehicle location information.

[0171] (F2) In some embodiments of the method of claim F1, the method further includes: when it is determined that the positioning information meets the accuracy standard, automatically storing the positioning information as vehicle position information without any instructions from the user.

[0172] (F3) In some embodiments of the method of claim F2, the method further includes: displaying a user interface object including vehicle location information based on determining that the user is heading to the geographic location.

[0173] (F4) In some embodiments of the method of claim F3, the user interface object is a map object that includes a user's current location identifier and a separate identifier for the geographic location.

[0174] (F5) In some embodiments of the method of claim F4, the user interface object is displayed on a lock screen of the electronic device.

[0175] (F6) In some embodiments of the method of claim F4, a user interface object is displayed in response to a swipe gesture that causes the electronic device to enter a search mode.

[0176] (F7) In some embodiments of the method of claim F6, in response to receiving the swipe gesture, a determination is performed to determine whether the user is heading to the geographic location.

[0177] (F8) In some embodiments of the method of any one of F1 to F7, the prompt is an audio prompt provided by a virtual assistant usable via an electronic device, receiving information from a user includes receiving a verbal description identifying the geographic location from the user, and displaying a user interface object includes displaying a selectable enable representation that, when selected, causes the device to playback the verbal description.

[0178] (F9) In some embodiments of the method of any one of F1 to F7, the prompt is displayed on a display of an electronic device, receiving information from a user includes receiving a text description identifying the geographic location from the user, and displaying a user interface object includes displaying the text description from the user.

[0179] (F10) In some embodiments of the method of any one of F1 to F7, determining whether the user is heading to the geographic location includes: using new positioning information received from the position sensor to determine that the electronic device is moving toward the geographic location.

[0180] (F11) In some embodiments of the method of F10, determining whether the user is heading to the geographic location includes: (i) determining that the electronic device has stayed at another geographic location for more than a threshold time period, and (ii) determining that new positioning information indicates that the electronic device is leaving the other geographic location and moving toward the geographic location.

[0181] (F12) In some embodiments of the method of any one of F1 to F11, determining that the user is in a vehicle that is stationary at the geographic location includes: (i) determining that the user is in the vehicle by determining that the electronic device is traveling at a speed greater than a threshold speed (ii) determining that the vehicle has stationary at the geographic location by one or more of the following: (a) determining that the electronic device has stationary at the geographic location for more than a threshold time period, (b) determining that the communication link between the electronic device and the vehicle has been disconnected, and (c) determining that the geographic location corresponds to a location within a parking lot.

[0182] (F13) In some embodiments of the method of claim F12, determining that the vehicle has stayed at the geographic location includes determining that the electronic device has stayed at the geographic location for more than a threshold time period.

[0183] (F14) In some embodiments of the method of any one of F12 to F13, determining that the vehicle has stopped at the geographic location includes determining that a communication link between the electronic device and the vehicle has been disconnected.

[0184] (F15) In some embodiments of the method of any one of F12 to F14, determining that the vehicle has stopped at the geographic location includes determining that the geographic location corresponds to a location within a parking lot.

[0185] (F16) In some embodiments of the method of any one of F1 to F15, the accuracy criterion includes a criterion that is met when an accuracy of a GPS reading associated with the positioning information is above a threshold level of accuracy.

[0186] (F17) In another aspect, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive surface, a display, a position sensor, one or more processors, and a memory storing one or more programs, which, when executed by the one or more processors, causes the electronic device to perform the method described in any one of F1 to F16.

[0187] (F18) On the other hand, an electronic device is provided, and the electronic device includes: a touch-sensitive surface, a display, a position sensor, and a device for performing any method described in F1 to F16.

[0188] (F19) In another aspect, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium is used to store executable instructions, which, when executed by an electronic device having a touch-sensitive surface, a display, and a position sensor, will cause the electronic device to perform the method described in any one of F1 to F16.

[0189] (F20) On the other hand, a graphical user interface is provided on an electronic device having a touch-sensitive surface, a display, and a position sensor. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of F1 to F16. On the other hand, an information processing device used in an electronic device including a touch-sensitive surface, a display, and a position sensor is provided. The information processing device includes: a device for performing the method described in any one of F1 to F16.

[0190] (F21) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.46 display unit 4601), a touch-sensitive surface unit (e.g., Fig.46 touch-sensitive surface unit 4603), position sensor unit (e.g., Fig.46 position sensor unit 4607) and a processing unit (e.g., Fig.46 The processing unit 4605 is coupled to the touch-sensitive surface unit, the display unit, and the position sensor unit. Figure 1E Any of the computing devices shown (i.e., computing devices A to D) configures an electronic device. For ease of explanation, Fig.46 A display unit 4601 and a touch-sensitive surface unit 4603 are shown integrated with an electronic device 4600, however, in some embodiments, one or both of these units communicate with the electronic device even though these units remain physically separated from the electronic device. In some embodiments, the touch-sensitive surface unit and the display unit are integrated into a single touch-sensitive display unit (also referred to herein as a touch-sensitive display). The processing unit includes a display unit (e.g., Fig.46 Display unit 4609), retrieval unit (for example, Fig.46 retrieval unit 4611), a determination unit (e.g., Fig.46 A determination unit 4613), a storage unit (eg, Fig.46 storage unit 4615), an identification unit (e.g., Fig.46 4617), a selection unit (e.g., Fig.46The selection unit 4619), the receiving unit (for example, Fig.46 A receiving unit 4621), a providing unit (for example, Fig.46 The providing unit 4623) and the playback unit (for example, Fig.46 The processing unit (or one or more components thereof, such as units 4607 to 4625) is configured to: automatically determine (e.g., by determining unit 4613) that the user of the electronic device is located in a vehicle that is parked at a certain geographic location without instructions from the user; upon determining that the user has left the vehicle at the geographic location, determine (e.g., by determining unit 4613) whether the positioning information retrieved (e.g., by retrieving unit 4611) from the position sensor unit (e.g., position sensor unit 4607) for identifying (e.g., by identifying unit 4617) the geographic location meets the accuracy standard; upon determining (e.g., by determining unit 4613) that the positioning information does not meet the accuracy standard, provide (e.g., by providing unit 4623) a prompt to the user to input information about the geographic location; and in response to providing the prompt, receive (e.g., by receiving unit 4621) information about the geographic location from the user and store (e.g., by storage unit 4615) the information as vehicle location information.

[0191] (F22) In some embodiments of the electronic device of F21, the processing unit is further configured to: when it is determined that the positioning information meets the accuracy standard, automatically store the positioning information (for example, through the storage unit 4615) as vehicle position information without any instructions from the user.

[0192] (F23) In some embodiments of the electronic device of F22, the processing unit is further configured to: based on determining that the user is heading to the geographic location, display (for example, through display unit 4609 in combination with display unit 4601) a user interface object including vehicle location information.

[0193] (F24) In some embodiments of the electronic device of F23, the user interface object is a map object that includes a user's current location identifier and a separate identifier for the geographic location.

[0194] (F25) In some embodiments of the electronic device of F24, the user interface object is displayed (for example, by display unit 4609 in combination with display unit 4601) on a lock screen of the electronic device.

[0195] (F26) In some embodiments of the electronic device of F24, in response to a swipe gesture that causes the electronic device to enter a search mode, a user interface object is displayed (e.g., via display unit 4609 in combination with display unit 4601).

[0196] (F27) In some embodiments of the electronic device of F26, in response to receiving the swipe gesture, a determination is performed to determine whether the user is heading to the geographic location.

[0197] (F28) In some embodiments of the electronic device of any one of F21 to F27, the prompt is an audio prompt provided by a virtual assistant that can be used through the electronic device, receiving information from the user includes receiving (e.g., via receiving unit 4621) a verbal description identifying the geographic location from the user, and displaying the user interface object includes displaying (e.g., via display unit 4609 in combination with display unit 4601) a selectable enable representation, which when selected (e.g., via selection unit 4619) causes the device to replay (e.g., via playback unit 4625) the verbal description.

[0198] (F29) In some embodiments of an electronic device of any one of F21 to F27, the prompt is displayed (for example, by display unit 4609 in combination with display unit 4601) on the display of the electronic device, receiving information from the user includes receiving (for example, by receiving unit 4621) a text description identifying the geographic location from the user, and displaying the user interface object includes displaying the text description from the user.

[0199] (F30) In some embodiments of the electronic device of any one of F21 to F27, determining whether the user is heading to the geographic location includes: using new positioning information received (e.g., through receiving unit 4621) from a position sensor unit (e.g., position sensor unit 4607) to determine (e.g., through determining unit 4613) that the electronic device is moving toward the geographic location.

[0200] (F31) In some embodiments of the electronic device of F30, determining whether the user is heading to the geographic location includes: (i) determining (e.g., by determining unit 4613) that the electronic device has stayed at another geographic location for more than a threshold time period, and (ii) determining (e.g., by determining unit 4613) that new positioning information indicates that the electronic device is leaving the other geographic location and moving toward the geographic location.

[0201] (F32) In some embodiments of the electronic device of any one of F21 to F31, determining that the user is in a vehicle that is stationary at the geographic location includes: (i) determining that the user is in the vehicle by determining (e.g., by determining unit 4613) that the electronic device is traveling at a speed greater than a threshold speed (ii) determining that the vehicle has stationary at the geographic location by one or more of the following: (a) determining (e.g., by determining unit 4613) that the electronic device has stationary at the geographic location for longer than a threshold time period, (b) determining (e.g., by determining unit 4613) that a communication link between the electronic device and the vehicle has been disconnected, and (c) determining (e.g., by determining unit 4613) that the geographic location corresponds to a location within a parking lot.

[0202] (F33) In some embodiments of the electronic device of F32, determining that the vehicle has stayed at the geographic location includes determining (e.g., by determining unit 4613) that the electronic device has stayed at the geographic location for more than a threshold time period.

[0203] (F34) In some embodiments of the electronic device of any one of F32 to F33, determining that the vehicle has stopped at the geographic location includes determining (e.g., by determining unit 4613) that the communication link between the electronic device and the vehicle has been disconnected.

[0204] (F35) In some embodiments of the electronic device of any one of F32 to F34, determining that the vehicle has stopped at the geographic location includes determining (e.g., by determining unit 4613) that the geographic location corresponds to a location within the parking lot.

[0205] (F36) In some embodiments of the electronic device of any one of F21 to F35, the accuracy criterion includes a criterion that is met when an accuracy of a GPS reading associated with the positioning information is above a threshold level of accuracy.

[0206] (G1) According to some embodiments, in a device having a position sensor, a touch-sensitive surface (e.g., touch-sensitive surface 195, Figure 1D ) and a display (e.g., display 194, Figure 1D ) of electronic devices (e.g. Figure 1A The portable multifunctional device 100, according to Figure 1EThe method is performed on a computing device A to D of any one configuration). The method includes: monitoring the geographic location of the electronic device using a location sensor. The method also includes: determining that the electronic device is within a threshold distance of a predetermined type of point of interest based on the monitored geographic location. The method also includes: based on determining that the electronic device is within the threshold distance of the point of interest: identifying at least one activity currently popular at the point of interest and retrieving information about the point of interest, including retrieving information about at least one activity currently popular at the point of interest. The method also includes: detecting a first input through a touch-sensitive surface, the first input causing the electronic device to enter a search mode when detected; and in response to detecting the first input, entering a search mode, wherein entering the search mode includes, before any user input is received at the search interface, presenting an enabling representation through a display including the following items: (i) information about the at least one activity, and (ii) an indication that the at least one activity has been determined to be currently popular at the point of interest.

[0207] (G2) In some embodiments of the method of claim G1, the method includes: detecting a second input; and in response to detecting the second input, updating the enable representation to include available information about current activity at a second point of interest that is different from the point of interest, and the point of interest is within a threshold distance of the electronic device.

[0208] (G3) In some embodiments of the method of any one of G1 to G2, the enable representation also includes selectable point of interest categories, and the method also includes: detecting a selection of the corresponding selectable category; and in response to detecting the selection, updating the enable representation to include information about additional points of interest, wherein the additional points of interest are located within a second threshold distance of the device.

[0209] (G4) In some embodiments of the method of any one of G1 to G3, the point of interest is an amusement park, and the retrieved information includes current wait times for rides at the amusement park.

[0210] (G5) In some embodiments of the method of claim G4, the retrieved information includes information about waiting times for rides located within a predefined distance of the electronic device.

[0211] (G6) In some embodiments of the method of any one of G1 to G3, the point of interest is a restaurant, and the retrieved information includes information about popular menu items in the restaurant.

[0212] (G7) In some embodiments of the method of claim G6, the retrieved information is information retrieved from a social network associated with the user of the electronic device.

[0213] (G8) In some embodiments of the method of any one of G1 to G3, the point of interest is a movie theater, and the retrieved information includes information about the movie theater's showtimes.

[0214] (G9) In some embodiments of the method of claim G8, the retrieved information is information retrieved from a social network associated with the user of the electronic device.

[0215] (G10) In some embodiments of the method of any one of G1 to G9, after unlocking the electronic device, in response to a light swipe in a generally horizontal direction on an initial page of a home screen of the electronic device, the indication is enabled and available.

[0216] (G11) In another aspect, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive surface, a display, a position sensor, one or more processors, and a memory storing one or more programs, which, when executed by the one or more processors, causes the electronic device to perform the method described in any one of G1 to G10.

[0217] (G12) On the other hand, an electronic device is provided, and the electronic device includes: a touch-sensitive surface, a display, a position sensor, and a device for executing any one of the methods described in G1 to G10.

[0218] (G13) In another aspect, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium is used to store executable instructions, which, when executed by an electronic device having a touch-sensitive surface, a display, and a position sensor, will cause the electronic device to perform the method described in any one of G1 to G10.

[0219] (G14) On the other hand, a graphical user interface is provided on an electronic device having a touch-sensitive surface, a display, and a position sensor. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of G1 to G10. On the other hand, an information processing device used in an electronic device including a touch-sensitive surface, a display, and a position sensor is provided. The information processing device includes: a device for performing the method described in any one of G1 to G10.

[0220] (G15) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.47 display unit 4701), a touch-sensitive surface unit (e.g., Fig.47 touch-sensitive surface unit 4703), position sensor unit 4707, and processing unit (e.g., Fig.47 The processing unit 4705 is coupled to the touch-sensitive surface unit, the display unit, and the position sensor unit. Figure 1E Any of the computing devices shown (i.e., computing devices A to D) configures an electronic device. For ease of explanation, Fig.47 A display unit 4701 and a touch-sensitive surface unit 4703 are shown integrated with an electronic device 4700, however, in some embodiments, one or both of these units communicate with the electronic device even though these units remain physically separated from the electronic device. In some embodiments, the touch-sensitive surface unit and the display unit are integrated into a single touch-sensitive display unit (also referred to herein as a touch-sensitive display). The processing unit includes a detection unit (e.g., Fig.47 Detection unit 4709), display unit (e.g., Fig.47 Display unit 4711), retrieval unit (for example, Fig.47 retrieval unit 4713), a determination unit (e.g., Fig.47 A determination unit 4715), an identification unit (e.g., Fig.47 Identification unit 4717), unlocking unit (e.g., Fig.47 unlocking unit 4719) and search mode entry unit (e.g., Fig.47 The processing unit (or one or more components thereof, such as units 4709 to 4721) is configured to: without receiving any instructions from a user of the electronic device: monitor the geographic location of the electronic device using a location sensor unit (e.g., location sensor unit 4707); determine (e.g., by determining unit 4715) that the electronic device is within a threshold distance of a predetermined type of point of interest based on the monitored geographic location; based on determining that the electronic device is within the threshold distance of the point of interest: identify (e.g., by identifying unit 4717) at least one activity currently popular at the point of interest; retrieve (e.g., by retrieving unit 4713) information about the point of interest, including retrieving information about at least one activity currently popular at the point of interest information about an activity; detecting (e.g., through detection unit 4709) a first input through a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4703), the first input causing the electronic device to enter a search mode when detected; and in response to detecting the first input, entering a search mode (e.g., through search mode entry unit 4721), and entering the search mode includes presenting (e.g., through display unit 4711) an enable representation including the following items before any user input is received at the search interface through a display unit (e.g., display unit 4701): (i) information about at least one activity, and (ii) an indication that at least one activity has been determined to be a currently popular activity at the point of interest.

[0221] (G16) In some embodiments of the electronic device of G15, the processing unit is further configured to: detect (e.g., via detection unit 4709) a second input; and in response to detecting the second input, update (e.g., via display unit 4711) the enable representation to include available information about current activity at a second point of interest that is different from the point of interest, and the point of interest is within a threshold distance of the electronic device.

[0222] (G17) In some embodiments of the electronic device of any one of G15 to G16, the enable representation also includes selectable point of interest categories, and the processing unit is further configured to: detect (e.g., by detection unit 4709) a selection of the corresponding selectable category; and in response to detecting the selection, update (e.g., by display unit 4711) the enable representation to include information about additional points of interest, wherein the additional points of interest are located within a second threshold distance of the device.

[0223] (G18) In some embodiments of the electronic device of any one of G15 to G17, the point of interest is an amusement park, and the retrieved information includes current waiting times for rides in the amusement park.

[0224] (G19) In some embodiments of the electronic device of G18, the retrieved information includes information about waiting times for rides located within a predefined distance of the electronic device.

[0225] (G20) In some embodiments of the electronic device of any one of G15 to G17, the point of interest is a restaurant, and the retrieved information includes information about popular menu items in the restaurant.

[0226] (G21) In some embodiments of the electronic device of G20, the retrieved information is information retrieved from a social network associated with the user of the electronic device.

[0227] (G22) In some embodiments of the electronic device of any one of G15 to G17, the point of interest is a movie theater, and the retrieved information includes information about the movie theater's showtimes.

[0228] (G23) In some embodiments of the electronic device of G22, the retrieved information is information retrieved from a social network associated with the user of the electronic device.

[0229] (G24) In some embodiments of the electronic device of any one of G15 to G23, after the electronic device is unlocked (e.g., by unlocking unit 4719), in response to a light swipe in a generally horizontal direction on an initial page of a home screen of the electronic device, the indication is available.

[0230] (H1) According to some embodiments, in a device having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, such as in Figure 1C The touch screen 112 is shown) of an electronic device (eg, Figure 1A The portable multifunctional device 100, according to Figure 1E A method is performed on a computing device A to D of any one of the configurations of . The method includes: receiving at least a portion of a voice communication (e.g., a real-time telephone call of 10 seconds or less or a recorded voicemail), the portion of the voice communication including speech provided by a remote user of a remote device, the remote user being different from a user of the electronic device. The method also includes: extracting a content item based at least in part on the speech provided by the remote user of the remote device. The method also includes: determining whether the content item is currently available on the electronic device. Based on determining that the content item is not currently available on the electronic device, the method includes: (i) identifying an application associated with the content item, and (ii) displaying an optional description of the content item on a display. In response to detecting a selection of the optional description, the method includes: storing the content item for presentation by the identified application (e.g., storing the content item so that it can be presented by the identified application). In this way, a user is able to store content items that have been mentioned or discussed in a voice communication without having to remember all the details of the discussion and then enter those details to create an appropriate content item. Instead, the electronic device is able to detect and extract content items based on speech describing various corresponding content items, and then provide selectable descriptions of the content items that can be selected by the user in order to store the corresponding content items on the electronic device.

[0231] (H2) In some implementations of the method of H1, the content item is a new event.

[0232] (H3) In some embodiments of the method of H1, the content item is new event details for an event currently associated with a calendar application on the electronic device.

[0233] (H4) In some implementations of the method of H1, the content item is a new contact.

[0234] (H5) In some embodiments of the method of H1, the content item is new contact information for an existing contact associated with a phone application on the electronic device.

[0235] (H6) In some embodiments of the method of any one of H1 to H5, the voice communication is a real-time telephone call.

[0236] (H7) In some embodiments of the method of any one of H1 to H5, the voice communication is a real-time FaceTime call.

[0237] (H8) In some embodiments of the method of any one of H1 to H5, the voice communication is a recorded voice mailbox.

[0238] (H9) In some embodiments of the method of any one of H1 to H8, displaying the optional description includes displaying the optional description within a user interface that includes recent calls made using a phone application. In this way, a user can easily and conveniently access the extracted content items (e.g., content items extracted during a corresponding voice communication) directly through the user interface that includes the recent calls.

[0239] (H10) In some embodiments of the method of H9, an optional description is displayed along with the following indication: the content item is associated with voice communication.

[0240] (H11) In some embodiments of the method of H9, detecting a selection includes: receiving a selection while displaying a user interface including a recent call.

[0241] (H12) In some embodiments of the method of any one of H1 to H11, the method further includes: providing feedback to the user of the electronic device that the content item has been detected (e.g., tactile feedback generated by the electronic device or a user interface object presented on the second device so that the user does not have to remove the phone from his ear during the phone call) in conjunction with displaying an optional description of the content item. In this way, a concise indication is provided to the user that the content item has been detected / extracted during the voice communication, and the user can then decide whether to store the content item.

[0242] (H13) In some embodiments of the method of H12, providing feedback includes sending information related to the detection of the content item to another electronic device (e.g., a laptop, a television monitor, a smart watch, etc.) near the electronic device. In this way, the user does not have to interrupt the voice communication, but can still view details related to the detected / extracted content item on the other device.

[0243] (H14) In some embodiments of the method of any one of H1 to H13, the method further includes: determining that the voice communication includes information about the first physical location (e.g., an address mentioned during the phone call or a restaurant name discussed during the phone call, etc., more details are provided below). The method further includes: detecting input, and in response to detecting the input, opening an application capable of accepting location data and populating the application with information about the first physical location. In this way, in addition to detecting and extracting event and contact information, the electronic device is also able to extract the location information discussed during the voice communication and provide the location information to the user in an appropriate application (e.g., to spare the user the trouble of remembering specific location details discussed on the phone, particularly new details that the user may not be familiar with, the device extracts these location details and displays them for the user to use).

[0244] (H15) In some embodiments of the method of H14, the application is a map application, and populating the map application with information related to the first physical location includes: populating a map displayed within the map application with a location identifier corresponding to the first physical location. In this way, the user can conveniently use the newly extracted location details to travel to a new destination, see how far a specific location is, and use other functions provided by the map application.

[0245] (H16) In some embodiments of the method of any one of H1 to H13, the method further includes: determining that the voice communication includes information related to the first physical location. The method further includes: detecting an input (e.g., a search activation gesture, such as the swipe gesture discussed in detail below), and in response to detecting the input, populating a search interface with information about the first physical location. In this way, in addition to (or as an alternative to) providing location information for a particular application to a user, the electronic device is also able to provide location information for use in a search interface (e.g., for searching for related points of interest or searching for more details about the first physical location, such as a phone number, a menu, etc.).

[0246] (H17) In some embodiments of the method of any one of H1 to H16, extracting the content item includes analyzing the portion of the voice communication to detect a predetermined type of content, and performing the analysis while the voice communication is output through an audio system that communicates with the electronic device (for example, analyzing the voice communication in real time while the voice communication is output to a user of the electronic device).

[0247] (H18) In some embodiments of the method of H17, analyzing voice communications includes: (i) converting speech provided by a remote user of a remote device into text; (ii) applying a natural language processing algorithm to the text to determine whether the text includes one or more predefined keywords; and (iii) determining that the voice communication includes speech describing a content item based on determining that the text includes corresponding predefined keywords.

[0248] (H19) In some embodiments of the method of any one of H1 to H18, receiving at least the portion of the voice communication includes receiving an indication from a user of the electronic device that the portion of the voice communication should be analyzed.

[0249] (H20) In some embodiments of the method of H19, the indication corresponds to selection of a hardware button (e.g., a user selects a hardware button while the audio system is outputting a voice communication that indicates the number of seconds of a predetermined voice communication that should be analyzed (e.g., the previous 10, 15, or 20 seconds)). In some embodiments, the button may also be a button presented on an electronic device display for user selection (e.g., a button that displays "Tap here to analyze voice communication for new content" during a voice communication).

[0250] (H21) In some embodiments of the method of H19, the instruction corresponds to a command from a user of the electronic device, including the words “Hey, Siri.” Thus, the user can easily instruct the electronic device to begin analyzing the portion of the voice communication to detect content items discussed in the voice communication (such as events, contact information, and information about physical locations).

[0251] (H22) In some embodiments of the method of any one of H1 to H21, the method further includes: receiving a second portion of the voice communication, the second portion including voice provided by a remote user of the remote device and voice provided by a user of the electronic device (for example, the voice communication is a real-time telephone call, and the second portion includes a discussion between the user and the remote user). The method further includes: extracting a second content item based at least in part on the voice provided by the remote user of the remote device and the voice provided by the user of the electronic device. Based on determining that the second content item is not currently available on the electronic device, the method includes: (i) identifying a second application associated with the second content item, and (ii) displaying a second optional description of the second content item on the display. In response to detecting a selection of the second optional description, the method includes: storing the second content item for presentation by the identified second application.

[0252] (H23) In some embodiments of the method of H22, the optional description and the second optional description are displayed within a user interface, wherein the user interface includes recent calls made using a phone application. In this way, a single interface is provided to the user that conveniently includes content items detected across multiple voice communications (e.g., multiple phone calls, voicemail, or phone calls and voicemail).

[0253] (H24) On the other hand, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive surface, a display, one or more processors, and a memory storing one or more programs, wherein when the one or more processors execute the program, the electronic device will perform the method described in any one of H1 to H23.

[0254] (H25) On the other hand, an electronic device is provided, comprising: a touch-sensitive surface, a display, and a device for executing any one of the methods described in H1 to H23.

[0255] (H26) In another aspect, a non-transitory computer-readable storage medium is provided, which is used to store executable instructions, which, when executed by an electronic device having a touch-sensitive surface and a display, will cause the electronic device to perform the method described in any one of H1 to H23.

[0256] (H27) On the other hand, a graphical user interface on an electronic device having a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of H1 to H23.

[0257] (H28) In another aspect, an information processing device for use in an electronic device including a touch-sensitive surface and a display is provided. The information processing device includes: a device for performing the method described in any one of H1 to H23.

[0258] (H29) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.48 display unit 4801), a touch-sensitive surface unit (e.g., Fig.48 touch-sensitive surface unit 4803) and a processing unit (e.g., Fig.48 In some embodiments, according to Figure 1E Any of the computing devices shown (e.g., computing devices A to D) configures an electronic device. For ease of illustration, Fig.48A display unit 4801 and a touch-sensitive surface unit 4803 are shown integrated with the electronic device 4800, however, in some embodiments, one or both of these units communicate with the electronic device even though these units remain physically separated from the electronic device. The processing unit includes a voice communication receiving unit (e.g., Fig.48 voice communication receiving unit 4807), a content item extraction unit (e.g., Fig.48 content item extraction unit 4809), availability determination unit (e.g., Fig.48 Availability determination unit 4811), application identification unit (e.g., Fig.48 application recognition unit 4813), a display unit (e.g., Fig.48 display unit 4815), a content item storage unit (e.g., Fig.48 content item storage unit 4817), a feedback providing unit (e.g., Fig.48 Feedback providing unit 4819), input detection unit (e.g., Fig.48 Input detection unit 4821), application opening unit (for example, Fig.48 application opening unit 4823), a filling unit (e.g., Fig.48 Filling unit 4825) and voice communication analysis unit (e.g., Fig.48 The processing unit (or one or more components thereof, such as units 4807 to 4827) is configured to: receive at least a portion of the voice communication (e.g., via voice communication receiving unit 4807), the portion of the voice communication comprising speech provided by a remote user of a remote device different from the user of the electronic device. The processing unit is also configured to: extract a content item based at least in part on the speech provided by the remote user of the remote device (e.g., via content item extraction unit 4809), and determine whether the content item is currently available on the electronic device (e.g., via availability determination unit 4811). Based on determining that the content item is not currently available on the electronic device, the processing unit is also configured to: (i) identify an application associated with the content item (e.g., via application identification unit 4813), and (ii) display an optional description of the content item on a display (e.g., via display unit 4815 and / or display unit 4801). In response to detecting selection of the optional description (e.g., via input detection unit 4821 and / or touch-sensitive surface unit 4803), the processing unit is configured to: store the content item for presentation via the identified application (e.g., via content item storage unit 4817).

[0259] (H30) In some embodiments of the electronic device of H29, the processing unit (or one or more components thereof, such as units 4907 to 4927) is further configured to perform a method described in any one of H2 to H23.

[0260] (I1) According to some embodiments, in a device having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, such as in Figure 1C The touch screen 112 is shown) of an electronic device (eg, Figure 1A The portable multifunctional device 100, according to Figure 1E A method is performed on a computing device A to D of any one of the configurations of the electronic device. The method includes: receiving at least a portion of a voice communication, the portion of the voice communication (e.g., a real-time telephone call, a recorded voicemail) including voice provided by a remote user of a remote device, the remote user being different from a user of the electronic device. The method also includes: determining that the voice communication includes voice identifying a first physical location. In response to determining that the voice communication includes voice identifying the physical location, the method includes: providing an indication (e.g., providing tactile feedback to a user, displaying a user interface object with information about the physical location, or sending information about the physical location to a nearby device for display on the nearby device) indicating that information about the physical location has been detected. The method further includes: detecting input through a touch-sensitive surface. In response to detecting the input, the method includes: (i) opening an application that accepts geographic location data; and (ii) populating the application with information about the physical location. In this way, users are able to store information about physical locations that have been mentioned or discussed in voice communications without having to remember all the details discussed and then enter those details in the appropriate application. Rather, the electronic device is able to detect and extract information about a physical location based on speech describing the physical location (e.g., a description of a restaurant, driving directions to a physical location, etc.), and then provide an indication that information related to the corresponding physical location has been detected.

[0261] (I2) In some embodiments of the method of I1, the voice communication is a real-time telephone call.

[0262] (I3) In some embodiments of the method of I1, the voice communication is a real-time FaceTime call.

[0263] (I4) In some embodiments of the method of I1, the voice communication is a recorded voice mailbox.

[0264] (I5) In some embodiments of the method of any of I1 to I4, providing an indication includes displaying an optional description of the physical location within a user interface, wherein the user interface includes recent calls made using a phone application.

[0265] (I6) In some embodiments of the method of I5, the optional description indicates that the content item is associated with voice communication.

[0266] (I7) In some embodiments of the method of any one of I5 to I6, detecting input includes: detecting input on the optional description while displaying a user interface including recent calls.

[0267] (I8) In some embodiments of the method of any one of I1 to I7, providing the indication includes providing tactile feedback to a user of the electronic device.

[0268] (I9) In some embodiments of the method of any one of I1 to I8, providing the indication includes sending information about the physical location to another electronic device that is proximate to the electronic device.

[0269] (I10) In some embodiments of the method of any one of I1 to I9, determining that the voice communication includes voice describing a physical location includes: analyzing the portion of the voice communication to detect information about the physical location, and performing the analysis while outputting the voice communication through an audio system that communicates with the electronic device.

[0270] (I11) In some embodiments of the method of any one of I1 to I10, receiving at least the portion of the voice communication includes receiving an instruction from a user of the electronic device that the portion of the voice communication should be analyzed.

[0271] (I12) In some embodiments of the method of I11, the instruction corresponds to the selection of a hardware button. In some embodiments, the button may also be a button presented on an electronic device display for the user to select (e.g., a button that displays "Tap here to analyze voice communication for new content" during voice communication).

[0272] (I13) In some embodiments of the method of I11, the instruction corresponds to a command from a user of the electronic device, including the words "Hey, Siri."

[0273] (I14) On the other hand, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive surface, a display, one or more processors, and a memory storing one or more programs, wherein when the one or more processors execute the program, the electronic device will perform the method described in any one of I1 to I13.

[0274] (I15) On the other hand, an electronic device is provided, comprising: a touch-sensitive surface, a display, and a device for performing any of the methods described in I1 to I13.

[0275] (I16) In another aspect, a non-transitory computer-readable storage medium is provided, which is used to store executable instructions, which, when executed by an electronic device having a touch-sensitive surface and a display, will cause the electronic device to perform any of the methods described in I1 to I13.

[0276] (I17) On the other hand, a graphical user interface on an electronic device having a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of I1 to I13.

[0277] (I18) In another aspect, an information processing device for use in an electronic device including a touch-sensitive surface and a display is provided. The information processing device includes: a device for performing the method described in any one of I1 to I13.

[0278] (I19) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.49 display unit 4901), a touch-sensitive surface unit (e.g., Fig.49 touch-sensitive surface unit 4903) and a processing unit (e.g., Fig.49 In some embodiments, according to Figure 1E Any of the computing devices shown (e.g., computing devices A to D) configures an electronic device. For ease of illustration, Fig.49 A display unit 4901 and a touch-sensitive surface unit 4903 are shown integrated with the electronic device 4900, however, in some embodiments, one or both of these units communicate with the electronic device even though these units remain physically separate from the electronic device. The processing unit includes a voice communication receiving unit (e.g., Fig.49 voice communication receiving unit 4907), a content item extraction unit (e.g., Fig.49 content item extraction unit 4909), an instruction providing unit (e.g., Fig.49 An instruction providing unit 4911), an input detection unit (e.g., Fig.49 Input detection unit 4913), application opening unit (for example, Fig.49 application opening unit 4915), an application filling unit (e.g., Fig.49 application filling unit 4917), a feedback providing unit (e.g., Fig.49 Feedback providing unit 4919) and voice communication analysis unit (e.g., Fig.49The processing unit (or one or more components thereof, such as units 4907 to 4921) is configured to: receive at least a portion of the voice communication (e.g., via voice communication receiving unit 4907), the portion of the voice communication comprising speech provided by a remote user of a remote device that is different from the user of the electronic device. The processing unit is also configured to: determine that the voice communication includes speech that identifies a physical location (e.g., via content item extraction unit 4909). In response to determining that the voice communication includes speech that identifies the physical location, the processing unit is configured to: provide an indication that information about the physical location has been detected (e.g., via content item extraction unit 4909). The processing unit is also configured to: detect input via the touch-sensitive surface unit (e.g., via input detection unit 4911). In response to detecting the input, the processing unit is configured to: (i) open an application that accepts geographic location data (e.g., via application opening unit 4913), and (ii) populate the application with information about the physical location (e.g., via application populating unit 4915).

[0279] (I20) In some embodiments of the electronic device of I19, the processing unit (or one or more components thereof, such as units 4907 to 4921) is further configured to execute a method described in any one of I2 to I13.

[0280] (J1) According to some embodiments, in a device having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, such as in Figure 1C The touch screen 112 is shown) of an electronic device (eg, Figure 1A The portable multifunctional device 100, according to Figure 1E The method is performed on a computing device A to D of any one configuration). The method includes: presenting a text input field and a conversation copy in an instant messaging application on a display. The method also includes: when the instant messaging application is presented on the display, determining that the next possible input from the user of the electronic device is information about a physical location. The method also includes: analyzing the content associated with the text input field and the conversation copy, and determining a suggested physical location based at least in part on a portion of the analyzed content. The method further includes: presenting an optional user interface element identifying the suggested physical location within the instant messaging application on the display, and receiving a selection of the optional user interface element. In response to receiving the selection, the method includes: presenting a representation of the suggested physical location in the text input field. In this way, the desired content can be conveniently provided to the user of the electronic device without having to type anything and without having to search for content (for example, the user only needs to select the optional user interface element to enter his current address without having to access a map application to determine his exact location and then switch back to the instant messaging application and provide explicit input to send the location information).

[0281] (J2) In some embodiments of the method of J1, the instant messaging application includes a virtual keyboard, and the selectable user interface elements are displayed in a suggestion portion adjacent to and above the virtual keyboard.

[0282] (J2) In some embodiments of the method of any one of J1 to J2, determining that the next possible input from the user of the electronic device is information about a physical location includes: processing content associated with the text input field and the conversation transcript to detect that the conversation transcript includes a question about the user's current location. In this way, suggested physical locations directly related to the discussion in the conversation transcript are provided to the user (for example, in response to a second user's question "Where are you?", a user interface object is presented to the user, which, when selected, causes the device to send information about the user's current location to the second user).

[0283] (J4) In some embodiments of the method of J3, processing the content includes applying a natural language processing algorithm to detect one or more predefined keywords that constitute a question (e.g., "Where are you?" or "What is your home address?").

[0284] (J5) In some embodiments of the method of any one of J3 to J4, the question is included in a message received from a second user different from the user.

[0285] (J6) In some embodiments of the method of any one of J1 to J5, determining that the next possible input from the user of the electronic device is information about a physical location includes: monitoring typed input received from the user in a text input portion of an instant messaging application.

[0286] (J7) In some embodiments of the method of any one of J1 to J6, the method further comprises: based on determining that the user is typing and has not yet selected the selectable user interface element, stopping presenting the selectable user interface element. Thus, if it can be determined that the user is not interested in selecting the object, the device will not continue to present the selectable user interface object.

[0287] (J8) In some embodiments of the method of any one of J1 to J7, the method further comprises: based on determining that the user has provided additional input, wherein the input indicates that the user will not select the selectable user interface element, stopping presenting the selectable user interface element. Thus, if it can be determined that the user is not interested in selecting the object, the device will not continue to present the selectable user interface object.

[0288] (J9) In some embodiments of the method of any one of J1 to J5, it is suggested that the representation of the physical location includes information for identifying the current geographic location of the electronic device.

[0289] (J10) In some embodiments of the method of any one of J1 to J9, the representation of the suggested physical location is an address.

[0290] (J11) In some embodiments of the method of any one of J1 to J9, the suggested physical location is a map object that includes an identifier of the suggested physical location.

[0291] (J12) In some embodiments of the method of any of J1 to J11, the suggested physical location corresponds to a location that the user recently viewed in an application other than an instant messaging application.

[0292] (J13) In some embodiments of the method of any one of J1 to J12, the instant messaging application is an email application.

[0293] (J14) In some embodiments of the method of any one of J1 to J12, the instant messaging application is a text messaging application.

[0294] (J15) On the other hand, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive surface, a display, one or more processors, and a memory storing one or more programs, wherein when the one or more processors execute the program, the electronic device will perform any method described in J1 to J14.

[0295] (J16) On the other hand, an electronic device is provided, comprising: a touch-sensitive surface, a display, and a device for executing any one of the methods described in J1 to J14.

[0296] (J17) In another aspect, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium is used to store executable instructions, which, when executed by an electronic device having a touch-sensitive surface and a display, will cause the electronic device to perform any method described in J1 to J14.

[0297] (J18) On the other hand, a graphical user interface on an electronic device having a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of J1 to J14.

[0298] (J19) In another aspect, an information processing device for use in an electronic device including a touch-sensitive surface and a display is provided. The information processing device includes: a device for executing the method described in any one of J1 to J14.

[0299] (J20) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.50 display unit 5001), a touch-sensitive surface unit (e.g., Fig.50 touch-sensitive surface unit 5003) and a processing unit (e.g., Fig.50 In some embodiments, according to Figure 1E Any of the computing devices shown (e.g., computing devices A to D) configures an electronic device. For ease of illustration, Fig.50 The display unit 5001 and the touch-sensitive surface unit 5003 are shown integrated with the electronic device 5000, however, in some embodiments, one or both of these units communicate with the electronic device even though these units remain physically separated from the electronic device. The processing unit includes a presentation unit (e.g., Fig.50 , a presentation unit 5007, a next input determination unit (eg, Fig.50 Next input determination unit 5009), content analysis unit (eg, Fig.50 content analysis unit 5011), selecting a receiving unit (eg, Fig.50 The selection receiving unit 5013), the input input monitoring unit (for example, Fig.50 The input monitoring unit 5015) and the rendering stopping unit (eg, Fig.50 The processing unit (or one or more components thereof, such as units 5007 to 5017) is configured to: present a text input field and a conversation copy in an instant messaging application on a display (e.g., via the presentation unit 5007 and / or the display unit 5001). When presenting the instant messaging application on the display, the processing unit is further configured to: determine that the next possible input from the user of the electronic device is information about a physical location (e.g., via the next input determination unit 5009). The processing unit is further configured to: analyze content associated with the text input field and the conversation copy, and determine a suggested physical location based at least in part on a portion of the analyzed content (e.g., via the content analysis unit 5011); present an optional user interface element identifying a suggested physical location within the instant messaging application on the display (e.g., via the presentation unit 5007); receive a selection of the optional user interface element (e.g., via the selection receiving unit 5013 and / or the touch-sensitive surface unit 5003); and in response to receiving the selection, present a representation of the suggested physical location in the text input field (e.g., via the presentation unit 5007).

[0300] (J21) In some embodiments of the electronic device of J20, the processing unit (or one or more components thereof, such as units 5007 to 5017) is further configured to execute the method described in any one of J2 to J14.

[0301] (K1) According to some embodiments, in a device having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, such as in Figure 1C The touch screen 112 is shown) of an electronic device (eg, Figure 1A The portable multifunctional device 100, according to Figure 1E The method is performed on a computing device A to D of any one of the configurations of the computing device A to D. The method includes: while displaying the first application, obtaining information for identifying a first physical location viewed by a user in the first application (for example, a restaurant that the user has searched for in an application that allows searching for local businesses). The method also includes: exiting the first application, and after exiting the first application, receiving a request from the user to open a second application that is different from the first application. In some embodiments, the request is received without receiving any input in the first application (for example, the request does not include clicking a link or button within the first application). In response to receiving the request and based on determining that the second application is capable of accepting geographic location information, the method includes: presenting the second application, and presenting the second application includes populating the second application with information based at least in part on the first physical location identification information. In this way, the user does not need to manually transfer information between two different applications. Instead, the device intelligently determines that the second application is capable of accepting geographic location information, and then directly populates the second application with information related to the physical location viewed in the first application (for example, populating a map object in the second application to include an identifier of the physical location).

[0302] (K2) In some embodiments of the method of K1, receiving a request to open the second application includes: after exiting the first application, detecting input on an affordance of the second application. In other words, the request does not correspond to clicking a link within the first application, but rather, the user explicitly and directly requests to open the second application, and then the device decides to populate the second application with information about a previously viewed physical location (previously viewed in a different first application), so that the user can further explore or investigate the previously viewed physical location in the second application.

[0303] (K3) In some embodiments of the method of K2, the enabling representation of the second application is an icon displayed on a home screen of the electronic device. In some embodiments, the home screen is a system-level component of an operating system, including icons for invoking applications available on the electronic device.

[0304] (K4) In some embodiments of the method of K2, detecting input includes: (i) detecting a double-click on a physical Home button, (ii) displaying an application switching user interface in response to detecting the double-click, and (iii) detecting a selection of an enable indication from within the application switching user interface.

[0305] (K5) In some embodiments of the method of any one of K1 to K4, populating the second application includes displaying a user interface object that includes information based at least in part on the first physical location identification information.

[0306] (K6) In some embodiments of the method of K5, the user interface object includes a text description that notifies the user that the first physical location was most recently viewed in the first application.

[0307] (K7) In some embodiments of the method of K6, the user interface object is a map displayed in the second application, and populating the second application includes populating the map to include an identifier of the first physical location.

[0308] (K8) In some embodiments of the method of any one of K6 to K7, the second application presents a virtual keyboard and the user interface object is displayed above the virtual keyboard.

[0309] (K9) In some embodiments of the method of any one of K6 to K8, obtaining information includes obtaining information about the second physical location, and displaying the user interface object includes displaying the user interface object having information related to the second physical location.

[0310] (K10) In some embodiments of the method of any one of K1 to K9, determining that the second application is capable of accepting geographic location information includes one or more of the following: (i) determining that the second application includes an input receiving field capable of accepting and processing geographic location data; (ii) determining that the second application is capable of displaying the geographic location information on a map; (iii) determining that the second application is capable of using the geographic location information to facilitate route guidance; and (iv) determining that the second application is capable of using the geographic location information to locate and provide transportation services.

[0311] (K11) In some embodiments of the K10 method, determining that the second application is capable of accepting geographic location information includes: determining that the second application includes an input receiving field capable of accepting and processing geographic location data, and the input receiving field is a search box that allows searching within a map displayed in the second application.

[0312] (K12) In some embodiments of the method of any one of K1 to K11, the method further includes: in response to receiving the request, determining whether the second application is associated with the first application based on the user's application usage history (e.g., opened a threshold number of times after opening the first application).

[0313] (K13) In some embodiments of the method of K12, the method further includes: providing the second application with access to information identifying the first physical location before presenting the second application, and before providing the access, the second application cannot access the information identifying the first physical location. In this way, the second application is able to receive information about actions performed by the user in the first application, thereby providing the user with a way to use the information in the second application (for example, searching for more information about the first physical location, or using the first physical location for some services available through the second application, such as ride-sharing).

[0314] (K14) In another aspect, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive surface, a display, one or more processors, and a memory storing one or more programs, wherein when the one or more processors execute the programs, the electronic device will perform any method described in K1 to K13.

[0315] (K15) On the other hand, an electronic device is provided, comprising: a touch-sensitive surface, a display, and a device for executing any one of the methods described in K1 to K13.

[0316] (K16) In another aspect, a non-transitory computer-readable storage medium is provided, which is used to store executable instructions, which, when executed by an electronic device having a touch-sensitive surface and a display, will cause the electronic device to perform any method described in K1 to K13.

[0317] (K17) On the other hand, a graphical user interface on an electronic device having a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of K1 to K13.

[0318] (K18) In another aspect, an information processing device for use in an electronic device including a touch-sensitive surface and a display is provided, wherein the information processing device includes: a device for performing the method described in any one of K1 to K13.

[0319] (K19) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.51 display unit 5101), a touch-sensitive surface unit (e.g., Fig.51 touch-sensitive surface unit 5103) and a processing unit (e.g., Fig.51 In some embodiments, according to Figure 1E Any of the computing devices shown (e.g., computing devices A to D) configures an electronic device. For ease of illustration, Fig.51 The display unit 5101 and the touch-sensitive surface unit 5103 are shown integrated with the electronic device 5100, however, in some embodiments, although these units remain physically separated from the electronic device, one or both of these units communicate with the electronic device. The processing unit includes an information acquisition unit (e.g., Fig.51 information acquisition unit 5107), an application exit unit (for example, Fig.51 application exit unit 5109), a request receiving unit (eg, Fig.51 Request receiving unit 5111), application capability determining unit (e.g., Fig.51 application capability determination unit 5113), an application presentation unit (eg, Fig.51 application presentation unit 5115), an application filling unit (e.g., Fig.51 application filling unit 5117), an input detection unit (e.g., Fig.51 input detection unit 5119), an application switching user interface display unit (eg, Fig.51 an application switching user interface display unit 5121), an application association determination unit (eg, Fig.51 5123) and an access providing unit (e.g., Fig.51 The processing unit (or one or more components thereof, such as units 5107 to 5125) is configured to: while displaying the first application, obtain information for identifying a first physical location viewed by a user in the first application (e.g., via information obtaining unit 5107). The processing unit is also configured to: exit the first application (e.g., via application exiting unit 5109), and after exiting the first application, receive a request from the user to open a second application different from the first application (e.g., via request receiving unit 5111). In response to receiving the request and based on a determination that the second application is capable of accepting geographic location information (e.g., a determination processed or performed by application capability determination unit 5113), present a second application (e.g., via application presenting unit 5115), and presenting the second application includes populating the second application with information based at least in part on the first physical location identification information (e.g., via application populating unit 5117).

[0320] (K20) In some embodiments of the electronic device of K19, the processing unit (or one or more components thereof, such as units 5107 to 5125) is further configured to execute the method described in any one of K2 to K13.

[0321] (L1) According to some embodiments, in a device having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, such as in Figure 1C The touch screen 112 is shown) of an electronic device (eg, Figure 1A The portable multifunctional device 100, according to Figure 1E The method is performed on a computing device A to D of any one configuration). The method includes: obtaining information for identifying a first physical location viewed by a user in a first application and detecting a first input. In response to detecting the first input, the method includes: (i) identifying a second application capable of accepting geographic location information, and (ii) presenting an enable representation different from the first application on at least a portion of the display, and suggesting opening the second application with information about the first physical location. The method also includes: detecting a second input at the enable representation. In response to detecting the second input at the enable representation: (i) opening the second application, and (ii) populating the second application to include information based at least in part on the first physical location identification information.

[0322] Compared to the operation associated with K1 described above, the operation associated with L1 does not receive a request from the user to open the second application before providing a suggestion to the user to open the second application having information related to the first physical location. In this way, by providing the operations associated with K1 and L1 described above (and using a combination of some processing steps from each of these methods), the electronic device can provide an efficient user experience, allowing the user to predictively use location data before or after opening an application that can accept geographic location information. In addition, for L1, determining whether the second application can accept geographic location information is performed before opening the second application, and in this way, in an implementation scheme of L1, where the input corresponds to a request to open an application switching user interface, the application switching user interface displays a request to open an application (e.g., the second application) having information related to the first physical location only when it is known that the application can accept location data.

[0323] (L2) In some embodiments of the method of L1, the first input corresponds to a request to open an application switching user interface (for example, the first input is a double-click on a physical Home button of the electronic device).

[0324] (L3) In some embodiments of the L2 method, an enable indication is presented within an application switching user interface.

[0325] (L4) In some embodiments of the L3 method, presenting the enabling representation includes: presenting a representation of an application being executed on the electronic device (e.g., a snapshot of application content of the application) within an application switching user interface in conjunction with presenting the enabling representation; and presenting the enabling representation in an area of ​​the display located below the application representation.

[0326] (L5) In some embodiments of the L1 method, the first input corresponds to a request to open the home screen of the electronic device (for example, the first input is a single tap on a physical Home button of the electronic device).

[0327] (L6) In some embodiments of the L5 method, an enable indication is presented on a portion of the home screen.

[0328] (L7) In some embodiments of the method of any of L1 to L6, the suggestion includes a text description that is specific to a type associated with the second application.

[0329] (L8) In some embodiments of the method of any one of L1 to L7, populating the second application includes displaying a user interface object that includes information based at least in part on the first physical location identification information.

[0330] (L9) In some embodiments of the method of L8, the user interface object includes a text description that notifies the user that the first physical location was most recently viewed in the first application.

[0331] (L10) In some embodiments of the method of L9, the user interface object is a map displayed in the second application, and populating the second application includes populating the map to include an identifier of the first physical location.

[0332] (L11) In some embodiments of the method of any one of L9 to L10, the second application presents a virtual keyboard, and the user interface object is displayed above the virtual keyboard.

[0333] (L12) In some embodiments of the method of any one of L1 to L11, identifying that the second application is capable of accepting geographic location information includes one or more of the following: (i) determining that the second application includes an input receiving field capable of accepting and processing geographic location data; (ii) determining that the second application is capable of displaying geographic location information on a map; (iii) determining that the second application is capable of using geographic location information to facilitate route guidance; and (iv) determining that the second application is capable of using geographic location information to locate and provide transportation services.

[0334] (L13) In some embodiments of the L12 method, identifying that the second application is capable of accepting geographic location information includes: determining that the second application includes an input receiving field capable of accepting and processing geographic location data, and the input receiving field is a search box that allows searching within a map displayed in the second application.

[0335] (L14) In another aspect, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive surface, a display, one or more processors, and a memory storing one or more programs, wherein when the one or more processors execute the program, the electronic device will perform any method described in L1 to L13.

[0336] (L15) On the other hand, an electronic device is provided, comprising: a touch-sensitive surface, a display, and a device for executing any one of the methods described in L1 to L13.

[0337] (L16) In another aspect, a non-transitory computer-readable storage medium is provided, which is used to store executable instructions, which, when executed by an electronic device having a touch-sensitive surface and a display, will cause the electronic device to perform any method described in L1 to L13.

[0338] (L17) On the other hand, a graphical user interface on an electronic device having a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of L1 to L13.

[0339] (L18) In another aspect, an information processing device for use in an electronic device including a touch-sensitive surface and a display is provided. The information processing device includes: a device for performing the method described in any one of L1 to L13.

[0340] (L19) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.52 display unit 5201), a touch-sensitive surface unit (e.g., Fig.52 touch-sensitive surface unit 5203) and a processing unit (e.g., Fig.52 In some embodiments, according to Figure 1E Any of the computing devices shown (e.g., computing devices A to D) configures an electronic device. For ease of illustration, Fig.52The display unit 5201 and the touch-sensitive surface unit 5203 are shown integrated with the electronic device 5200, however, in some embodiments, although these units remain physically separated from the electronic device, one or both of these units communicate with the electronic device. The processing unit includes an information acquisition unit (e.g., Fig.52 information acquisition unit 5207), an input detection unit (for example, Fig.52 input detection unit 5209), application recognition unit (e.g., Fig.52 application recognition unit 5211), an affordance presentation unit (e.g., Fig.52 The affordance presentation unit 5213), the application opening unit (for example, Fig.52 application opening unit 5215), an application filling unit (e.g., Fig.52 application filling unit 5217), an application switching user interface presentation unit (eg, Fig.52 an application switching user interface presenting unit 5219), an application capability determining unit (eg, Fig.52 The processing unit (or one or more components thereof, such as units 5207 to 5221) is configured to: obtain information for identifying a first physical location viewed by a user in a first application (e.g., via information acquisition unit 5207) and detect a first input (e.g., via input detection unit 5209). In response to detecting the first input, the processing unit is configured to: (i) identify a second application capable of accepting geographic location information (e.g., via application identification unit 5209), and (ii) present an enable representation different from the first application on at least a portion of the display and suggest opening the second application having information about the first physical location (e.g., via enable representation presentation unit 5213). The processing unit is also configured to: detect a second input at the enable representation (e.g., via input detection unit 5209). In response to detecting a second input at the enabling representation, the processing unit is configured to: (i) open a second application (e.g., via application opening unit 5215), and (ii) populate the second application to include information based at least in part on the first physical location identification information (e.g., via application populating unit 5217).

[0341] (L20) In some embodiments of the electronic device of L19, the processing unit (or one or more components thereof, such as units 5207 to 5221) is further configured to execute the method described in any one of L2 to L13.

[0342] (M1) According to some embodiments, in a device having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, such as in Figure 1C The touch screen 112 is shown) of an electronic device (eg, Figure 1A The portable multifunctional device 100, according to Figure 1E The method is executed on a computing device A to D of any one configuration). The method includes: obtaining information for identifying a first physical location, wherein the first physical location is a location viewed by a user in a first application being executed on the electronic device. The method also includes: determining that the user has entered a vehicle. In response to determining that the user has entered the vehicle, providing a prompt to the user, prompting the user to use the first physical location as a destination for route guidance. In response to providing the prompt, the method includes: receiving an instruction from the user, instructing the user to use the first physical location as a destination for route guidance. The method also includes: facilitating route guidance to the first physical location. In this way, it is convenient to provide suggestions to users, suggesting that users plan routes to destinations based on physical locations that they have previously viewed in applications on electronic devices.

[0343] (M2) In some embodiments of the method of M1, the method further includes: detecting that the electronic device has received a message, including detecting that the message includes information for identifying a second physical location; and in response to the detection, providing a new prompt to the user, prompting the user to use the second physical location as a new destination for route guidance. In this way, the user can also dynamically add waypoints or add new destinations for route guidance based on information included in a message (e.g., a text message, an email, a voicemail, etc.).

[0344] (M3) In some embodiments of the method of M2, the method further includes: in response to receiving an instruction from a user to use the second physical location as a new destination, facilitating route guidance to the second physical location.

[0345] (M4) In some embodiments of the method of any one of M2 to M3, detecting that the message contains information identifying the second physical location includes: performing detection when a virtual assistant available on the electronic device is reading the message to the user through an audio system that communicates with the electronic device. In this way, when the user is listening to the message read out by the audio system (for example, through a personal assistant available via the electronic device), the electronic device detects the information used to identify the second physical location and uses the detected information to suggest using the second physical location as a new destination. Therefore, the user does not have to divert his or her attention from the road while driving, and is still able to dynamically adjust the route guidance settings and destination.

[0346] (M5) In some embodiments of the method of any one of M2 to M4, determining that the user has entered the vehicle includes detecting that the electronic device has established a communication link with the vehicle.

[0347] (M6) In some embodiments of the method of any one of M2 to M5, facilitating route guidance includes providing route guidance via a display of an electronic device.

[0348] (M7) In some embodiments of the method of any one of M2 to M5, facilitating route guidance includes sending information identifying the first physical location to the vehicle.

[0349] (M8) In some embodiments of the method of any one of M2 to M7, facilitating route guidance includes providing route guidance via an audio system that communicates with the electronic device (e.g., a car speaker or an internal speaker of the device itself).

[0350] (M9) In another aspect, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive surface, a display, one or more processors, and a memory storing one or more programs, wherein when the one or more processors execute the program, the electronic device will perform the method described in any one of M1 to M8.

[0351] (M10) On the other hand, an electronic device is provided, comprising: a touch-sensitive surface, a display, and a device for performing any of the methods described in M1 to M8.

[0352] (M11) In another aspect, a non-transitory computer-readable storage medium is provided, which is used to store executable instructions, which, when executed by an electronic device having a touch-sensitive surface and a display, will cause the electronic device to perform any method described in M1 to M8.

[0353] (M12) On the other hand, a graphical user interface on an electronic device having a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of M1 to M8.

[0354] (M13) In another aspect, an information processing device for use in an electronic device including a touch-sensitive surface and a display is provided. The information processing device includes: a device for performing the method described in any one of M1 to M8.

[0355] (M14) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.53 display unit 5301), a touch-sensitive surface unit (e.g., Fig.53 touch-sensitive surface unit 5303) and a processing unit (e.g., Fig.53 In some embodiments, according to Figure 1E Any of the computing devices shown (e.g., computing devices A to D) configures an electronic device. For ease of illustration, Fig.53 A display unit 5301 and a touch-sensitive surface unit 5303 are shown integrated with an electronic device 5300, however, in some embodiments, one or both of these units communicate with the electronic device even though these units remain physically separated from the electronic device. The processing unit includes an information acquisition unit (e.g., Fig.53 information acquisition unit 5307), a vehicle entry determination unit (eg, Fig.53 The vehicle enters the determining unit 5309, the prompt providing unit (for example, Fig.53 A prompt providing unit 5311), an instruction receiving unit (for example, Fig.53 The instruction receiving unit 5313), the route guidance promotion unit (for example, Fig.53 route guidance facilitation unit 5315) and a message detection unit (e.g., Fig.53 message detection unit 5317). The processing unit (or one or more components thereof, such as units 5307 to 5317) is configured to: obtain information for identifying a first physical location viewed by a user in the first application being executed on the electronic device (e.g., through the information acquisition unit 5307). The processing unit is also configured to: determine that the user has entered a vehicle (e.g., through the vehicle entry determination unit 5309). In response to determining that the user has entered the vehicle, the processing unit is configured to: provide a prompt to the user, prompting the user to use the first physical location as a destination for route guidance (e.g., through the prompt providing unit 5311). In response to providing the prompt, receiving an instruction from the user, instructing the user to use the first physical location as a destination for route guidance (e.g., through the instruction receiving unit 5313). The processing unit is further configured to: facilitate route guidance to the first physical location (e.g., through the route guidance facilitation unit 5307).

[0356] (M15) In some embodiments of the electronic device of M14, the processing unit (or one or more components thereof, such as units 5307 to 5317) is further configured to execute the method described in any one of M2 to M8.

[0357] (N1) According to some embodiments, in a device having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, such as in Figure 1C The touch screen 112 is shown) of an electronic device (eg, Figure 1A The portable multifunctional device 100, according to Figure 1EA method is performed on a computing device A to D of any one configuration). The method includes: presenting content in a first application. The method also includes: receiving a request from a user to open a second application different from the first application, the second application including an input receiving field. In response to receiving the request, the method includes: presenting a second application having an input receiving field. Before receiving any user input at the input receiving field, the method includes: providing an optional user interface object to allow a user to paste at least a portion of the content into the input receiving field. In response to detecting selection of the optional user interface object, the method includes: pasting the portion of the content into the input receiving field. In this way, an active paste action is provided to the user in the second application-based content previously viewed in the first action (for example, this enables the user to paste the content into the second application without having to reopen the first application, perform an explicit copy operation, reopen the second application, and then explicitly request that the copied content be pasted into the second application).

[0358] (N2) According to some embodiments of the method of N1, before providing the selectable user interface object, the method includes: identifying the input receiving field as a field capable of accepting the portion of content.

[0359] (N3) According to some embodiments of the method of N2, in response to detecting selection of the input receiving field, the input receiving field is identified as a field capable of accepting the portion of the content.

[0360] (N4) In some embodiments of the method of any one of N1 to N3, the portion of the content corresponds to an image.

[0361] (N5) In some embodiments of the method of any one of N1 to N3, the portion of content corresponds to text content.

[0362] (N6) In some embodiments of the method of any one of N1 to N3, the portion of content corresponds to text content and an image.

[0363] (N7) In some embodiments of the method of any one of N1 to N6, the first application is a web browsing application and the second application is an instant messaging application.

[0364] (N8) In some embodiments of the method of any one of N1 to N6, the first application is a photo browsing application and the second application is an instant messaging application.

[0365] (N9) In some embodiments of the method of any one of N1 to N8, the method includes receiving a request to copy at least the portion of the content before receiving a request to open a second application.

[0366] (N10) In some embodiments of the method of any one of N1 to N9, the selectable user interface object is displayed together with an indication of the most recently viewed content portion in the first application, so that a clear indication is provided to the user as to why the paste suggestion was made.

[0367] (N11) In another aspect, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive surface, a display, one or more processors, and a memory storing one or more programs, wherein when the one or more processors execute the programs, the electronic device will perform the method described in any one of N1 to N10.

[0368] (N12) On the other hand, an electronic device is provided, comprising: a touch-sensitive surface, a display, and a device for performing any of the methods described in N1 to N10.

[0369] (N13) In another aspect, a non-transitory computer-readable storage medium is provided, which is used to store executable instructions, which, when executed by an electronic device having a touch-sensitive surface and a display, will cause the electronic device to perform the method described in any one of N1 to N10.

[0370] (N14) On the other hand, a graphical user interface on an electronic device having a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of N1 to N10.

[0371] (N15) In another aspect, an information processing device for use in an electronic device including a touch-sensitive surface and a display is provided. The information processing device includes: a device for performing the method described in any one of N1 to N10.

[0372] (N16) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.54 display unit 5401), a touch-sensitive surface unit (e.g., Fig.54 touch-sensitive surface unit 5403) and a processing unit (e.g., Fig.54 In some embodiments, according to Figure 1E Any of the computing devices shown (e.g., computing devices A to D) configures an electronic device. For ease of illustration, Fig.54 A display unit 5401 and a touch-sensitive surface unit 5403 are shown integrated with an electronic device 5400, however, in some embodiments, one or both of these units communicate with the electronic device even though these units remain physically separated from the electronic device. The processing unit includes a presentation unit (e.g., Fig.54, a presentation unit 5407, a request receiving unit (eg, Fig.54 Request receiving unit 5409), user interface object providing unit (eg, Fig.54 The user interface object providing unit 5411), the active pasting unit (for example, Fig.54 The active pasting unit 5413) and the capability determination unit (e.g., Fig.54 The processing unit (or one or more components thereof, such as units 5407 to 5415) is configured to: present content in a first application (e.g., via presentation unit 5407 and / or display unit 5401); receive a request from a user to open a second application different from the first application (e.g., via request receiving unit and / or touch-sensitive surface unit 5403), the second application including an input receiving field; in response to receiving the request, present the second application having the input receiving field (e.g., via presentation unit 5407 and / or display unit 5401); before any user input is received at the input receiving field, provide an optional user interface object to allow the user to paste at least a portion of the content into the input receiving field (e.g., via user interface object providing unit 5411 and / or display unit 5401); and in response to detecting selection of the optional user interface object, paste the portion of the content into the input receiving field (e.g., via active pasting unit 5413).

[0373] (N17) In some embodiments of the electronic device of N16, the processing unit (or one or more components thereof, such as units 5407 to 5415) is further configured to perform a method described in any one of N1 to N10.

[0374] (O1) According to some embodiments, in a device having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, such as in Figure 1C The touch screen 112 is shown) of an electronic device (eg, Figure 1A The portable multifunctional device 100, according to Figure 1EThe method is performed on a computing device A to D of any one of the configurations of the present invention. The method includes: presenting text content associated with an application on a display. The method also includes: determining that a portion of the text content is related to the following items: (i) a location, (ii) a contact, or (iii) an event. When it is determined that the portion of the text content is related to a certain location, the method includes: acquiring location information from a location sensor on the electronic device, and preparing the acquired location information to be displayed as a predicted content item. When it is determined that the portion of the text content is related to a contact, the method includes: performing a search on the electronic device, searching for contact information related to the portion of the text content, and preparing information associated with at least one contact retrieved by the search to be displayed as a predicted content item. When it is determined that the portion of the text content is related to an event, the method includes: performing a new search on the electronic device, searching for event information related to the portion of the text content, and preparing information based at least in part on at least one event (retrieved by the new search) to be displayed as a predicted content item. The method also includes: displaying an affordance within the application that includes the predicted content item; detecting a selection of the affordance via the touch-sensitive surface; and in response to detecting the selection, displaying information associated with the predicted content item on the display adjacent to the text content. In this way, the user is conveniently provided with predicted content items that can be used to complete a statement (or respond to questions posed by other users, such as in an instant messaging application) without having to type anything and without having to browse through information available on the electronic device to find the desired information. For example, the electronic device provides a phone number, a current location, scheduling functionality for new events, details associated with existing events, all without requiring any explicit request or additional work by the user, thereby saving time while still ensuring that the desired information is efficiently provided to the user.

[0375] (O2) According to some embodiments of the method of O1, the portion of the text content corresponds to text content most recently presented in the application.

[0376] (O3) In some embodiments of the method of any one of O1 to O2, the application is an instant messaging application, and the portion of the text content is a question received in the instant messaging application from a remote user of a remote device that is different from the electronic device.

[0377] (O4) According to some embodiments of the method of any one of O1 to O2, the portion of the text content is input provided by a user of the electronic device in an input receiving field within an application.

[0378] (O5) According to some embodiments of the method of O1, in response to user input selecting a user interface object that includes the portion of text content, the portion of text content is identified.

[0379] (O6) According to some embodiments of the method of O5, the application is an instant messaging application, and the user interface object is a message bubble in a conversation displayed within the instant messaging application.

[0380] (O7) In some embodiments of the method of any of O5 to O6, the method further comprises: detecting a selection of a second user interface object; in response to detecting the selection, (i) stopping displaying the affordance with the predicted content item and determining that the text content associated with the second user interface object relates to a location, a contact, or an event; and based on the determination, displaying a new predicted content item within the application. In this way, the user can easily return to the instant messaging conversation to select a previously received message and still be provided with the appropriate predicted content item.

[0381] (O8) In some embodiments of the method of any one of O1 to O7, the indication is displayed in an input receiving field within the application and adjacent to the virtual keyboard.

[0382] (O9) In some embodiments of the method of any one of O8, the input receiving field is a field that displays typed input received on a virtual keyboard.

[0383] (O10) In some embodiments of the method of any one of O1 to O9, the determining includes parsing the text content as it is received by the application to detect stored patterns known to be associated with contacts, events, and / or locations.

[0384] (O11) In another aspect, an electronic device is provided. In some embodiments, the electronic device includes: a touch-sensitive surface, a display, one or more processors, and a memory storing one or more programs, wherein when the one or more processors execute the program, the electronic device will perform any method described in O1 to O10.

[0385] (O12) On the other hand, an electronic device is provided, comprising: a touch-sensitive surface, a display, and a device for executing any one of the methods described in O1 to O10.

[0386] (O13) In another aspect, a non-transitory computer-readable storage medium is provided, which is used to store executable instructions, which, when executed by an electronic device having a touch-sensitive surface and a display, will cause the electronic device to perform any of the methods described in O1 to O10.

[0387] (O14) On the other hand, a graphical user interface on an electronic device having a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface includes a user interface displayed according to the method described in any one of O1 to O10.

[0388] (O15) In another aspect, an information processing device for use in an electronic device including a touch-sensitive surface and a display is provided. The information processing device includes: a device for performing the method described in any one of O1 to O10.

[0389] (O16) In another aspect, an electronic device is provided, the device including a display unit (e.g., Fig.55 display unit 5501), a touch-sensitive surface unit (e.g., Fig.55 touch-sensitive surface unit 5503) and a processing unit (e.g., Fig.55 In some embodiments, according to Figure 1E Any of the computing devices shown (e.g., computing devices A to D) configures an electronic device. For ease of illustration, Fig.55 A display unit 5501 and a touch-sensitive surface unit 5503 are shown integrated with the electronic device 5500, however, in some embodiments, one or both of these units communicate with the electronic device even though these units remain physically separated from the electronic device. The processing unit includes a presentation unit (e.g., Fig.55 5507), a determination unit (e.g., Fig.55 A determination unit 5509), an acquisition unit (for example, Fig.55 The acquisition unit 5511), the search execution unit (for example, Fig.55 The search execution unit 5513), the information preparation unit (for example, Fig.55 information preparation unit 5515), an indication display unit (for example, Fig.55 The indication means display unit 5517) and the detection unit (for example, Fig.55The processing unit (or one or more components thereof, such as units 5507 to 5519) is configured to: present text content associated with the application on a display (e.g., via the presenting unit 5507 and / or the display unit 5501); determine that a portion of the text content is related to: (i) a location, (ii) a contact, or (iii) an event (e.g., via the determining unit 5509); upon determining that the portion of the text content is related to the location, obtain location information from a location sensor on the electronic device (e.g., via the obtaining unit 5511), and prepare the obtained location information to be displayed as a predicted content item (e.g., via the information preparing unit 5515); upon determining that the portion of the text content is related to the contact, perform a search on the electronic device for contact information related to the portion of the text content (e.g., via the search executing unit 5513), and prepare at least one of the items retrieved by the search. The electronic device may further include: preparing information associated with the contact to be displayed as a predicted content item (e.g., by information preparation unit 5515); upon determining that the portion of the text content is related to an event, performing a new search on the electronic device to search for event information related to the portion of the text content (e.g., by search execution unit 5513), and preparing information based at least in part on at least one event retrieved by the search to be displayed as a predicted content item (e.g., by information preparation unit 5515); displaying an enable representation including the predicted content item in an application (e.g., by enable representation display unit 5517 and / or display unit 5501); detecting selection of the enable representation through the touch-sensitive surface (e.g., by detection unit 5519); and in response to detecting the selection, displaying information associated with the predicted content item adjacent to the text content on the display (e.g., by presentation unit 5507 and / or display unit 5501).

[0390] (O17) In some embodiments of the electronic device of O16, the processing unit (or one or more components thereof, such as units 5507 to 5519) is further configured to execute any of the methods described in O1 to O10.

[0391] As described above (and described in detail below), one aspect of the present technology is to collect and use data obtained from various sources (e.g., based on the voice provided during voice communications) to improve the delivery of content that may be of interest to users. The present disclosure contemplates that, in some instances, such collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, home addresses, or any other identifying information.

[0392] The present disclosure recognizes that the use of such personal information data in the present invention technology can be used to benefit users. For example, the personal information data can be used to deliver targeted content that users are more interested in. Therefore, the use of such personal information data enables planned control of the delivered content. In addition, the present disclosure also anticipates other uses of personal information data that benefit users.

[0393] The disclosure also anticipates that entities responsible for the collection, analysis, disclosure, transmission, storage or other uses of such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. For example, personal information from users should be collected for the legal and reasonable purposes of the entity and not shared or sold outside these legal uses. In addition, such collection should only be carried out after the user's informed consent. In addition, such entities should take any necessary steps to safeguard and protect access to such personal information data, and ensure that others who have access to personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices.

[0394] Regardless of the foregoing, the present disclosure also contemplates implementation scenarios in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, in terms of monitoring voice communications or monitoring actions performed by users in applications, the technology of the present invention may be configured to allow users to choose to "opt in" or "opt out" of participating in the collection of personal information data during registration for a service. For another example, a user may choose not to provide location information for a target content delivery service. For another example, a user may choose not to provide precise location information but permit the transmission of location area information.

[0395] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences based on non-personal information data or an absolute minimum of personal information, such as content requested by a device associated with the user, other non-personal information available to a content delivery service, or publicly available information.

[0396] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in this specification are not comprehensive, and in particular, many additional features and advantages will be apparent to those of ordinary skill in the art based on the drawings, the specification, and the claims. In addition, it should be noted that the language used in this specification is selected in principle for readability and instructional purposes, and may not be selected to describe or define the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0397] For a better understanding of the various described embodiments, reference should be made to the following detailed description section in conjunction with the following drawings, wherein like reference numerals refer to corresponding parts throughout the drawings.

[0398] Figure 1A is a high-level block diagram of a computing device with a touch-sensitive display according to some embodiments.

[0399] Figure 1B is a block diagram of exemplary components for event processing according to some embodiments.

[0400] Figure 1C is a schematic diagram of a portable multifunction device with a touch-sensitive display according to some embodiments.

[0401] Figure 1D is a schematic diagram used to illustrate a computing device according to some embodiments, the computing device having a touch-sensitive surface separate from the display.

[0402] Figure 1E An example electronic device that communicates with a display and a touch-sensitive surface according to some embodiments is shown.

[0403] Figure 2 is a schematic diagram of a touch screen according to some embodiments, which is used to show a user interface of a menu of applications.

[0404] FIG. 3A to FIG. 3B is a block diagram illustrating a data structure for storing application usage data according to some embodiments.

[0405] FIG. 4A to FIG. 4B is a block diagram illustrating a data structure for storing trigger conditions according to some embodiments.

[0406] Figure 5 is a block diagram illustrating an example of a trigger condition establishment system according to some embodiments.

[0407] FIG. 6A to FIG. 6B is a flowchart representation of a method for actively identifying and displaying (e.g., displaying for user selection) relevant content on an electronic device with a touch-sensitive display according to some embodiments.

[0408] FIG. 7A to FIG. 7B is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface designed to actively identify and display relevant content.

[0409] FIG. 8A to FIG. 8B is a flowchart representation of a method for actively identifying and displaying (e.g., displaying for user selection) relevant content on an electronic device with a touch-sensitive display according to some embodiments.

[0410] 9A to 9D is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface designed to actively identify and display relevant content.

[0411] FIG. 10A to FIG. 10C is a flowchart representation of a method for proactively suggesting search queries based on content currently displayed on an electronic device according to some embodiments.

[0412] FIG. 11A to FIG. 11J is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface that proactively suggests search queries based on content currently displayed on the touch-sensitive display.

[0413] Fig.12 is a flowchart representation of a method for entering a search mode based on a heuristic process according to some embodiments.

[0414] FIG. 13A to FIG. 13B is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface for entering a search mode based on a heuristic process.

[0415] Fig.14 is a flowchart representation of a method for proactively providing a vehicle location on an electronic device with a touch-sensitive display according to some embodiments.

[0416] FIG. 15A to FIG. 15B is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface designed to proactively provide a vehicle's location.

[0417] FIG. 16A to FIG. 16B is a flowchart representation of a method for proactively providing nearby point of interest (POI) information for a search query according to some embodiments.

[0418] FIG. 17A to FIG. 17E is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface designed to proactively provide nearby point of interest (POI) information for search queries.

[0419] FIG. 18A to FIG. 18Bis a flowchart representation of a method of extracting content items from a voice communication and interacting with the extracted content items according to some embodiments.

[0420] FIG. 19A to FIG. 19F is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface designed to display and interact with content items extracted from voice communications.

[0421] Fig. 20 is a flowchart representation of a method for determining voice communications including recognizing voice at a physical location and populating an application with information about the physical location, according to some embodiments.

[0422] FIG. 21A to FIG. 21B is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface for determining that voice communications include recognizing a physical location of the voice and populating an application with information about the physical location.

[0423] FIG. 22A to FIG. 22B is a flowchart representation of a method for proactively suggesting physical locations for use in an instant messaging application according to some embodiments.

[0424] Fig. 22C is a flowchart representation of a method for proactively suggesting information related to a location, event, or contact, according to some embodiments.

[0425] FIG. 23A to FIG. 23O is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface intended to proactively suggest information related to locations, events, or contacts (e.g., for user selection and inclusion in an instant messaging application).

[0426] FIG. 24A to FIG. 24B is a flowchart representation of a method for proactively populating an application with information that a user previously viewed in another application, according to some embodiments.

[0427] FIG. 25A to FIG. 25J is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface designed to proactively populate an application with information that a user previously viewed in another application (e.g., populating a ride-hailing application with information about a location that a user viewed in a check application).

[0428] FIG. 26A to FIG. 26B is a flowchart representation of a method of proactively suggesting that suggestion information previously viewed by a user in a first application be used in a second application, according to some embodiments.

[0429] Fig. 27 is a flowchart representation of proactively suggesting physical locations for use as destinations for route guidance in a vehicle, according to some embodiments.

[0430] Fig.28 is a schematic diagram of a touch-sensitive display according to some embodiments, illustrating a user interface intended to proactively suggest physical locations for use as destinations for route guidance in a vehicle.

[0431] Fig.29 is a flowchart representation of a method for proactively suggesting a paste action according to some embodiments.

[0432] FIG. 30A to FIG. 30D is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface designed to proactively suggest a paste action.

[0433] Figure 31_1 A mobile device is shown that is configured to perform dynamic adjustments of the mobile device according to some embodiments.

[0434] Figure 31_2 An exemplary process for invoking a heuristic process according to some embodiments is shown.

[0435] Figure 31_3 A process for adjusting mobile device settings using a heuristic process according to some embodiments is shown.

[0436] Figure 31_4 An exemplary system for performing background fetching of application updates according to some embodiments is shown.

[0437] Figure 31_5 A peer forecast for determining a user invocation probability of an application on a mobile device 100 is shown according to some embodiments.

[0438] Figure 31_6 is a flowchart of an exemplary process for predictively launching an application to perform background updates according to some embodiments.

[0439] Figure 31_7 is a flowchart of an exemplary process for determining when to launch an application on a mobile device according to some embodiments.

[0440] Figure 31_8 is a flow diagram illustrating state transitions of entries in a trend table according to some embodiments.

[0441] Figure 31_9 is a block diagram illustrating a system for providing push notifications to a mobile device according to some embodiments.

[0442] Figure 31_10 is a flowchart of an exemplary process for performing a non-wake-up push at a push notification server according to some embodiments.

[0443] Figure 31_11is a flowchart of an exemplary process for performing background updates of an application in response to a low priority push notification according to some embodiments.

[0444] Figure 31_12 is a flowchart of an exemplary process for performing background updates of an application in response to a high priority push notification according to some embodiments.

[0445] Figure 31_13 is a block diagram of an exemplary system for performing background downloads and / or uploads of data on a mobile device according to some embodiments.

[0446] Figure 31_14 is a flowchart of an exemplary process for performing background downloads and uploads according to some embodiments.

[0447] Figure 31_15 An exemplary graphical user interface (GUI) is shown for enabling and / or disabling background updates of applications on a mobile device according to some embodiments.

[0448] Figure 31_16 An exemplary system for sharing data between peer devices according to some embodiments is shown.

[0449] Figure 31_17 An exemplary process for sharing data between peer devices according to some embodiments is shown.

[0450] Figure 32_1 is a block diagram of one embodiment of a system for returning search results based on an input query prefix, according to some embodiments.

[0451] Figure 32_2 is a flow diagram of one embodiment of a process for determining query completions and related results based on an input query prefix, according to some embodiments.

[0452] Figure 32_3 is a block diagram of one embodiment of an aggregator and multiple search domains according to some embodiments.

[0453] Figure 32_4 is an illustration of one embodiment of a query completion term search domain according to some embodiments.

[0454] Figure 32_5 is a diagram of one embodiment of a map search domain. Figure 32_6 is a flow diagram of one embodiment of a process for determining query completion terms from multiple search domains according to some embodiments.

[0455] Figure 32_7is a flow diagram of one embodiment of a process for determining relevant results from multiple search domains for determining query completion terms according to some embodiments.

[0456] Figure 32_8 is a block diagram of one embodiment of a system for incorporating user feedback into a feedback search index according to some embodiments.

[0457] Figure 32_9 is a flow diagram of one embodiment of a process for incorporating user feedback into a citation search index according to some embodiments.

[0458] Figure 32_10 is a flow diagram of one embodiment of a process for collecting user feedback during a user search session according to some embodiments.

[0459] Figure 32_11 is a flow diagram of one embodiment of a process for incorporating user feedback into a feedback index according to some embodiments.

[0460] Figure 32_12 is a flow diagram of one embodiment of a process for updating a result cache using user feedback according to some embodiments.

[0461] Figure 32_13 is a block diagram of one embodiment of a federator that performs multi-domain searches using representative query completion terms according to some embodiments.

[0462] Figure 32_14 is a flow diagram of one embodiment of a process for determining relevant results using a vocabulary service according to some embodiments.

[0463] Figure 32_15 is a flow diagram of one embodiment of a process for characterizing query completion items according to some embodiments.

[0464] Figure 32_16 is a block diagram of one embodiment of a completions module for determining query completions from multiple search domains according to some embodiments.

[0465] Figure 32_17 is a block diagram of one embodiment of a results module for determining relevant results across multiple search domains from determining query completion terms, according to some embodiments.

[0466] Figure 32_18 is a block diagram of one embodiment of a collect feedback module for collecting user feedback during a user search session according to some embodiments.

[0467] Figure 32_19is a block diagram of one embodiment of a process feedback module for incorporating user feedback into a feedback index according to some embodiments.

[0468] Figure 32_20 is a block diagram of one embodiment of an update query results module for updating a results cache using user feedback, according to some embodiments.

[0469] Figure 32_21 is a block diagram of one embodiment of a process feedback module for incorporating user feedback into a feedback index according to some embodiments.

[0470] Figure 32_22 is a block diagram of one embodiment of an update query results module for updating a results cache using user feedback, according to some embodiments.

[0471] Figure 33_1 A local search subsystem and a remote search subsystem on a computing device as is known in the art are shown in block diagram form.

[0472] Figure 33_2 A local search subsystem having local learning capabilities that can be used to improve results returned from a remote search application on a computing device is shown in block diagram form according to some embodiments.

[0473] Figure 33_3 A method for locally learning query features using local search queries, local results, and local feedback based on the local results is shown in block diagram form according to some embodiments.

[0474] Figure 33_4 A method for locally learning query features using search results returned from both local and remote search queries and local feedback of both local and remote search query results is shown in block diagram form according to some embodiments.

[0475] Figure 33_5 A method of learning query features that are communicated from a remote service to a local service in response to a query sent to the remote service is illustrated in block diagram form according to some embodiments.

[0476] Figure 33_6 Methods of receiving or determining new features, training features locally, and utilizing features according to some embodiments are shown in block diagram form.

[0477] Figure 33_7 An exemplary embodiment of a software stack that may be used in some embodiments of the present invention according to some embodiments is shown.

[0478] Figure 34_1AA block diagram of an exemplary data architecture for suggesting contacts according to some embodiments is shown.

[0479] Figure 34_1B A block diagram of an exemplary data architecture for suggesting calendar events according to some embodiments is shown.

[0480] Figure 34_2A to Figure 34_2G An exemplary user interface for providing suggested contacts and calendar events is shown according to some embodiments. Figure 1A to Figure 1B , Figure 2 and FIG. 3 depict example devices for performing the techniques described in this section for suggesting contact and event information. Figure 34_2A to Figure 34_2G Exemplary user interfaces for suggesting contact and event information are shown, and the user interfaces in these figures are also used to illustrate the processes described below, including Figure 34_3A to Figure 34_9 process.

[0481] Figure 34_3A and Figure 34_3B A flowchart of an exemplary process for generating suggested contacts according to some embodiments is shown.

[0482] Figure 34_4A and Figure 34_4B A flowchart is shown of an exemplary process for updating an existing contact using suggested contact information items according to some embodiments.

[0483] Figure 34_5A and Figure 34_5B A flowchart is shown of an exemplary process for displaying contacts with suggested contact information according to some embodiments.

[0484] Figure 34_6 A flowchart is shown of an exemplary process for displaying suggested contact information with a message according to some embodiments.

[0485] Figure 34_7A and Figure 34_7B A flowchart of an exemplary process for generating suggested calendar events according to some embodiments is shown.

[0486] Figure 34_8 A flowchart is shown of an exemplary process for displaying suggested event information along with a message according to some embodiments.

[0487] Figure 34_9 A flowchart is shown of an exemplary process for displaying multiple suggested contact information or event information along with a message according to some embodiments.

[0488] Figure 34_10 is a functional block diagram of an electronic device according to some embodiments.

[0489] Figure 34_11 is a functional block diagram of an electronic device according to some embodiments.

[0490] Figure 35_1 is a flowchart of method 35_100 for suggesting applications based on detected events according to some embodiments.

[0491] Figure 35_2 The segmentation process 35_200 according to some embodiments is shown.

[0492] Figure 35_3 A decision tree 35_300 is shown that can be generated according to some embodiments.

[0493] Figure 35_4 is a flowchart of method 35_400 for recommending applications to a user of a computing device based on an event according to some embodiments.

[0494] Figure 35_5A to Figure 35_5D A graph showing an exemplary binomial distribution of various correct and incorrect numbers according to some embodiments.

[0495] Figure 35_6A and Figure 35_6B Parent models and child models resulting from segmentation are shown according to some embodiments.

[0496] Figure 35_7 An exemplary architecture 35_700 is shown for providing a user interface for interacting with one or more applications according to some embodiments.

[0497] Figure 36_1 is a flowchart of a method for identifying an application based on a triggering event according to some embodiments.

[0498] Figure 36_2 A block diagram of a system for determining a triggering event according to some embodiments is shown.

[0499] Figure 36_3 A block diagram of a system for identifying applications for a user based on a triggering event according to some embodiments is shown.

[0500] Figure 36_4 A block diagram of a system for identifying an application with multiple predictive models is shown according to some embodiments.

[0501] Figure 36_5 is a flowchart of a method for identifying an application by a device based on a triggering event according to some embodiments.

[0502] Figure 36_6 is a simplified diagram of a device with a music application user interface according to some embodiments.

[0503] Figure 36_7A and Figure 36_7B is a flowchart of a method for removing an identified application from a user interface according to some embodiments.

[0504] Figure 37_1 is a flow diagram of a method 100 for suggesting recipients to contact based on detected events, according to some embodiments.

[0505] Figure 37_2 A block diagram of a system for determining a triggering event according to some embodiments is shown.

[0506] Figure 37_3 A block diagram of a system for identifying a recipient to contact based on a triggering event according to some embodiments is shown.

[0507] Figure 37_4 An example of suggesting recipients to contact in a user interface of an email application according to some embodiments is shown.

[0508] Figure 37_5 Examples of suggesting recipients to contact in a user interface of a search application are shown according to some embodiments.

[0509] Figure 37_6 is a flowchart of method 37_600 according to some embodiments, which is used to suggest recipients to a user of a computing device based on an event.

[0510] Figure 37_7 An exemplary data flow for suggesting recipients to contact is shown in accordance with some embodiments.

[0511] Figure 37_8 A block diagram of an interaction module according to some embodiments is shown.

[0512] Figure 37_9 An exemplary architecture 37_900 is shown according to some embodiments for providing a user interface to suggest recipients to contact.

[0513] Figure 38_1 A block diagram is shown of different components of a mobile computing device configured to implement the various techniques described herein, according to some embodiments.

[0514] Figure 38_2 According to some embodiments, Figure 38_1 The application predicts the method performed by the engine.

[0515] Figure 38_3 A method performed by the search application of Figure 1 according to some embodiments is shown.

[0516] Figure 38_4 According to some embodiments, Figure 38_1 A conceptual diagram of an exemplary user interface for a search application.

[0517] Figure 39_1 A block diagram is shown of different components of a mobile computing device configured to implement the various techniques described herein, according to some embodiments.

[0518] Figure 39_2 It is shown that according to some embodiments, Figure 39_1 (or Figure 1A ) is a block diagram of a more detailed view of specific components of the mobile computing device shown in .

[0519] Figure 39_3A A method for high-level initialization and operation of a prediction engine according to some embodiments is shown.

[0520] Figure 39_3B Methods for synchronously providing predictions at a prediction engine according to some embodiments are shown.

[0521] Figure 39_3C Methods for asynchronously providing predictions at a prediction engine according to some embodiments are shown.

[0522] Figure 39_4A A method is shown in which a consumer application requests synchronous receipt of a prediction according to some embodiments.

[0523] Figure 39_4B A method is shown for a consumer application to register to asynchronously receive predictions according to some embodiments.

[0524] Figure 39_5A A method for managing prediction engine registrations at a prediction engine center is shown according to some embodiments.

[0525] Figure 39_5B Methods for synchronously providing predictions at a prediction engine hub to consumer applications are shown according to some embodiments.

[0526] Figure 39_5C Methods for asynchronously providing predictions to consumer applications at a prediction engine hub are shown according to some embodiments.

[0527] Figure 40_1 is a block diagram of an exemplary system for monitoring, predicting, and notifying of changes in a current context of a contextual client computing device, according to some embodiments.

[0528] Figure 40_2A Examples of context items that may constitute a current context according to some embodiments are shown.

[0529] Figure 40_2B Examples of new context items added to the current context are shown according to some embodiments.

[0530] Figure 40_3 An exemplary callback predicate database is shown according to some embodiments.

[0531] Figure 40_4 is a diagram illustrating example state changes associated with context items over time according to some embodiments.

[0532] Figure 40_5 is a diagram illustrating an exemplary flow of events associated with context items according to some embodiments.

[0533] Figure 40_6 An exemplary historical event stream database is shown according to some embodiments.

[0534] Figure 40_7 is a block diagram of an exemplary system for providing contextual callback notifications to requesting clients, according to some embodiments.

[0535] Figure 40_8A is a block diagram of an exemplary system according to some embodiments, illustrating restarting a requesting client that has been terminated.

[0536] Figure 40_8B is a block diagram of an exemplary system according to some embodiments, illustrating restarting a requesting client that has been terminated.

[0537] Figure 40_9A is a block diagram of an exemplary system illustrating restarting a context daemon that has been terminated, according to some embodiments.

[0538] Figure 40_9B is a block diagram of an exemplary system illustrating restarting a context daemon that has been terminated, according to some embodiments.

[0539] Figure 40_10A is a block diagram of an exemplary system illustrating restarting a context daemon and a requesting client that have been terminated, according to some embodiments.

[0540] Figure 40_10B is a block diagram of an exemplary system illustrating restarting a context daemon and a requesting client that have been terminated, according to some embodiments.

[0541] Figure 40_11 is a block diagram of an exemplary system configured to restart a client and / or context daemon based on device state information received by a launch daemon, according to some embodiments.

[0542] Figure 40_12Ais a block diagram of an exemplary system illustrating restarting a context daemon using a launch daemon according to some embodiments.

[0543] Figure 40_12B is a block diagram of an exemplary system illustrating restarting a context daemon using a launch daemon according to some embodiments.

[0544] Figure 40_13A is a block diagram of an exemplary system according to some embodiments, illustrating the use of a launch daemon to restart a requesting client.

[0545] Figure 40_13B is a block diagram of an exemplary system according to some embodiments, illustrating the use of a launch daemon to restart a requesting client.

[0546] Figure 40_14 is a diagram illustrating an example of time-slot-wise averaging for predicting future events according to some embodiments.

[0547] Figure 40_15 An exemplary diagram illustrating time slot weighting is depicted in accordance with some embodiments.

[0548] Figure 40_16A is a diagram illustrating an exemplary method for predicting future context according to some embodiments.

[0549] Figure 40_16B is a diagram illustrating an exemplary method of converting slot-wise probabilities into probability curves according to some embodiments.

[0550] Figure 40_17 An exemplary event flow including predicted future events is shown according to some embodiments.

[0551] Figure 40_18 is a flowchart of an exemplary process for notifying a client computing device of a context change according to some embodiments.

[0552] Figure 40_19 is a flow diagram of an exemplary process for restarting a context daemon to service a callback request according to some embodiments.

[0553] Figure 40_20 is a flow diagram of an exemplary process for restarting a callback client to receive callback notifications according to some embodiments.

[0554] Figure 40_21 is a flow chart of an exemplary process for predicting future events based on historical contextual information according to some embodiments.

[0555] Figure 40_22 is a flow diagram of an exemplary process for servicing a sleep context callback request according to some embodiments.

[0556] Figure 41_1 is a block diagram of one embodiment of a system for indexing application state for use in a local device search index.

[0557] Figure 41_2 is a block diagram of one embodiment of a system for searching application state using an on-device application state search index.

[0558] Figure 41_3 is a block diagram of an embodiment of a user interface for displaying application state query results among other query results.

[0559] Figure 41_4A is a flow diagram of one embodiment of a process for indexing application state received from multiple different applications on a device.

[0560] Figure 41_4B is a flow diagram of one embodiment of a process for determining query results for a query using an application state index.

[0561] Figure 41_5 is a flow diagram of one embodiment of a process for receiving and presenting application state as part of query results.

[0562] Figure 41_6 is a block diagram of one embodiment of a system for indexing application state for use in a remote search index.

[0563] Figure 41_7 is a block diagram of one embodiment of a system for searching application state using a remote application state search index.

[0564] Figure 41_8 is a flow diagram of one embodiment of a process for adding application state to an application state index.

[0565] Figure 41_9 is a flow diagram of one embodiment of a process for exporting application state to an application state indexing service.

[0566] Figure 41_10 is a flow diagram of one embodiment of a process for performing a query search using an application state index.

[0567] Figure 41_11 is a flow diagram of one embodiment of a process for receiving and presenting application state as part of query results.

[0568] Figure 41_12 is a block diagram of one embodiment of a system for indexing application state views for use in a remote search index.

[0569] Figure 41_13 is a block diagram of one implementation of an application view.

[0570] Figure 41_14 is a flow diagram of one embodiment of a process for generating an application state view using application state.

[0571] Figure 41_15 is a flow diagram of one embodiment of a process for receiving and presenting application state (including application state views) as part of query results.

[0572] Figures 42 to 55 A functional block diagram of an electronic device according to some embodiments. DETAILED DESCRIPTION

[0573] As described above and described in detail below, there is a need for electronic devices with faster and more efficient methods and interfaces for quickly accessing applications and desired functions within these applications. Specifically, there is a need for a device that can help users avoid repetitive tasks and provide active assistance by identifying and providing relevant information before the user explicitly requests it. In addition, it is necessary to quickly access applications and desired functions in applications in specific time periods (e.g., accessing a calendar application after waking up every morning), specific locations (e.g., accessing a music application at a gym), etc. This article discloses novel methods and interfaces for addressing these needs, and provides users with methods for quickly accessing these applications and functions in applications in these specific locations, time periods, etc. Such methods and interfaces optionally supplement or replace conventional methods for accessing applications. Such methods and interfaces reduce the cognitive burden on users and produce more efficient human-computer interfaces. For battery-driven devices, such methods and interfaces can save power and increase the time between battery charges. In addition, such methods and interfaces help extend the life of touch-sensitive displays by requiring a smaller number of touch inputs (e.g., the methods and interfaces disclosed herein can actively provide a desired piece of information without user input, without having to continuously and purposelessly tap the touch-sensitive display to locate the piece of information).

[0574] under, Figures 1A to 1E and Figure 2 A description of an exemplary device is provided. Figures 10 and 11 provide functional block diagrams of exemplary electronic devices. FIG. 3A to FIG. 3B and FIG. 4A to FIG. 4B is a block diagram of exemplary data structures for proactively identifying and displaying relevant content (these data structures are described in detail in reference to FIG. 6A to FIG. 6B Methods described and references FIG. 8A to FIG. 8B used in the described method). Figure 5is a block diagram illustrating an exemplary system for establishing trigger conditions for proactively identifying and displaying relevant content (the exemplary system is described in reference to FIG. 6A to FIG. 6B Methods described and references FIG. 8A to FIG. 8B used in the described method). FIG. 6A to FIG. 6B is a flow chart depicting a method for proactively identifying and displaying relevant content. FIG. 7A to FIG. 7B is a schematic diagram of a touch-sensitive display used to illustrate an exemplary user interface and gestures designed to actively identify and display relevant content. FIG. 8A to FIG. 8B is a flow chart depicting a method for proactively identifying and displaying relevant content. 9A to 9D is a schematic diagram of a touch-sensitive display illustrating an additional user interface designed to actively identify and display relevant content. FIG. 3A to FIG. 3B , FIG. 4A to FIG. 4B , Figure 5 and FIG. 7A to FIG. 7B Used to show FIG. 6A to FIG. 6B The method and / or process in. FIG. 3A to FIG. 3B , FIG. 4A to FIG. 4B , Figure 5 and 9A to 9D Used to show FIG. 8A to FIG. 8B The method and / or process in.

[0575] FIG. 10A to FIG. 10C is a flow chart depicting a method for proactively suggesting search queries based on content currently displayed on an electronic device having a touch-sensitive display. FIG. 11A to FIG. 11J is a schematic diagram of a touch-sensitive display illustrating a user interface that proactively suggests search queries based on content currently displayed on the touch-sensitive display. FIG. 11A to FIG. 11J Used to show FIG. 10A to FIG. 10C The method and / or process in.

[0576] Fig.12 is a flowchart representation of a method for entering a search mode based on a heuristic process. FIG. 13A to FIG. 13B is a schematic diagram of a touch-sensitive display for illustrating a user interface for entering a search mode based on a heuristic process. FIG. 13A to FIG. 13B Used to show Fig.12 method and / or process.

[0577] Fig.14 is a flowchart representation of a method for proactively providing a vehicle location on an electronic device with a touch-sensitive display according to some embodiments. FIG. 15A to FIG. 15B is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface designed to proactively provide a vehicle's location. FIG. 15A to FIG. 15B Used to show Fig.14 method and / or process.

[0578] FIG. 16A to FIG. 16Bis a flowchart representation of a method for proactively providing nearby point of interest (POI) information for a search query according to some embodiments. FIG. 17A to FIG. 17E is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface designed to proactively provide nearby point of interest (POI) information for search queries. FIG. 16A to FIG. 16B Used to show FIG. 17A to FIG. 17E The method and / or process in.

[0579] FIG. 18A to FIG. 18B is a flowchart representation of a method of extracting content items from a voice communication and interacting with the extracted content items according to some embodiments. FIG. 19A to FIG. 19F is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface designed to display and interact with content items extracted from voice communications. FIG. 19A to FIG. 19F Used to show FIG. 18A to FIG. 18B The method and / or process in.

[0580] Fig. 20 is a flowchart representation of a method for determining voice communications including recognizing voice at a physical location and populating an application with information about the physical location, according to some embodiments. FIG. 21A to FIG. 21B is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface for determining that voice communications include recognizing a physical location of the voice and populating an application with information about the physical location. FIG. 19A to FIG. 19F and FIG. 21A to FIG. 21B Used to show Fig. 20 method and / or process.

[0581] FIG. 22A to FIG. 22B is a flowchart representation of a method for proactively suggesting physical locations for use in an instant messaging application according to some embodiments. FIG. 23A to FIG. 23O is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface intended to proactively suggest information related to locations, events, or contacts (e.g., for user selection and inclusion in an instant messaging application). FIG. 23A to FIG. 23O Used to show FIG. 22A to FIG. 22B The method and / or process in.

[0582] Fig. 22C is a flowchart representation of a method for proactively suggesting information related to a location, event, or contact, according to some embodiments. FIG. 23A to FIG. 23O Used to show Fig. 22C The method and / or process in.

[0583] FIG. 24A to FIG. 24B is a flowchart representation of a method for proactively populating an application with information that a user previously viewed in another application, according to some embodiments. FIG. 25A to FIG. 25Jis a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface designed to proactively populate an application with information that a user previously viewed in another application (e.g., populating a ride-hailing application with information about a location that a user viewed in a check application). FIG. 25A to FIG. 25J Used to show FIG. 24A to FIG. 24B The method and / or process in.

[0584] FIG. 26A to FIG. 26B is a flowchart representation of a method of proactively suggesting that suggestion information previously viewed by a user in a first application be used in a second application, according to some embodiments. FIG. 25A to FIG. 25J Used to show FIG. 26A to FIG. 26B The method and / or process in.

[0585] Fig. 27 is a flowchart representation of proactively suggesting physical locations for use as destinations for route guidance in a vehicle, according to some embodiments. Fig.28 is a schematic diagram of a touch-sensitive display according to some embodiments, illustrating a user interface intended to proactively suggest physical locations for use as destinations for route guidance in a vehicle. Fig.28 Used to show Fig. 27 method and / or process.

[0586] Fig.29 is a flowchart representation of a method for proactively suggesting a paste action according to some embodiments. FIG. 30A to FIG. 30D is a schematic diagram of a touch-sensitive display according to some embodiments, which is used to illustrate a user interface designed to proactively suggest a paste action. FIG. 30A to FIG. 30D Used to show Fig.29 method and / or process.

[0587] More details are described in Sections 1 to 11 of the "Additional Description of the Implementation Plan" section for additional reference. FIG. 1A to FIG. 30D Details provided.

[0588] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments can be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail, so as not to unnecessarily obscure various aspects of the embodiments.

[0589] It will also be understood that, although in some cases, the terms "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish an element from another element. For example, the first contact can be named as the second contact, and similarly, the second contact can be named as the first contact, without departing from the scope of the various described embodiments. Both the first contact and the second contact are contacts, but they are not the same contact.

[0590] The terms used in the description of various described embodiments herein are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various described embodiments and in the appended claims, the singular forms "a", "an" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the items listed in association. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.

[0591] As used herein, the term "if" is optionally interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0592] The disclosure herein interchangeably refers to detecting touch input at, at, above, on top of, or substantially within a particular user interface element or portion of a touch-sensitive display. As used herein, a touch input detected “at” a particular user interface element can also be detected “on,” “over,” “on top of,” or “substantially within” the same user interface element. In some embodiments and as described in more detail below, the desired sensitivity level for detecting touch input is configured by a user of the electronic device (e.g., a user can decide (and configure the electronic device to be operated) that touch input should only be detected when the touch input is completely within a user interface element).

[0593] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc (Cupertino, California). IPOD and Device. Other portable electronic devices, such as laptops or tablets with touch-sensitive surfaces (e.g., touch screen displays and / or trackpads) are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or trackpads).

[0594] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse and / or joystick.

[0595] The device typically supports a variety of applications, such as one or more of the following applications: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a health / fitness application, a photo management application, a digital camera application, a digital camcorder application, a web browsing application, a digital music player application, and / or a digital video player application.

[0596] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.

[0597] Attention is now directed to embodiments of portable electronic devices having touch-sensitive displays. Figure 1A 1 is a block diagram illustrating a portable multifunction device 100 (also interchangeably referred to herein as electronic device 100 or device 100) with a touch-sensitive display 112 according to some embodiments. Touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes referred to as or referred to as a touch-sensitive display system. Device 100 includes memory 102 (which optionally includes one or more computer-readable storage media), controller 120, one or more processing units (CPUs) 122, peripheral device interface 118, RF circuit 108, audio circuit 110, speaker 111, microphone 113, input / output (I / O) subsystem 106, other input or control devices 116, and external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on device 100 (e.g., a touch-sensitive surface, such as touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface, such as touch-sensitive display system 112 of device 100 or a touch pad of device 100). These components optionally communicate via one or more communication buses or signal lines 103.

[0598] As used in this specification and in the claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate (surrogate) for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted average) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or changes thereto, the capacitance of the touch-sensitive surface near the contact and / or changes thereto, and / or the resistance of the touch-sensitive surface near the contact and / or changes thereto are optionally used as substitutes for the force or pressure of the contact on the touch-sensitive surface. In some implementations, the substitute measurement of contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some implementations, the substitute measurement of contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in pressure units).

[0599] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation that corresponds to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface, even when there is no change in the smoothness of the touch-sensitive surface. While such interpretation of touch by the user will be limited by the user's individualized sensory perceptions, there are many sensory perceptions of touch that are common to most users. Thus, when tactile outputs are described as corresponding to a particular sensory perception of a user (e.g., "release click," "press click," "roughness"), unless otherwise stated, the tactile outputs generated correspond to physical displacements of the device or a component thereof that would generate the described sensory perceptions for a typical (or average) user.

[0600] It should be understood that device 100 is only one embodiment of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the drawings may be implemented in hardware, software, or a combination of software and hardware, including one or more signal processing and / or application specific integrated circuits.

[0601] The memory 102 optionally includes high-speed random access memory (e.g., DRAM, SRAM, DDR RAM, or other random access solid-state memory devices), and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The memory 102 optionally includes one or more storage devices located remotely from the one or more processors 122. Access to the memory 102 by other components of the device 100 (such as the CPU 122 and the peripheral device interface 118) is optionally controlled by the controller 120.

[0602] Peripheral interface 118 may be used to couple the device's input and output peripherals to CPU 122 and memory 102. The one or more processors 122 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0603] In some embodiments, peripherals interface 118, CPU 122, and controller 120 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0604] RF (radio frequency) circuit 108 receives and transmits RF signals, also called electromagnetic signals. RF circuit 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuit 108 optionally communicates with a network and other devices via wireless communications, such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). Wireless communication optionally uses any of a variety of communication standards, protocols, and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n).

[0605] The audio circuit 110, the speaker 111, and the microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts the electrical signal into a sound wave audible to humans. The audio circuit 110 also receives an electrical signal converted by the microphone 113 according to the sound wave. The audio circuit 110 converts the electrical signal into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack. The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a single-ear or dual-ear headset) and input (e.g., a microphone).

[0606] The I / O subsystem 106 connects input / output peripherals on the device 100, such as a touch screen 112 and other input control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive electrical signals from / send electrical signals to other input control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, one or more input controllers 160 are optionally coupled to any one of the following (or not coupled to any one of the following): a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. The one or more buttons may optionally include an increase / decrease button for volume control of the speaker 111 and / or microphone 113. One or more buttons optionally include a push button.

[0607] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen 112 and / or sends electrical signals to the touch screen. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to a user interface object.

[0608] The touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on tactile and / or haptic contact. The touch screen 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of the contact) on the touch screen 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the area under the user's finger.

[0609] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, or OLED (organic light emitting diode) technology, although other display technologies are used in other embodiments. The touch screen 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112 to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as that found in Apple Inc. (Cupertino, California) IPOD and Technology in.

[0610] The touch screen 112 optionally has a video resolution of more than 400 dpi. In some embodiments, the touch screen 112 has a video resolution of at least 600 dpi. In other embodiments, the touch screen 112 has a video resolution of at least 1000 dpi. The user optionally uses any suitable object or finger such as a stylus or finger, etc. to contact the touch screen 112. In some embodiments, the user interface is designed to work primarily with finger-based contacts and gestures. In some embodiments, the device converts finger-based input into precise pointer / cursor positions or commands for performing actions desired by the user.

[0611] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface separate from the touch screen 112, or an extension of the touch-sensitive surface formed by the touch screen.

[0612] Device 100 also includes a power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0613] Device 100 optionally also includes one or more optical sensors 164 . Figure 1A An optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. The optical sensor 164 optionally includes a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the optical sensor 164 optionally captures a static image or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite to the touch screen 112 on the front of the device, so that the touch-sensitive display can be used as a viewfinder for static image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device so that the user can optionally obtain an image of the user for video conferencing while watching other video conference participants on the touch-sensitive display.

[0614] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor is shown coupled to force sensor controller 159 in I / O subsystem 106. Contact force sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). Contact force sensor 165 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the rear of device 100 opposite touch screen 112 located on the front of device 100.

[0615] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1AA proximity sensor 166 is shown coupled to peripherals interface 118. Alternatively, proximity sensor 166 is coupled to input controller 160 in I / O subsystem 106. In some embodiments, when the multifunction device is placed near a user's ear (e.g., when the user is making a phone call), the proximity sensor turns off and disables touch screen 112.

[0616] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1A A tactile output generator is shown coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices such as a speaker or other audio component; and / or an electromechanical device for converting energy into linear motion such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component for converting an electrical signal into a tactile output on the device). The contact force sensor 165 receives tactile feedback generation instructions from the tactile feedback module 133 and generates a tactile output on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is arranged in parallel or adjacent to a touch-sensitive surface (e.g., a touch-sensitive display system 112) and optionally generates tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on a rear portion of the device 100 opposite the touch-sensitive display 112 located on the front of the device 100 .

[0617] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 is shown coupled to the peripheral device interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch-sensitive display based on an analysis of data received from the one or more accelerometers. The device 100 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.

[0618] In some embodiments, the software components stored in memory 102 include an operating system 126, an active module 163 (optionally including one or more application usage data tables 335, trigger condition tables 402, trigger establishment modules 163-1, and / or usage data collection modules 163-2), a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 stores device / global internal states 157, such as Figure 1A . The device / global internal state 157 includes one or more of the following: an active application state indicating which applications, if any, are currently active; a display state indicating what applications, views, or other information occupy various areas of the touch-sensitive display 112; a sensor state including information obtained from the device's various sensors and input control devices 116; and position information regarding the device's position and / or posture (e.g., device orientation).

[0619] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitates communication between various hardware and software components.

[0620] The communication module 128 facilitates communication with other devices through one or more external ports 124, and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., a universal serial bus (USB), FireWire, etc.) is suitable for direct coupling to other devices or indirect coupling through a network (e.g., the Internet, a wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is identical to, or similar to, and / or compatible with the 30-pin connector used in some embodiments of the IPOD device from APPLE Inc. In other embodiments, the external port is a multi-pin (e.g., 8-pin) connector that is identical to, or similar to, and / or compatible with the 8-pin connector used in the LIGHTNING connector from APPLE Inc.

[0621] The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger press event), determining the contact strength (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of the contact point being represented by a series of contact data. These operations are optionally applied to a single contact (e.g., one finger contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and display controller 156 detect contact on a touch pad.

[0622] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the device 100). For example, without changing the touchpad or touch-sensitive display hardware, the mouse "click" threshold of the touchpad or touch-sensitive display can be set to any one of a large range of predefined thresholds. In addition, in some specific implementations, a software setting is provided to the user of the device for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0623] The contact / motion module 130 optionally detects gesture input from a user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event and then detecting one or more finger drag events, and in some embodiments, also subsequently detecting a finger lift (lift-off) event.

[0624] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, and the like.

[0625] In some embodiments, the graphics module 132 stores data to be used to represent graphics. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data together if necessary, and then generates screen image data to be output to the display controller 156.

[0626] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 167 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0627] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts module 137, email client module 140, IM module 141, browser module 147, and any other application requiring text input).

[0628] The GPS module 135 determines the location of the device and provides such information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and maps / navigation widgets).

[0629] Application program ("app") 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0630] Contacts module 137 (sometimes called address book or contact list);

[0631] Telephone module 138;

[0632] Video conferencing module 139;

[0633] Email client module 140;

[0634] Instant messaging (IM) module 141;

[0635] Health module 142;

[0636] A camera module 143 for still images and / or video images;

[0637] Image management module 144;

[0638] Browser module 147;

[0639] Calendar module 148;

[0640] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets obtained by the user, and a user-created widget 149-6;

[0641] Search module 151;

[0642] Video and music player module 152, optionally composed of a video player module and a music player module;

[0643] Notepad module 153;

[0644] Map module 154; and / or

[0645] Online video module 155.

[0646] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, website creation applications, disk editing applications, spreadsheet applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, a widget creator module 149-6 for making user-created widgets, and voice replication.

[0647] In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, contact module 137 is optionally used to manage an address book or contact list (e.g., stored in contact module 137 in memory 102 or memory 370), including: adding one or more names to the address book; deleting one or more names from the address book; associating one or more telephone numbers, one or more email addresses, one or more physical addresses, or other information with a name; associating an image with a name; categorizing and classifying names; providing a telephone number or email address to initiate and / or facilitate communications via telephone module 138, video conferencing module 139, email client module 140, or IM module 141; and the like.

[0648] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 is optionally used to enter a sequence of characters corresponding to a phone number, access one or more phone numbers in address book 137, modify an entered phone number, dial a corresponding phone number, conduct a conversation, and disconnect or hang up when the conversation is completed. As described above, wireless communications optionally use any of a variety of communication standards, protocols, and technologies.

[0649] In combination with RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137 and telephone module 138, video conferencing module 139 includes executable instructions for initiating, conducting and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0650] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0651] In conjunction with the RF circuit 108, the touch screen 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for inputting a character sequence corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages transmitted using SMS or MMS) and Internet-based messages (e.g., messages transmitted using XMPP, SIMPLE, or IMPS).

[0652] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the video and music player module 146, the health module 142 includes executable instructions for the following operations: creating a fitness (e.g., with a time, distance, and / or calorie burn goal); communicating with a fitness sensor (a sports device such as a watch or a pedometer); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting fitness music and playing it; and displaying, storing, and transmitting fitness data.

[0653] In conjunction with touch screen 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for the following operations: capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, or deleting still images or videos from memory 102.

[0654] In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slide show or album), and storing still images and / or video images.

[0655] In combination with the RF circuit 108, the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving and displaying web pages or parts thereof and attachments and other files linked to web pages) according to user instructions.

[0656] In combination with the RF circuit 108, the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0657] In conjunction with the RF circuit 108, the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5) or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0658] In combination with the RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134 and browser module 147, a desktop widget creator module (not shown) is optionally used by a user to create a desktop widget (e.g., to transfer a user-specified portion of a web page to a desktop widget).

[0659] In conjunction with the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, image, video, and / or other files that match one or more search conditions (e.g., one or more user-specified search terms) in accordance with user instructions. In some embodiments, the search module 151 also includes executable instructions for displaying a search entry portion and a prediction portion (e.g., Fig. 9BThe search entry part 920 and the prediction part 930, and refer to 6A to 9C In some embodiments, before any user input is received in the search entry portion, search module 151 in conjunction with proactive module 163 also populates the prediction portion with affordances of suggested or predicted people, in-application actions, applications, nearby places, and / or news articles (as described below with reference to FIG. 3A to FIG. 9C discussed in more detail).

[0660] In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions to display, present, or otherwise play back video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes functionality of an MP3 player, such as the IPOD from APPLE Inc.

[0661] In conjunction with the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, and the text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0662] In combination with the RF circuit 108, the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the GPS module 135 and the browser module 147, the map module 154 is optionally used to receive, display, modify and store maps and data associated with the maps (e.g., driving directions; data relating to stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0663] In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on the touch screen or on an external display connected via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video.

[0664] like Figure 1A As shown, portable multifunction device 100 also includes an active module 163, which is used to actively identify and display relevant content (e.g., display a user interface object corresponding to a lock screen action within an application or an action within a search interface (e.g., a UI object for playing a playlist within a music application)). Active module 163 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0665] Application usage table 335;

[0666] Trigger condition table 402;

[0667] Trigger establishment module 163-1;

[0668] Using data collection module 163-2;

[0669] Active suggestion module 163-3; and

[0670] (Voice communication) content extraction module 163-4.

[0671] In conjunction with application 136, GPS module 135, operating system 126, I / O subsystem 106, RF circuitry 108, external portion 124, proximity sensor 166, audio circuitry 110, accelerometer 168, speaker 111, microphone 113, and peripheral device interface 118, application usage table 335 and usage data collection module 163-2 receive (e.g., from the components of device 100 identified above, Figure 1A) and / or store application usage data. In some embodiments, application usage is reported to usage data collection module 163-2 and then stored in application usage table 335. In some embodiments, application usage data includes all (or the most important, relevant, or predicted) contextual usage information corresponding to a user's use of a particular application 136. In some embodiments, each particular application stores usage data while a user interacts with the application, and then reports the usage data to application usage data table 335 for storage (e.g., Figure 3B The usage data 193 for the particular application 136-1 shown includes all sensor readings, actions performed within the application, device coupling information, etc., and this usage data 193 is sent to the application usage table 335 to be stored as a record within the table. For example, when a user interacts with the browser module 147, the application usage data receives and stores all contextual usage information, including the current GPS coordinates of the device 100 (e.g., determined by the GPS module 135), motion data (e.g., determined by the accelerometer 168), ambient light data (e.g., determined by the optical sensor 164), and the in-application actions performed by the user within the browser module 147 (e.g., URLs visited, time spent visiting each page), as well as other sensor data and other contextual usage information received and stored by the application usage table 335. The following references FIG. 3A to FIG. 3B Provides additional information about the application's use of table 335. Figure 5 As discussed, in some embodiments, application usage data is stored remotely (e.g., in Figure 5 One or more servers 502 shown).

[0672] The trigger condition table 402 and the trigger establishment module 163-1 receive and / or store trigger conditions established based on the usage data stored in the application usage table 335. In some embodiments, the trigger establishment module 163-1 searches and analyzes the data stored in the application usage table 335 to identify patterns. For example, if the application usage data indicates that the user always launches the music application between 3:00 p.m. and 4:00 p.m. every day, the trigger establishment module 163-1 creates and stores trigger conditions in the trigger condition table 402, which, when met (e.g., the current time is between 3:00 p.m. and 4:00 p.m.), will cause the device 100 to launch the music application or at least provide an indication to the user that the music application is available (e.g., displaying a UI object on the lock screen that allows the user to easily access the music application). The following references FIG. 4A to FIG. 4B Provides additional information about the trigger condition table 402. Figure 5As discussed, in some embodiments, a remote server identifies a pattern and establishes a trigger condition (e.g., Figure 5 One or more servers 502 are shown).

[0673] The proactive suggestion module 163-3 works in conjunction with other components of the device 100 to proactively provide content to the user for use with various different applications available on the electronic device. For example, the proactive suggestion module 163-3 provides suggested search queries and other suggested content to include in the search interface (e.g., as described below with reference to FIG. 10A to FIG. 10C discussed above), providing information to help users locate their parked vehicles (e.g., as described below with reference to Fig.14 discussed above), providing information about nearby points of interest (e.g., as described below with reference to FIG. 16A to FIG. 16B discussed above), providing content items extracted from speech provided during voice communication (e.g., as described below with reference to FIG. 18A to FIG. 18B discussed), and helped provide a number of other suggestions (e.g., as referenced below). Fig. 20 , FIG. 21A to FIG. 21B , FIG. 24A to FIG. 24B , FIG. 26A to FIG. 26B , Fig. 27 and Fig.29 As discussed above), these suggestions are intended to help users efficiently locate desired content with a minimal amount of input (e.g., without having to search for the content, the proactive suggestion module 163-3 helps ensure that the content is provided for user selection at the appropriate time).

[0674] The (voice communication) content extraction module 163-4 works in conjunction with other components of the device 100 to recognize speech related to new content items and extract new content items (e.g., contact information, information about events, and information about locations, as described below) from voice communications. FIG. 18A to FIG. 18B and Fig. 20 discussed in more detail).

[0675] Each module and application program in the above-mentioned identified modules and applications corresponds to a set of executable instructions for performing one or more functions and the methods described in this patent application (e.g., computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) do not have to be implemented as independent software programs, processes or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above-mentioned modules and data structures. In addition, memory 102 optionally stores additional modules and data structures not described above.

[0676] In some embodiments, device 100 is a device on which operation of a predefined set of functions is performed exclusively through a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for operation of device 100, the number of physical input control devices (such as push buttons, dials, etc.) on device 100 is optionally reduced.

[0677] The predefined set of functions performed exclusively through the touch screen and / or trackpad optionally includes navigating between user interfaces. In some embodiments, the trackpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to a main menu, a home menu, or a root menu. In such embodiments, a "menu button" is implemented using a trackpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a trackpad.

[0678] Figure 1B 1 is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) includes an event classifier 170 (e.g., in an operating system 126) and a selected from a portable multifunction device 100 ( Figure 1A )'s corresponding application 136-1 among the applications 136 (eg, any of the aforementioned applications stored in the memory 102 together with the application 136).

[0679] Event classifier 170 receives event information and determines the application 136-1 and the application view 191 of application 136-1 to which the event information is to be delivered. Event classifier 170 includes event monitor 171 and event distributor module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine application view 191 to which the event information is to be delivered.

[0680] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when application 136-1 resumes execution, user interface state information indicating information being displayed by application 136-1 or information ready to be displayed by application 136-1, a state queue for enabling a user to return to a previous state or view of application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0681] Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user touch on touch-sensitive display 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (through audio circuit 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

[0682] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0683] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0684] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0685] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events that are considered to be correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.

[0686] Hit view determination module 172 receives information related to sub-events of a contact-based gesture. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies a hit view as the lowest view in the hierarchy where sub-events should be processed. In most cases, the hit view is the lowest level view in which an initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0687] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, views higher in the hierarchy will still remain actively participating views.

[0688] Event distributor module 174 distributes event information to event recognizers (e.g., event recognizers 180). In embodiments including active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receivers 182.

[0689] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In another embodiment, event classifier 170 is a standalone module or part of another module stored in memory 102, such as contact / motion module 130.

[0690] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more event recognizers in event recognizers 180 are part of an independent module, such as a user interface toolkit (not shown) or a higher level object from which application 136-1 inherits methods and other properties. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178 and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177 or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views in the application views 191 include one or more corresponding event handlers 190. In addition, in some embodiments, one or more of the data updater 176, the object updater 177, and the GUI updater 178 are included in the corresponding application view 191.

[0691] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0692] Event receiver 182 receives event information from event classifier 170. Event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of the touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (e.g., from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0693] Event comparator 184 compares event information with predefined event or sub-event definitions, and based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0694] In some embodiments, event definition 186 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view that displays three user interface objects on touch display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with a sub-event and the object that triggered the hit test.

[0695] In some embodiments, the definition of the corresponding event 187 also includes a delay action that delays the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0696] When a corresponding event recognizer 180 determines that the sub-event string does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0697] In some embodiments, corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0698] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag obtains the tag and performs a predefined process.

[0699] In some embodiments, the event delivery instructions 188 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with a sub-event series or to an actively participating view. The event handler associated with the sub-event series or with an actively participating view receives the event information and performs a predetermined process.

[0700] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video or music player module 145. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 176 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends it to graphics module 132 for display on a touch-sensitive display.

[0701] In some embodiments, one or more event handlers 190 include or have access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0702] In some embodiments, each specific application 136-1 stores usage data while a user interacts with the application and then reports the usage data to the application usage data table 335 for storage (e.g., Figure 3B The usage data 193 for the specific application 136-1 shown includes all sensor readings, actions performed within the application, device coupling information, etc., and this usage data 193 is sent to the application usage table 335 of the specific application to be stored as a record within the table). In some embodiments, the usage data 193 stores data reported by the usage data collection module 163-2 while the specific application 136-1 is in use (for example, the user actively interacts with the specific application 136-1).

[0703] It should be understood that the above discussion of event processing for user touches on a touch-sensitive display also applies to other forms of user input using input devices to operate the multifunction device 100, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses, optionally combined with single or multiple keyboard presses or holds; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movements; biometric input; and / or any combination thereof, as input corresponding to a sub-event, define an...

Claims

1. A method comprising: At an electronic device having a touch-sensitive display: displaying, on the display, content associated with an application executed on the electronic device; determining at least one suggested search query based at least in part on information associated with the content; after determining the at least one suggested search query, detecting, via the touch-sensitive surface, a swipe gesture for invoking a search mode other than the application; and In response to detecting the swipe gesture, entering the search mode, wherein entering the search mode includes: Displaying a search interface on the display; and The displayed search interface is populated with the at least one suggested search query prior to receiving any user input at the search interface. 2 . The method of claim 1 , wherein detecting the swipe gesture comprises detecting the swipe gesture on at least a portion of the content currently displayed.

3. The method according to any one of claims 1 to 2, further comprising: Prior to detecting the swipe gesture, detecting an input corresponding to a request to view a home screen of the electronic device; and In response to detecting the input, ceasing to display the content associated with the application and displaying a corresponding page of the home screen of the electronic device, wherein: The corresponding page is an initial page in a sequence of home screen pages; and The swipe gesture is detected while the initial page of the home screen is displayed on the display. 4 . The method according to claim 1 , wherein the search interface is displayed to semi-transparently cover the application.

5. The method according to any one of claims 1 to 2, further comprising: Based on determining that the content includes textual content, the at least one suggested search query is determined based at least in part on the textual content.

6. The method of claim 5, wherein determining the at least one suggested search query based at least in part on the text content comprises: The textual content is analyzed to detect one or more predefined keywords for use in determining the at least one suggested search query.

7. The method according to any one of claims 1 to 2, wherein: determining the at least one suggested search query comprises determining a plurality of suggested search queries, and Populating the search interface includes populating the search interface with the plurality of suggested search queries.

8. The method according to any one of claims 1 to 2, further comprising: detecting, via the touch-sensitive surface, a new swipe gesture on new content currently displayed; and In response to detecting the new swipe gesture, entering the search mode, wherein entering the search mode includes displaying the search interface on the display; and In conjunction with entering the search mode and based on determining that the new content does not include textual content, populating the search interface with suggested search queries based on a set of historical search queries from selections of a user of the electronic device.

9. The method of any one of claims 1-2, wherein the search interface is displayed with points of interest based on location information provided by a second application different from the application.

10. The method of any one of claims 1-2, wherein the search interface further comprises one or more suggested applications. The method of claim 8 , wherein the set of historical search queries is selected based at least in part on a frequency of recent search queries.

12. The method according to any one of claims 1-2, further comprising: In conjunction with entering the search mode, the information associated with the content is obtained by using one or more auxiliary functions available on the electronic device.

13. The method of any one of claims 12, wherein using the one or more auxiliary functions comprises using the one or more auxiliary functions to generate information associated with the content by: (i) applying a natural language processing algorithm to the text content currently displayed in said application, and (ii) using data obtained from the natural language processing algorithm to determine one or more keywords that describe the content, wherein the at least one suggested search query is determined based on the one or more keywords.

14. The method of claim 13, wherein determining the one or more keywords describing the content further comprises: (i) retrieving metadata corresponding to non-text content currently displayed in the application, and (ii) using the retrieved metadata in addition to the data obtained from the natural language processing algorithm to determine the one or more keywords.

15. The method of any one of claims 1-2, wherein the search interface further comprises one or more trending queries.

16. The method of any one of claims 1-2, wherein the search interface further comprises one or more applications predicted to be of interest to a user of the electronic device.

17. The method of any of claims 1-2, wherein the swipe gesture is not received in any user interface field included in the content associated with the application.

18. A computer-readable storage medium storing executable instructions, which, when executed by an electronic device having a touch-sensitive display, cause the electronic device to perform the method according to any one of claims 1-17.

19. An electronic device comprising: Touch-sensitive display; one or more processors; and A memory storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the method according to any one of claims 1-17.

20. An electronic device, comprising: Apparatus for performing the method according to any one of claims 1-17.

21. A computer program product comprising one or more programs configured to be executed by one or more processors of an electronic device comprising a touch-sensitive surface, the one or more programs comprising instructions for performing the method of any one of claims 1-17.

22. A method comprising: At an electronic device including one or more processors, a memory, a touch-sensitive surface, a display, and a position sensor: Without receiving any instruction from the user of the electronic device: monitoring the geographic location of the electronic device using the location sensor; determining, based on the monitored geographic location, that the electronic device is within a threshold distance of a predetermined type of point of interest; According to determining that the electronic device is within the threshold distance of the point of interest: identifying at least one activity currently popular at the point of interest; retrieving information about the point of interest, including retrieving information about the at least one activity currently popular at the point of interest; detecting, via the touch-sensitive surface, a first input, the first input, when detected, causing the electronic device to enter a search mode; as well as In response to detecting the first input, entering the search mode, wherein entering the search mode includes: before any user input is received at the search interface, presenting via the display an enable representation including: (i) information about the at least one activity, and (ii) an indication that the at least one activity has been identified as currently popular at the point of interest.

23. The method according to claim 22, comprising: detecting a second input; as well as In response to detecting the second input, the affordance is updated to include available information about current activity at a second point of interest different from the point of interest, wherein the point of interest is within a threshold distance of the electronic device.

24. The method of any one of claims 22-23, wherein the affordance further comprises selectable categories of points of interest, and the method further comprises: Detecting the selection of the corresponding optional category; as well as In response to detecting the selection, the affordance is updated to include information about additional points of interest, the additional points of interest being within a second threshold distance of the device.

25. The method of any one of claims 22-23, wherein the point of interest is an amusement park, and the retrieved information includes current wait times for rides at the amusement park.

26. The method of claim 25, wherein the retrieved information includes information regarding wait times for rides located within a predefined distance of the electronic device.

27. The method of any of claims 22-23, wherein the point of interest is a restaurant, and the retrieved information includes information about popular menu items in the restaurant.

28. The method of claim 27, wherein the retrieved information is retrieved from a social network associated with the electronic device user.

29. The method of any one of claims 22-23, wherein the point of interest is a movie theater, and the retrieved information includes information about showtimes for the movie theater.

30. The method of claim 29, wherein the retrieved information is retrieved from a social network associated with the electronic device user.

31. A method according to any one of claims 22-23, wherein after unlocking the electronic device, the enabling indication is available in response to a swipe generally in a horizontal direction on an initial page of a home screen of the electronic device.

32. A computer-readable storage medium storing executable instructions which, when executed by an electronic device having a touch-sensitive surface, a display, and a position sensor, cause the electronic device to perform the method according to any one of claims 22-31.

33. An electronic device comprising: monitor; Touch-sensitive surfaces; Position sensor; as well as one or more processors; and A processing unit coupled to the touch-sensitive display, the processing unit being configured to perform the method according to any one of claims 22-31.

34. An electronic device comprising a touch-sensitive surface, a display, and a position sensor, the electronic device comprising: Apparatus for performing the method according to any one of claims 22-31.

35. A computer program product comprising one or more programs configured to be executed by one or more processors of an electronic device comprising a touch-sensitive surface, a display, and a position sensor, the one or more programs comprising instructions for performing the method of any one of claims 22-31.

Images