System and method for navigating path

Abstract

In some embodiments, an electronic device navigates and / or travels using a predetermined path and / or presence path. In some embodiments, an electronic device generates a presence path based on characteristics, criteria, and / or factors related to an environment of the electronic device. In some embodiments, an electronic device outputs an indication of a presence path to be used for navigation and / or travel.

Description

Cross-reference to related applications

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 587,091, filed September 30, 2023, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0002] This disclosure relates in general to electronic devices that present content, such as content captured using one or more cameras. Background Technology

[0003] In recent years, user interaction with electronic devices has increased significantly. These devices can include computers, tablets, televisions, multimedia devices, and mobile devices. When using electronic devices to present content, users may want to modify the playback of that content. Therefore, users may expect an efficient way to play content. Summary of the Invention

[0004] Providing efficient ways to navigate to pre-defined and on-the-spot routes can improve the user experience of electronic devices and reduce the amount of input required to reach a destination, thereby reducing power consumption and extending the battery life of electronic devices.

[0005] In some embodiments, the electronic device navigates or travels along a predetermined path and / or a pre-defined path. A full description of the embodiments is provided in the accompanying drawings and detailed descriptions; it should be understood that the above-described summary of the invention does not limit the scope of this disclosure in any way.

[0006] As is widely recognized, the use of personally identifiable information should comply with privacy policies and measures that are generally accepted to meet or exceed industry or governmental requirements for protecting user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of authorized use should be clearly explained to users. Attached Figure Description

[0007] To better understand the various described embodiments, reference should be made to the following detailed description in conjunction with the accompanying drawings, in which the same reference numerals indicate corresponding parts in all the drawings.

[0008] Figure 1A This is a block diagram illustrating a portable multi-functional device with a touch-sensitive display according to some implementation schemes.

[0009] Figure 1B This is a block diagram illustrating exemplary components for event handling according to some implementation schemes.

[0010] Figure 2Examples of portable multi-functional devices with touchscreens according to some implementation schemes are shown.

[0011] Figures 3A to 3G This is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface according to some implementation schemes.

[0012] Figure 4A An exemplary user interface for an application menu on a portable multifunction device according to some implementations is shown.

[0013] Figure 4B An exemplary user interface of a multifunctional device having a touch-sensitive surface separate from the display is illustrated according to some embodiments.

[0014] Figure 5A Examples of personal electronic devices according to some implementation schemes are shown.

[0015] Figure 5B This is a block diagram illustrating a personal electronic device according to some implementation schemes.

[0016] Figures 5C to 5D Exemplary components of a personal electronic device having a touch-sensitive display and an intensity sensor according to some embodiments are illustrated.

[0017] Figures 5E to 5H Exemplary components and user interfaces of a personal electronic device according to some implementation schemes are illustrated.

[0018] Figure 5I An example system for implementing the techniques described herein is illustrated.

[0019] Figures 6A to 6J Exemplary methods are illustrated for electronic devices according to some embodiments of the present disclosure to navigate or travel on predetermined paths and / or on-site paths.

[0020] Figure 7 This is a flowchart illustrating a method for an electronic device, according to some embodiments of the present disclosure, to navigate or travel on a predetermined path and / or a pre-defined path. Detailed Implementation

[0021] The following description illustrates exemplary techniques for use in-situ pathways. This description is not intended to limit the scope of this disclosure, but is provided as a description of specific example implementations.

[0022] Electronic devices that need to navigate and / or travel on predetermined paths and / or pre-defined paths. In some implementations, the electronic device generates one or more pre-defined paths based on one or more characteristics, criteria, and / or factors related to the environment of the electronic device. Furthermore, such techniques enable efficient navigation and / or travel on predetermined paths and / or pre-defined paths, thus reducing the amount of input required to reach the destination, thereby reducing power consumption and extending the battery life of the electronic device.

[0023] The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and in the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the items listed in connection with the description. It will also be understood that the terms “comprising” and / or “including” as used in this specification specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0024] Depending on the context, the term "if" may optionally be interpreted as meaning "when," "in response to," or "in response to detection." Similarly, depending on the context, the phrases "if it is determined..." or "if [the stated condition or event] is detected" may optionally be interpreted as meaning "in response to determining..." or "in response to detecting [the stated condition or event]."

[0025] Exemplary device This document describes implementations of electronic devices, user interfaces for such devices, and associated processes for using such devices. In some implementations, the device is a portable communication device, such as a mobile phone, that also includes other functionalities such as PDA and / or music player functionality. Exemplary implementations of portable multi-functional devices include, but are not limited to, the iPhone from Apple Inc. of Cupertino, California. ® Devices, iPod Touch ® Devices and iPads ®Device. Alternatively, other portable electronic devices may be used, such as laptops or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer or television with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). In some embodiments, the device does not have a touchscreen display and / or touchpad, but is capable of outputting display information (such as the user interface of this disclosure) for display on a separate display device, and is capable of receiving input information from a separate input device having one or more input mechanisms (such as one or more buttons, touchscreen displays, and / or touchpads). In some embodiments, the device has a display, but is capable of receiving input information from a separate input device having one or more input mechanisms (such as one or more buttons, touchscreen displays, and / or touchpads). In some embodiments, the electronic device is a computer system that communicates (e.g., via wireless or wired communication) with a display generating component (e.g., a display device such as a head-mounted display (HMD), monitor, projector, touch-sensitive display; or other device or component that presents visual content to a user, such as visual content generated on or in or from the display generating component and visible elsewhere). The display generating component is configured to provide visual output, such as display via a CRT monitor, via an LED monitor, or via image projection. In some embodiments, the display generating component is integrated with the computer system. In some embodiments, the display generating component is separate from the computer system. As used herein, “display” content includes content (e.g., video data rendered or decoded by display controller 156) that is displayed by sending data (e.g., image data or video data) to an integrated or external display generating component via a wired or wireless connection to visually generate content.

[0026] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device may optionally include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick. Additionally, as described above, it should be understood that the described electronic device, display, and touch-sensitive surface may optionally be distributed across two or more devices. Therefore, as used in this disclosure, information on or displayed by the electronic device may optionally be used to describe information output by the electronic device for display on a separate display device (touch-sensitive or non-touch-sensitive). Similarly, as used in this disclosure, input received on the electronic device (e.g., touch input received on a touch-sensitive surface of the electronic device) may optionally be used to describe input received on a separate input device from which the electronic device receives input information.

[0027] The device typically supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, game applications, telephone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, TV channel browsing applications, and / or digital video player applications.

[0028] Various applications running on this device may optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device may optionally be adapted and / or changed for different applications, and / or within the respective applications. In this way, the common physical architecture of the device (such as the touch-sensitive surface) may optionally support various applications using a user interface that is intuitive and clear to the user.

[0029] Now let’s turn our attention to implementations of portable or non-portable devices with touch-sensitive displays, but the device does not necessarily include a touch-sensitive display or a general display, as described above. Figure 1A This is a block diagram illustrating a portable or non-portable multi-functional device 100 with a touch-sensitive display 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a “touchscreen” for convenience, and is sometimes referred to as a touch-sensitive display system. Device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of device 100). Device 100 may optionally include one or more haptic output generators 167 for generating haptic output on device 100 (e.g., generating haptic output on a touch-sensitive surface such as the touch-sensitive display system 112 of device 100 or the touchpad 355 of device 300). These components may optionally communicate via one or more communication buses or signal lines 103.

[0030] As used in this specification and claims, the term "intensity" for contact on a tactile surface refers to the force or pressure (force per unit area) of a contact (e.g., finger contact) on a tactile surface, or to a substitute (alternative) for the force or pressure of a contact on a tactile surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically hundreds of different values ​​(e.g., at least 256). The intensity of a contact may optionally be determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the tactile surface may optionally be used to measure the force at different points on the tactile surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted average) to determine the estimated contact force. Similarly, the pressure-sensitive tip of a stylus may optionally be used to determine the pressure of the stylus on the tactile surface. Alternatively, the size and / or change of the contact area detected on the touch-sensitive surface, the capacitance and / or change of the touch-sensitive surface adjacent to the contact, and / or the resistance and / or change of the touch-sensitive surface adjacent to the contact may be used as substitutes for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some embodiments, the substitute measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in pressure units). Using the intensity of the contact as an attribute of user input allows users to access additional device functionality that would otherwise be inaccessible to the user on a space-constrained, scaled-down device used (e.g., on a touch-sensitive display) to display an indication and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls such as knobs or buttons).

[0031] As used in this specification and claims, the term "haptic output" refers to a physical displacement of the device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device, which is detected by the user using the user's tactile sense. For example, when the device or a component of the device comes into contact with a touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or touchpad) may optionally be interpreted by the user as a "press-click" or "release-click" on a physically actuated button. In some cases, the user will feel a tactile sensation, such as a "press-click" or "release-click," even when a physically actuated button associated with the touch-sensitive surface, which has been physically pressed (e.g., displaced) by the user's movement, does not move. For example, even when the smoothness of a tactile surface remains unchanged, movement of the tactile surface can optionally be interpreted or sensed by the user as “roughness” of the tactile surface. While such interpretations of touch by users will be limited by their individualized sensory perceptions, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a user’s specific sensory perception (e.g., “release click,” “press click,” “roughness”), unless otherwise stated, the generated tactile output corresponds to the physical displacement of the device or its components, which will generate the sensory perception described by a typical (or average) user.

[0032] It should be understood that device 100 is merely an example of a portable or non-portable multifunctional device, and device 100 may optionally have more or fewer components than shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of these components. Figure 1A The various components shown are implemented in hardware, software, or a combination of both, including one or more signal processing and / or application-specific integrated circuits (ASICs). Additionally, Figure 1A The various components shown may optionally be implemented on two or more devices; for example, a display and audio circuitry on a display device, a touch-sensitive surface on an input device, and other components on device 100. In such embodiments, device 100 may optionally communicate with display devices and / or input devices to facilitate system operation as described herein, and the various display and / or input-related components described herein are retained in device 100 or optionally included in display and / or input devices where appropriate.

[0033] Memory 102 may optionally include high-speed random access memory, and may also optionally include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 may optionally control access to memory 102 by other components of device 100.

[0034] Peripheral interface 118 can be used to couple the device's input and output peripherals to CPU 120 and memory 102. One or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0035] In some implementations, the peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In other implementations, they are optionally implemented on separate chips.

[0036] RF (Radio Frequency) circuit 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 108 converts electrical signals into electromagnetic signals / converts electromagnetic signals into electrical signals, and communicates with communication networks and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, subscriber identity module (SIM) cards, memory, etc. RF circuit 108 optionally communicates wirelessly with networks (such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs))) and other devices. RF circuit 108 optionally includes well-known circuitry for detecting near-field communication (NFC) fields, such as via short-range communication radio components. Wireless communication may optionally use any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Evolution, Pure Data (EV-DO), HSPA, HSPA+, Dual-Unit HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), and Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n and / or IEEE 802.11ac), Internet Protocol Voice (VoIP), Wi-MAX, email protocols (e.g., Internet Messaging Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence with Extended Utility (SIMPLE), Instant Messaging and Presence Service (IMPS)) and / or Short Message Service (SMS), or any other suitable communication protocol that has not been developed as of the date of this document submission.

[0037] Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and device 100. Audio circuitry 110 receives audio data from peripheral interface 118, converts the audio data into electrical signals, and sends the electrical signals to speaker 111. Speaker 111 converts the electrical signals into sound waves that are audible to humans. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuitry 110 converts the electrical signals into audio data and sends the audio data to peripheral interface 118 for processing. Audio data may optionally be retrieved by peripheral interface 118 from memory 102 and / or RF circuitry 108 and / or sent to that memory and / or RF circuitry. In some embodiments, audio circuitry 110 also includes a headset jack (e.g., ...). Figure 2 (212 in the text). The headset jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both outputs (e.g., a mono-ear headset or a binaural headset) and inputs (e.g., a microphone).

[0038] I / O subsystem 106 couples input / output peripherals (such as touchscreen 112 and other input control devices 116) on device 100 to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, optical sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive electrical signals from / transmit electrical signals to other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, joysticks, click dials, etc. In some alternative embodiments, input controllers 160 may optionally be coupled to (or not coupled to) any of the following: keyboard, infrared port, USB port, and pointing devices such as a mouse. The one or more buttons (e.g., Figure 2 Optionally, 208 may include increase / decrease buttons for volume control of speaker 111 and / or microphone 113. These one or more buttons may optionally include a push-button (e.g., Figure 2 (206 in the middle).

[0039] A quick press of the push button may optionally unlock the touchscreen 112 or optionally initiate a process of unlocking the device using gestures on the touchscreen, as described in U.S. Patent Application 11 / 322,549 (U.S. Patent No. 7,657,849), filed December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," the entire contents of which are incorporated herein by reference. A long press of the push button (e.g., 206) may optionally power on or off the device 100. The functionality of one or more of these buttons may optionally be user-customizable. The touchscreen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.

[0040] The touch-sensitive display 112 provides input and output interfaces between the device and the user. As described above, the touch-sensitive operation and display operation of the touch-sensitive display 112 can optionally be separated from each other, such that the display device is used for display purposes, while the touch-sensitive surface (whether or not a display) is used for input detection purposes, and the described components and functions are modified accordingly. However, for the sake of brevity, the following description refers to the touch-sensitive display. The display controller 156 receives electrical signals from the touchscreen 112 and / or transmits electrical signals to the touchscreen. The touchscreen 112 displays visual output to the user. The visual output may optionally include graphics, text, icons, video, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects.

[0041] Touchscreen 112 has a touch-sensitive surface, sensor, or sensor array that accepts input from a user based on tactile and / or haptic contact. Touchscreen 112 and display controller 156 (along with any associated modules and / or instruction set in memory 102) detect contact on touchscreen 112 (and any movement or interruption of that contact) and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touchscreen 112. In an exemplary embodiment, the contact point between touchscreen 112 and the user corresponds to the user's finger.

[0042] Touchscreen 112 may optionally employ LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, but other display technologies may be used in other embodiments. Touchscreen 112 and display controller 156 may optionally use any of a variety of touch sensing technologies now known or to be developed hereafter, along with other proximity sensor arrays or other elements for determining one or more points of contact with touchscreen 112, to detect contact and any movement or interruption thereof. These various touch sensing technologies include, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. In one exemplary embodiment, projected mutual capacitance sensing technology, such as that used in Apple Inc.'s iPhone (Cupertino, California), is used. ® iPod Touch ® and iPad ® The technology that was discovered.

[0043] In some embodiments of the touchscreen 112, the touch-sensitive display may optionally resemble a multi-touch-sensitive touchpad described in the following U.S. patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman) and / or U.S. Patent Publication 2002 / 0015024A1, each of which is incorporated herein by reference in its entirety. However, the touchscreen 112 displays visual output from the device 100, while the touch-sensitive touchpad does not provide visual output.

[0044] The touch-sensitive display in some embodiments of the touchscreen 112 is described in the following applications: (1) U.S. Patent Application No. 11 / 381,313, filed May 2, 2006, “Multipoint Touch Surface Controller”; (2) U.S. Patent Application No. 10 / 840,862, filed May 6, 2004, “Multipoint Touchscreen”; (3) U.S. Patent Application No. 10 / 903,964, filed July 30, 2004, “Gestures For Touch Sensitive Input Devices”; (4) U.S. Patent Application No. 11 / 48,264, filed January 31, 2005, “Gestures For Touch Sensitive Input Devices”; and (5) U.S. Patent Application No. 11 / 38,590, filed January 18, 2005, “Mode-Based Graphical User Interfaces For Touch Sensitive Input”. (6) U.S. Patent Application No. 11 / 228,758, filed September 16, 2005, “Virtual Input Device Placement On A Touch Screen User Interface”; (7) U.S. Patent Application No. 11 / 228,700, filed September 16, 2005, “Operation Of A Computer With A Touch Screen Interface”; (8) U.S. Patent Application No. 11 / 228,737, filed September 16, 2005, “Activating Virtual Keys Of ATouch-Screen Virtual Keyboard”; and (9) U.S. Patent Application No. 11 / 367,749, filed March 3, 2006, “Multi-Functional Hand-Held Device”. The full text of all these applications is incorporated herein by reference.

[0045] The touchscreen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 160 dpi. Users may optionally use any suitable object or accessory, such as a stylus, finger, etc., to interact with the touchscreen 112. In some embodiments, the user interface is designed to operate primarily through finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on the touchscreen. In some embodiments, the device translates coarse finger-based input into precise pointer / cursor positioning or commands for performing the user-desired actions.

[0046] In some embodiments, in addition to the touchscreen, device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display visual output. The touchpad may optionally be a touch-sensitive surface separate from the touchscreen 112, or an extension of the touch-sensitive surface formed by the touchscreen.

[0047] The device 100 also includes a power system 162 for supplying power to various components. The power system 162 may optionally include a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power in portable or non-portable devices.

[0048] The device 100 may optionally also include one or more optical sensors 164. Figure 1AAn optical sensor 164 coupled to an optical sensor controller 158 in I / O subsystem 106 is shown. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with an imaging module 143 (also referred to as a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of device 100, opposite to a touchscreen display 112 on the front of the device, allowing the touchscreen display to be used as a viewfinder for still image and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device, allowing images of the user to be optionally acquired for video conferencing while the user views other video conferencing participants on the touchscreen display. In some embodiments, the positioning of the optical sensor 164 can be changed by the user (e.g., by rotating the lenses and sensors in the device housing), allowing a single optical sensor 164 to be used in conjunction with the touchscreen display for both video conferencing and still image and / or video image acquisition.

[0049] The device 100 may optionally also include one or more contact strength sensors 165. Figure 1A A contact strength sensor is shown coupled to a strength sensor controller 159 in I / O subsystem 106. The contact strength sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). The contact strength sensor 165 receives contact strength information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact strength sensor is arranged juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact strength sensor is located on the rear of device 100, opposite to the touchscreen display 112 located on the front of device 100.

[0050] The device 100 may optionally also include one or more proximity sensors 166. Figure 1AA proximity sensor 166 coupled to a peripheral device interface 118 is shown. Alternatively, the proximity sensor 166 may be coupled to an input controller 160 in an I / O subsystem 106. The proximity sensor 166 may optionally perform as described in the following U.S. patent applications: 11 / 241,839, entitled "Proximity Detector In Handheld Device"; 11 / 240,788, entitled "Proximity Detector In Handheld Device"; 11 / 620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; 11 / 586,862, entitled "Automated Response To And Sensing Of User Activity In Portable Devices"; and 11 / 638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals", the entire contents of which are incorporated herein by reference. In some implementations, when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call), the proximity sensor is turned off and the touchscreen 112 is disabled.

[0051] The device 100 may optionally also include one or more tactile output generators 167. Figure 1A A haptic output generator coupled to a haptic feedback controller 161 in I / O subsystem 106 is shown. The haptic output generator 167 optionally includes one or more electroacoustic devices, such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert electrical signals into haptic outputs on the device). A contact intensity sensor 165 receives haptic feedback generation instructions from a haptic feedback module 133 and generates a haptic output on device 100 that can be felt by a user of device 100. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a haptic surface (e.g., a haptic display system 112) and optionally generates the haptic output by moving the haptic surface vertically (e.g., in / outward from the surface of device 100) or laterally (e.g., back and forth in the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is located on the rear of the device 100, opposite to the touch screen display 112 located on the front of the device 100.

[0052] The device 100 may optionally also include one or more accelerometers 168. Figure 1A An accelerometer 168 coupled to a peripheral device interface 118 is shown. Alternatively, the accelerometer 168 may be coupled to an input controller 160 in an I / O subsystem 106. The accelerometer 168 may optionally perform as described in the following U.S. patent publications: U.S. Patent Publication No. 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication No. 20060017692, entitled "Methods and Apparatuses For Operating A Portable Device Based On An Accelerometer," both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from one or more accelerometers. In addition to the accelerometer 168, the device 100 may optionally include a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the location and orientation (e.g., longitudinal or lateral) of the device 100.

[0053] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Furthermore, in some embodiments, memory 102 ( Figure 1A ) or 370 ( Figure 3A Storage device / global internal state 157, such as Figure 1A As shown in Figure 3, the device / global internal state 157 includes one or more of the following: active application state, which indicates which applications (if any) are currently active; display state, which indicates what applications, views or other information occupy various areas of the touchscreen display 112; sensor state, which includes information obtained from the device's various sensors and input control devices 116; and position information relating to the device's position and / or orientation.

[0054] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0055] The communication module 128 facilitates communication with other devices via one or more external ports 124 and includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., the Internet, Wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used on iPod (Apple Inc. trademark) devices.

[0056] The contact / motion module 130 optionally detects contact with the touchscreen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether a contact has occurred (e.g., detecting a finger press event), determining the contact intensity (e.g., the force or pressure of the contact, or an alternative to force or pressure), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or a contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations can optionally be applied to a single contact (e.g., a single finger contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contact). In some implementations, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0057] In some implementations, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., determining whether the user has “clicked” an icon). In some implementations, at least one subset of intensity thresholds is determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a specific physical actuator and can be adjusted without changing the physical hardware of device 100). For example, the mouse “click” threshold of a touchpad or touchscreen display can be set to any threshold in a wide range of predefined thresholds without changing the touchpad or touchscreen display hardware. Additionally, in some specific implementations, the user of the device is provided with software settings for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting the individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using system-level clicks on the “intensity” parameter).

[0058] The touch / motion module 130 optionally detects gesture input performed by the user. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different movements, timings, and / or intensities of the detected contact). Therefore, gestures can be detected optionally by detecting specific contact patterns. For example, detecting a finger tap gesture includes: detecting a finger press event, and then detecting a finger lift-off (lift-away) event at the same (or substantially the same) location as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes: detecting a finger press event, then detecting one or more finger drag events, and subsequently detecting a finger lift-off (lift-away) event.

[0059] The graphics module 132 includes various known software components for rendering and displaying graphics on the touchscreen 112 or other displays, including components for altering the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). As used herein, the term "graphics" includes any object that can be displayed to a user, and includes, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, and the like.

[0060] In some implementations, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from applications, etc., to specify the graphics to be displayed, and also receives coordinate data and other graphic attribute data if necessary, and then generates screen image data to output to the display controller 156.

[0061] The haptic feedback module 133 includes various software components for generating instructions which are used by the haptic output generator 167 to generate haptic output at one or more locations on the device 100 in response to user interaction with the device 100.

[0062] Optionally, the text input module 134, which is a component of the graphics module 132, provides a soft keyboard for entering text in various applications, such as contacts 137, email 140, IM 141, browser 147, and any other application that requires text input.

[0063] GPS module 135 determines the location of the device and provides that information for use in various applications (e.g., to telephone 138 for location-based dialing; to camera 143 as image / video metadata; and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0064] Application 136 may optionally include the following modules (or instruction sets) or subsets or supersets thereof: • Contacts module 137 (sometimes called address book or contact list); • Telephone module 138; • Video conferencing module 139; • Email client module 140; • Instant Messaging (IM) module 141; • Fitness support module 142; • Camera module 143 for still images and / or video images; • Image management module 144; • Video player module; • Music player module; • Browser module 147; • Calendar module 148; • Widget module 149, which may optionally include one or more of the following: weather widget 149-1, stock market widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets acquired by the user, and user-created widgets 149-6. • Widget creator module 150 for creating user-created widgets 149-6; • Search module 151; • Video and music player module 152, which combines a video player module and a music player module; • Memo module 153; • Map module 154; and / or • Online video module 155.

[0065] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0066] In conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., in application internal state 192 of the contact module 137 stored in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing module 139, email 140, or IM 141; and so on.

[0067] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, telephone module 138 is optionally used to input character sequences corresponding to telephone numbers, access one or more telephone numbers in contact module 137, modify input telephone numbers, dial corresponding telephone numbers, initiate conversations, and disconnect or hang up when a conversation is completed. As noted above, wireless communication may optionally use any of a variety of communication standards, protocols, and technologies.

[0068] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, text input module 134, contact module 137, and telephone module 138, video conferencing module 139 includes executable instructions to initiate, conduct, and terminate video conferences between the user and one or more other participants based on user instructions.

[0069] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. Combined with image management module 144, email client module 140 makes it very easy to create and send emails containing still images or video images captured by camera module 143.

[0070] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the instant messaging module 141 includes executable instructions for performing the following operations: entering a character sequence corresponding to an instant message, modifying previously entered characters, sending a corresponding instant message (e.g., using Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocols for telephone-based instant messaging or using XMPP, SIMPLE, or IMPS for internet-based instant messaging), receiving an instant message, and viewing received instant messages. In some embodiments, the sent and / or received instant messages may optionally include graphics, photographs, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Services (EMS). As used herein, "instant messaging" refers to both telephone-based messages (e.g., messages delivered using SMS or MMS) and internet-based messages (e.g., messages delivered using XMPP, SIMPLE, or IMPS).

[0071] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module, fitness support module 142 includes executable instructions for performing the following operations: creating fitness activities (e.g., with time, distance, and / or calorie burning goals); communicating with fitness sensors (exercise equipment); receiving fitness sensor data; calibrating sensors used to monitor fitness; selecting and playing music for fitness activities; and displaying, storing, and transmitting fitness data.

[0072] In conjunction with the touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, and image management module 144, the camera module 143 includes executable instructions for performing the following operations: capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, or deleting still images or videos from memory 102.

[0073] Incorporating the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, the image management module 144 includes executable instructions for performing operations such as arranging, modifying (e.g., editing) or otherwise manipulating, marking, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0074] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions for performing the following operations: browsing the Internet according to user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, as well as links to attachments and other files on web pages.

[0075] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions to create, display, modify, and store calendars and associated data (e.g., calendar entries, to-dos, etc.) according to user instructions.

[0076] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and browser module 147, widget module 149 may be a micro-application (e.g., weather widget 149-1, stock market widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or a user-created micro-application (e.g., user-created widget 149-6). In some embodiments, the widget includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the widget includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! widgets).

[0077] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and browser module 147, widget creator module 150 can optionally be used by the user to create widgets (e.g., turning a user-specified section of a webpage into a widget).

[0078] In conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the search module 151 includes executable instructions for performing the following operations: searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.

[0079] Incorporating touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions allowing users to download and play back recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touchscreen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0080] Combining the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the memo module 153 includes executable instructions for creating and managing memos, to-do items, etc., according to user instructions.

[0081] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 may optionally be used to receive, display, modify, and store maps and map-related data (e.g., driving directions, data related to shops and other points of interest at or near a specific location, and other location-based data) according to user instructions.

[0082] Incorporating touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, the online video module 155 includes instructions for performing the following operations: allowing users to access, browse, receive (e.g., via streaming and / or downloading), play back (e.g., on the touchscreen or on an external display connected via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, an instant messaging module 141 is used instead of the email client module 140 to send links to specific online videos. Additional descriptions of the online video application can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11 / 968,67, filed December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are incorporated herein by reference in their entirety.

[0083] Each of the modules and applications described above corresponds to an executable instruction set for performing one or more of the functions described above and the methods described in this patent application (e.g., computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) need not be implemented as separate software programs, processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. For example, a video player module may optionally be combined with a music player module into a single module (e.g., Figure 1A (e.g., video and music player module 152). In some embodiments, memory 102 may optionally store a subset of the modules and data structures identified above. Additionally, memory 102 may optionally store additional modules and data structures not described above.

[0084] In some implementations, device 100 is a device whose operation of a predefined set of functions is performed solely via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for operating device 100, the number of physical input control devices (such as push-buttons and dial pads) on device 100 can be optionally reduced.

[0085] A predefined set of functions unique to touchscreens and / or touchpads may optionally include navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 100 from any user interface displayed on device 100 to the main menu, main desktop menu, or root menu. In such embodiments, a "menu button" is implemented using a touchpad. In some other embodiments, the menu button is a physical push-button or other physical input control device, rather than a touchpad.

[0086] Figure 1B This is a block diagram illustrating exemplary components for event handling according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3A This includes an event classifier 170 (e.g., in operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).

[0087] Event classifier 170 receives event information and determines the application 136-1 and application view 191 of application 136-1 to which the event information should be delivered. Event classifier 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or running. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information should be delivered.

[0088] In some implementations, the application internal state 192 includes additional information such as one or more of the following: recovery information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or ready to be displayed by the application 136-1, a state queue for enabling the user to return to the previous state or view of the application 136-1, and a repeat / undo queue for the user's previous actions.

[0089] Event monitor 171 receives event information from peripheral interface 118. The event information includes information about sub-events, such as a user touch on touch-sensitive display 112 as part of a multi-touch gesture. Peripheral interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, accelerometer 168, and / or microphone 113 (via audio circuitry 110). The information received by peripheral interface 118 from I / O subsystem 106 includes information from touch-sensitive display 112 or touch-sensitive surfaces.

[0090] In some implementations, the event monitor 171 sends requests to the peripheral device interface 118 at predetermined intervals. In response, the peripheral device interface 118 sends event information. In other implementations, the peripheral device interface 118 sends event information only when a significant event occurs (e.g., receiving input above a predetermined noise threshold and / or receiving input for a predetermined duration).

[0091] In some implementations, the event classifier 170 also includes a hit view determination module 172 and / or an activity event recognizer determination module 173.

[0092] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides a software process for determining where a sub-event has occurred within one or more views. A view consists of controls and other elements that the user can see on the display.

[0093] Another aspect of the user interface associated with an application is a collection of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected optionally corresponds to a procedural level within the application's procedural or view hierarchy. For example, the lowest-level view in which a touch is detected may optionally be called the hit view, and the set of events identified as correct input may optionally be determined at least in part based on the hit view of the initial touch that initiates the touch-based gesture.

[0094] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchical structure, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchical structure from which the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events forming an event or potential event) occurs. Once the hit view is identified by the hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source to which it was identified as the hit view.

[0095] The activity event recognizer determination module 173 determines which views(s) within the view hierarchy should receive a specific sub-event sequence. In some embodiments, the activity event recognizer determination module 173 determines that only the hit view should receive the specific sub-event sequence. In other embodiments, the activity event recognizer determination module 173 determines that all views including the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive the specific sub-event sequence. In other embodiments, even if the touch sub-event is entirely confined to the area associated with a particular view, higher views in the hierarchy will still remain actively participating views.

[0096] Event assigner module 174 assigns event information to event identifiers (e.g., event identifier 180). In embodiments that include active event identifier determination module 173, event assigner module 174 delivers event information to the event identifier determined by active event identifier determination module 173. In some embodiments, event assigner module 174 stores event information in an event queue, which is retrieved by the corresponding event receiver 182.

[0097] In some implementations, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another implementation, event classifier 170 is a separate module or part of another module (such as contact / motion module 130) stored in memory 102.

[0098] In some implementations, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each application view including instructions for handling touch events occurring within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes a plurality of event recognizers 180. In other implementations, one or more of the event recognizers 180 are part of a separate module, such as a user interface toolkit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some implementations, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handlers 190 may optionally utilize or invoke the data updater 176, the object updater 177, or the GUI updater 178 to update the application's internal state 192. Alternatively, one or more application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0099] The corresponding event identifier 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies the event based on the event information. The event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event identifier 180 also includes at least one subset of metadata 183 and event delivery instructions 188 (which may optionally include sub-event delivery instructions).

[0100] Event receiver 182 receives event information from event classifier 170. The event information includes information about sub-events, such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves touch movement, the event information may optionally also include the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a lateral orientation, or vice versa), and the event information includes corresponding information about the device's current orientation (also referred to as device orientation).

[0101] Event comparator 184 compares event information with predefined event or sub-event definitions and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 contains definitions of events (e.g., predefined sequences of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, event 1 (187-1) is defined as a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift-off of a predetermined duration (touch end), a second touch (touch start) of a predetermined duration on the displayed object, and a second lift-off of a predetermined duration (touch end). In another example, event 2 (187-2) is defined as a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on the displayed object, movement of the touch on the touch-sensitive display 112, and lifting the touch (end of touch). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0102] In some implementations, event definition 187 includes definitions of events for corresponding user interface objects. In some implementations, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub-event and the object that triggered the hit test.

[0103] In some implementations, the definition of the corresponding event (187) also includes a delay action that delays the delivery of event information until it has been determined whether the sub-event sequence actually corresponds to or does not correspond to the event type of the event recognizer.

[0104] When the corresponding event recognizer 180 determines that the sub-event sequence does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events based on touch gestures are ignored. In this case, other event recognizers (if any) that remain active in the hit view continue to track and process the ongoing sub-events based on touch gestures.

[0105] In some embodiments, the corresponding event recognizer 180 includes metadata 183 having configurable attributes, flags, and / or lists instructing how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing how or how event recognizers can interact with each other. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0106] In some implementations, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates the event handler 190 associated with the event. In some implementations, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from delivering (and deferred delivering) the sub-events to the corresponding hit view. In some implementations, the event recognizer 180 throws a flag associated with the identified event, and the event handler 190 associated with the flag retrieves the flag and performs a predefined process.

[0107] In some implementations, event delivery instruction 188 includes a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or the actively participating view receives the event information and performs a predetermined process.

[0108] In some implementations, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some implementations, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the positioning of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and transmits that display information to graphics module 132 for display on a touch-sensitive display.

[0109] In some implementations, event handler 190 includes, or has access to, a data updater 176, an object updater 177, and a GUI updater 178. In some implementations, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other implementations, they are included in two or more software modules.

[0110] It should be understood that the above discussion regarding event handling for user touch on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, and not all user input is initiated on the touchscreen. For example, mouse movement and mouse button presses optionally in conjunction with single or multiple keyboard presses or holds; touch movements on the touchpad, such as taps, drags, scrolls, etc.; stylus input; device movement; verbal commands; detected eye movements; biometric input; and / or any combination thereof may optionally be used as input corresponding to sub-events that define the event to be identified.

[0111] Figure 2 Portable or non-portable multifunction device 100 with touchscreen 112 according to some embodiments is illustrated. As stated above, multifunction device 100 is described as having various illustrated structures (such as touchscreen 112, speaker 111, accelerometer 168, microphone 113, etc.); however, it should be understood that these structures may optionally reside on separate devices. For example, display-related structures (e.g., display, speaker, etc.) and / or functions may optionally reside on separate display devices, input-related structures (e.g., touch-sensitive surface, microphone, accelerometer, etc.) and / or functions may optionally reside on separate input devices, and other structures and / or functions may optionally reside on multifunction device 100.

[0112] Touchscreen 112 optionally displays one or more graphics within user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more graphics by gesturing over the graphics, for example, using one or more fingers 202 (not drawn to scale in the diagram) or one or more styluses 203 (not drawn to scale in the diagram). In some embodiments, selection of one or more graphics occurs when the user breaks contact with the graphics. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or scrolling (from right to left, from left to right, up and / or down) of a finger already in contact with device 100. In some embodiments or in some cases, unintentional contact with a graphic does not select the graphic. For example, a swipe gesture over an application icon may optionally not select the corresponding application when the gesture corresponding to selection is a tap.

[0113] Device 100 may optionally also include one or more physical buttons, such as a "main desktop" or menu button 204. As previously described, menu button 204 may optionally be used to navigate to any application 136 of a set of applications optionally executed on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on touchscreen 112.

[0114] In one embodiment, device 100 includes a touchscreen 112, a menu button 204, a push-button 206 for powering on / off and locking the device, one or more volume control buttons 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push-button 206 is optionally used to: power on / off the device by pressing the button and holding it in the pressed state for a predefined time interval; lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or unlock the device or initiate an unlocking process. In another embodiment, device 100 also accepts voice input via microphone 113 for activating or deactivating certain functions. Device 100 may also optionally include one or more contact strength sensors 165 for detecting the intensity of contact on the touchscreen 112, and / or one or more haptic output generators 167 for generating haptic output for a user of device 100.

[0115] Figure 3AThis is a block diagram of an exemplary multifunctional device with a display and a touch-sensitive surface according to some embodiments. As described above, device 300 does not necessarily include a display and a touch-sensitive surface; instead, in some embodiments, it optionally communicates with displays and touch-sensitive surfaces on other devices. Additionally, device 300 does not necessarily have to be portable. In some embodiments, device 300 is a laptop computer, desktop computer, tablet computer, multimedia player device (such as a television or set-top box), navigation device, educational device (such as a children's learning toy), gaming system, or control device (e.g., a home or industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuitry (sometimes referred to as a chipset) that interconnects system components and controls communication between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touchscreen display. I / O interface 330 may also optionally include keyboard and / or mouse (or other pointing device) 350 and touchpad 355, and haptic output generator 357 for generating haptic output on device 300 (e.g., similar to the above reference). Figure 1A The described tactile output generator 167) and sensor 359 (e.g., optical sensor, accelerometer, proximity sensor, touch sensor and / or similar to those described above) are referenced in the reference. Figure 1A The contact strength sensor 165 described herein is a contact strength sensor. Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310. In some embodiments, memory 370 stores programs, modules, and data structures, or subsets thereof, which are similar to portable or non-portable multifunction device 100 (…). Figure 1A The memory 370 stores programs, modules, and data structures in the memory 102 of the portable or non-portable multifunction device 100. Additionally, the memory 370 may optionally store additional programs, modules, and data structures not present in the memory 102 of the portable or non-portable multifunction device 100. For example, the memory 370 of the device 300 may optionally store a drawing module 380, a rendering module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while the portable or non-portable multifunction device 100 ( Figure 1A The memory 102 may optionally not store these modules.

[0116] Figure 3A Each of the elements identified above may optionally be stored in one or more of the previously mentioned memory devices. Each of the modules identified above corresponds to an instruction set for performing the functions described above. The modules or programs identified above (i.e., instruction sets) need not be implemented as standalone software programs, processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 may optionally store a subset of the modules and data structures described above. Furthermore, memory 370 may optionally store additional modules and data structures not described above.

[0117] Specific embodiments within the scope of this disclosure may be implemented, in whole or in part, using a tangible computer-readable storage medium (or a plurality of tangible computer-readable storage media of one or more types) that encodes one or more computer-readable instructions. It should be understood that computer-readable instructions may be organized in any format, including applications, widgets, processes, software, and / or components.

[0118] Specific embodiments within the scope of this disclosure include computer-readable storage media that encode instructions organized as an application (e.g., application 3160) that, when executed by one or more processing units, control the execution of an electronic device (e.g., device 3150). Figure 3B Methods Figure 3C The methods and / or one or more other processes and / or methods described herein.

[0119] It should be recognized that, ( Figure 3D Application 3160 (as shown) can be any suitable type of application, including one or more of the following: browser applications, applications used as execution environments for plugins, widgets, or other applications, fitness applications, health applications, digital payment applications, media applications, social networking applications, messaging applications, and / or map applications. In some embodiments, application 3160 is an application pre-installed on device 3150 at the time of purchase (e.g., a first-party application). In some embodiments, application 3160 is an application provided to device 3150 via operating system update files (e.g., a first-party or second-party application). In some embodiments, application 3160 is an application provided via an app store. In some embodiments, the app store can be an app store pre-installed on device 3150 at the time of purchase (e.g., a first-party app store). In some embodiments, the app store is a third-party app store (e.g., an app store provided by another app store, downloaded via a network, and / or read from a storage device).

[0120] refer to Figure 3B and Figure 3FApplication 3160 obtains information (e.g., 3010). In some embodiments, at 3010, information is obtained from at least one hardware component of device 3150. In some embodiments, at 3010, information is obtained from at least one software module of device 3150. In some embodiments, at 3010, information is obtained from at least one hardware component external to device 3150 (e.g., peripheral devices, accessory devices, and / or servers). In some embodiments, the information obtained at 3010 includes location information, time information, notification information, user information, environmental information, electronic device status information, weather information, media information, historical information, event information, hardware information, and / or motion information. In some embodiments, in response to obtaining information at 3010 and / or thereafter, application 3160 provides the information to the system (e.g., 3020).

[0121] In some implementations, the system (e.g., Figure 3E The 3110 shown is the operating system hosted on the device 3150. In some implementations, the system (e.g., Figure 3E 3110 shown in the figure is an external device (e.g., a server, peripheral device, accessory and / or personal computing device) that includes an operating system.

[0122] refer to Figure 3C and Figure 3G Application 3160 obtains information (e.g., 3030). In some embodiments, the information obtained at 3030 includes location information, time information, notification information, user information, environmental information, electronic device status information, weather information, media information, historical information, event information, hardware information, and / or motion information. In response to obtaining information at 3030 and / or thereafter, application 3160 performs an operation on the information (e.g., 3040). In some embodiments, the operation performed at 3040 includes: providing notifications based on the information, sending messages based on the information, displaying information, controlling the user interface of a fitness application based on the information, controlling the user interface of a health application based on the information, controlling focus mode based on the information, setting reminders based on the information, adding calendar entries based on the information, and / or calling the API of system 3110 based on the information.

[0123] In some implementations, execution is performed in response to a trigger. Figure 3B Methods and / or Figure 3C The method involves one or more steps. In some implementations, triggering includes detecting an event, receiving a notification from system 3110, user input, and / or responding to a call to an API provided by system 3110.

[0124] In some implementations, when the instructions of application 3160 are executed, control device 3150 executes them by calling an application programming interface (API) (e.g., API 3190) provided by system 3110. Figure 3B Methods and / or Figure 3C The method. In some implementations, application 3160 executes without calling API 3190. Figure 3B Methods and / or Figure 3C At least a part of the method.

[0125] In some implementation schemes, Figure 3B Methods and / or Figure 3C One or more steps of the method involve calling the API (e.g., API 3190) using one or more parameters defined by the API. In some implementations, one or more parameters include constants, keys, data structures, objects, object classes, variables, data types, pointers, arrays, lists, or pointers to functions or methods and / or references to data or other items to be passed via the API in another way.

[0126] refer to Figure 3D Example 3150 is shown. In some embodiments, device 3150 is a personal computing device, smartphone, smartwatch, fitness tracker, head-mounted display (HMD) device, media device, public utility, speaker, television, and / or tablet computer. Figure 3D As illustrated, device 3150 includes application 3160 and operating system (e.g., Figure 3E System 3110 is shown in the diagram. Application 3160 includes application implementation module 3170 and API call module 3180. System 3110 includes API 3190 and implementation module 3100. It should be understood that device 3150, application 3160 and / or system 3110 may include... Figure 3D and Figure 3E The examples illustrate more, fewer, and / or different components.

[0127] In some implementations, application implementation module 3170 includes a set of one or more instructions corresponding to one or more operations performed by application 3160. For example, when application 3160 is a messaging application, application implementation module 3170 may include operations for receiving and transmitting messages. In some implementations, application implementation module 3170 communicates with API calling module 3180 via API 3190 (in... Figure 3E (As shown in the figure) communicates with system 3110.

[0128] In some implementations, API 3190 is a software module (e.g., a set of computer-readable instructions) that provides an interface that allows different modules (e.g., API calling module 3180) to access and / or use one or more functions, methods, procedures, data structures, classes, and / or other services provided by implementation module 3100 of system 3110. For example, API calling module 3180 can access features of implementation module 3100 through one or more API calls or enablements (e.g., embodied by function or method calls) exposed by API 3190 (e.g., software and / or hardware modules that can receive, respond to, and / or transmit API calls), and can pass data and / or control information via API calls or enablements using one or more parameters. In some implementations, API 3190 allows application 3160 to use services provided by a software development kit (SDK) library. In some implementations, application 3160 combines calls to functions or methods provided by the SDK library and API 3190, or uses data types or objects defined in the SDK library and provided by API 3190. In some implementations, API calling module 3180 makes API calls via API 3190 to access and use features of implementation module 3100 specified by API 3190. In such implementations, implementation module 3100 may return a value to API calling module 3180 via API 3190 in response to an API call. This value may report to application 3160 the capabilities or status of hardware components of device 3150, including those capabilities or statuses related to aspects such as input capabilities and status, output capabilities and status, processing capabilities, power status, storage capacity and status, and / or communication capabilities. In some implementations, API 3190 is implemented in part by firmware, microcode, or other low-level logic executed in part on the hardware components.

[0129] In some implementations, API 3190 allows the developer of API calling module 3180 (which may be a third-party developer) to utilize features provided by implementation module 3100. In such implementations, one or more API calling modules (e.g., including API calling module 3180) may exist that communicate with implementation module 3100. In some implementations, API 3190 allows multiple API calling modules written in different programming languages ​​to communicate with implementation module 3100 (e.g., API 3190 may include features for translating calls and returns between implementation module 3100 and API calling module 3180), and API 3190 is implemented in a specific programming language. In some implementations, API calling module 3180 calls APIs from different providers, such as a set of APIs from an OS provider, another set of APIs from a plugin provider, and / or another set of APIs from another provider (e.g., a software library provider) or the creator of another set of APIs.

[0130] Examples of API 3190 may include one or more of the following: pairing API (e.g., for establishing a secure connection, such as with an accessory), device detection API (e.g., for locating nearby devices, such as media devices and / or smartphones), payment API, UIKit API (e.g., for generating user interfaces), location detection API, locator API, map API, health sensor API, sensor API, messaging API, push notification API, streaming API, collaboration API, video conferencing API, app store API, advertising service API, web browser API (e.g., WebKit API), transportation API, networking API, WiFi API, Bluetooth API, NFC API, UWB API, fitness API, smart home API, contact transfer API, photo API, camera API, and / or image processing API. In some implementations, a sensor API is an API for accessing data associated with sensors of device 3150. For example, a sensor API may provide access to raw sensor data. Alternatively, a sensor API may provide data derived (and / or generated) from raw sensor data. In some implementations, sensor data includes temperature data, image data, video data, audio data, heart rate data, IMU (Inertial Measurement Unit) data, lidar data, location data, GPS data, and / or camera data. In some implementations, sensors include one or more of accelerometers, temperature sensors, infrared sensors, optical sensors, heart rate sensors, barometers, gyroscopes, proximity sensors, and / or biometric sensors.

[0131] In some embodiments, implementation module 3100 is a system (e.g., an operating system and / or server system) software module (e.g., a set of computer-readable instructions) configured to perform operations in response to receiving an API call via API 3190. In some embodiments, implementation module 3100 is configured to provide an API response (via API 3190) as a result of processing the API call. For example, implementation module 3100 and API call module 3180 can each be any of an operating system, library, device driver, API, application, or other module. It should be understood that implementation module 3100 and API call module 3180 can be the same or different types of modules. In some embodiments, implementation module 3100 is at least partially embodied in firmware, microcode, or hardware logic.

[0132] In some implementations, implementation module 3100 returns a value via API 3190 in response to an API call from API calling module 3180. While API 3190 defines the syntax and results of the API call (e.g., how to enable the API call and what the API call does), API 3190 may not reveal how implementation module 3100 performs the functionality specified by the API call. Various API calls are transmitted via one or more application programming interfaces between API calling module 3180 and implementation module 3100. Transmitting API calls may include issuing, initiating, referencing, calling, receiving, returning, and / or responding to function calls or messages. In other words, transmissions may describe the actions of API calling module 3180 or implementation module 3100. In some implementations, function calls or other references to API 3190 transmit and / or receive one or more parameters via parameter lists or other structures.

[0133] In some implementations, implementation module 3100 provides more than one API, each API providing a different view or aspect of the functionality implemented by implementation module 3100. For example, one API of implementation module 3100 may provide a first set of functions and may be exposed to third-party developers, while another API of implementation module 3100 may be hidden (e.g., not exposed) and provide a subset of the first set of functions, and also provide another set of functions, such as test or debug functions not in the first set of functions. In some implementations, implementation module 3100 calls one or more other components via lower-level APIs, thus acting as both an API calling module and an implementation module. It should be recognized that implementation module 3100 may include additional functions, methods, classes, data structures, and / or other features not specified through API 3190 and not available to API calling module 3180. It should also be recognized that API calling module 3180 may be on the same system as implementation module 3100, or may be remotely located and accessed via a network using API 3190. In some implementations, implementation module 3100, API 3190, and / or API calling module 3180 are stored in a machine-readable medium, which includes any means for storing information in a machine-readable (e.g., computer or other data processing system) form. For example, a machine-readable medium may include a magnetic disk, optical disk, random access memory, read-only memory, and / or flash memory devices.

[0134] An Application Programming Interface (API) is an interface between a first software process and a second software process, specifying the format for communication between the two processes. Limited APIs (e.g., private or partner APIs) are APIs accessible to a limited set of software processes (e.g., only software processes within the operating system or only software processes authorized to access the limited API). Public APIs are accessible to a broader set of software processes. Some APIs enable a software process to communicate or set the state of one or more input devices (e.g., one or more touch sensors, proximity sensors, vision sensors, motion / or orientation sensors, pressure sensors, intensity sensors, sound sensors, wireless proximity sensors, biometric sensors, buttons, switches, rotatable elements, and / or external controllers). Some APIs enable a software process to communicate and / or set the state of one or more output generation components (e.g., one or more audio output generation components, one or more display generation components, and / or one or more haptic output generation components). Some APIs enable specific capabilities (e.g., scrolling, handwriting, text input, image editing, and / or image creation) to be accessed, executed, and / or used by a software process (e.g., generating output for use by the software process based on input from the software process). Some APIs enable content from software processes to be inserted into templates and displayed in user interfaces with layouts and / or behaviors specified by the templates.

[0135] Many software platforms include a set of frameworks that provide core objects and behaviors that software developers need to build software applications that can be used on the platform. Software developers use these objects to display content on a screen, interact with that content, and manage interactions with the software platform. The basic behavior of a software application depends on this framework, and this framework provides software developers with numerous ways to customize the application's behavior to match the specific needs of the application. Many of these core objects and behaviors are accessed via APIs. APIs typically specify the format for communication between software processes, including specifying and grouping available variables, functions, and protocols. API calls (sometimes called API requests) are typically passed from a sending software process to a receiving software process as a way to achieve one or more of the following: the sending software process requests information from the receiving software process (e.g., for the sending software process to take an action); the sending software process provides information to the receiving software process (e.g., for the receiving software process to take an action); the sending software process requests an action from the receiving software process; or the sending software process provides information to the receiving software process about the action taken by the sending software process. In some cases, interaction with a device (e.g., using a user interface) will involve transmitting and / or receiving one or more API calls (e.g., multiple API calls) between multiple different software processes (e.g., different parts of an operating system, applications and operating systems, or different applications) via one or more APIs (e.g., via multiple different APIs). For example, when input is detected, direct sensor data is frequently processed into one or more input events, which are provided (e.g., via an API) to a receiving software process, which makes some determinations based on the input events and then (e.g., via an API) transmits information to the software process to perform an operation (e.g., change the device state and / or the user interface) based on the determinations. While the determinations and the operations performed in response can be made by the same software process, alternatively, the determinations can be made in a first software process and relayed (e.g., via an API) to a second software process different from the first software process, allowing the operation to be performed by the second software process. Alternatively, the second software process can relay instructions (e.g., via an API) to a third software process different from the first and / or second software processes to perform the operation. It should be understood that some or all user interactions with a computer system may involve one or more API calls within the steps of interacting with the computer system (e.g., between different software components of the computer system or between software components of the computer system and software components of one or more remote computer systems).It should be understood that some or all user interactions with a computer system may involve one or more API calls between steps of interaction with the computer system (e.g., between different software components of the computer system or between software components of the computer system and software components of one or more remote computer systems).

[0136] In some implementations, the application can be any suitable type of application, including one or more of the following: browser applications, applications used as execution environments for plugins, widgets or other applications, fitness applications, health applications, digital payment applications, media applications, social networking applications, messaging applications and / or map applications.

[0137] In some embodiments, the application is an application pre-installed on the first computer system at the time of purchase (e.g., a first-party application). In some embodiments, the application is an application provided to the first computer system via an operating system update file (e.g., a first-party application). In some embodiments, the application is an application provided via an app store. In some embodiments, the app store is pre-installed on the first computer system at the time of purchase (e.g., a first-party app store) and allows the download of one or more applications. In some embodiments, the app store is a third-party app store (e.g., an app store provided by another device, downloaded via a network, and / or read from a storage device). In some embodiments, the application is a third-party application (e.g., an application provided by an app store, downloaded via a network, and / or read from a storage device). In some embodiments, the application controls the first computer system to execute method 700 by calling an application programming interface (API) provided by a system process using one or more parameters. Figure 7 ) and / or method 900 (Figure 9).

[0138] In some implementations, exemplary APIs provided by system processes include one or more of the following: pairing API (e.g., for establishing a secure connection, such as with an accessory), device detection API (e.g., for locating nearby devices, such as media devices and / or smartphones), payment API, UIKit API (e.g., for generating user interfaces), location detection API, locator API, map API, health sensor API, sensor API, messaging API, push notification API, streaming API, collaboration API, video conferencing API, app store API, advertising service API, web browser API (e.g., WebKit API), transportation API, networking API, WiFi API, Bluetooth API, NFC API, UWB API, fitness API, smart home API, contact transfer API, photo API, camera API, and / or image processing API.

[0139] In some embodiments, at least one API is a software module (e.g., a set of computer-readable instructions) that provides an interface allowing different modules (e.g., an API calling module) to access and use one or more functions, methods, procedures, data structures, classes, and / or other services provided by an implementation module of a system process. The API may define one or more parameters passed between the API calling module and the implementation module. In some embodiments, API 3190 defines a first API call that can be provided by API calling module 3180. An implementation module is a system software module (e.g., a set of computer-readable instructions) configured to perform operations in response to receiving an API call via the API. In some embodiments, the implementation module is configured to provide an API response (via the API) as a result of processing the API call. In some embodiments, the implementation module is included in a device (e.g., 3150) running an application. In some embodiments, the implementation module is included in an electronic device separate from the device running the application.

[0140] Now let’s turn our attention to the implementation of the user interface, which may be optionally implemented on, for example, a portable multifunction device 100.

[0141] Figure 4A An exemplary user interface for an application menu on a portable multifunction device 100 according to some embodiments is illustrated. A similar user interface may optionally be implemented on device 300. In some embodiments, user interface 400 includes the following elements or a subset or superset thereof: • Signal strength indicator 402 for wireless communications (such as cellular signals and Wi-Fi signals); • Time 404; • Bluetooth indicator 405; • Battery status indicator 406; • Tray tray 408 with icons for frequently used applications, such as: • The telephone module 138 has an icon 416 labeled “telephone”, which optionally includes an indicator 414 indicating the number of missed calls or voicemail messages; • An icon 418 labeled “Mail” in the email client module 140, which optionally includes an indicator 410 for the number of unread emails; • The icon 420 of browser module 147, labeled "Browser"; and • The video and music player module 152 (also known as the iPod (a trademark of Apple Inc.) module 152) is marked with an icon 422 labeled "iPod"; and • Icons of other applications, such as: • Icon 424 of IM module 141 labeled "Message"; • The calendar module 148 has an icon 426 labeled "Calendar"; • The icon 428 of the image management module 144 is labeled "Photo". • The icon 430 of the camera module 143, which is labeled "camera"; • The icon 432 of the online video module 155, which is labeled "Online Video"; • Icon 434 labeled "Stocks" in the Stocks widget 149-2; • The icon 436 of the map module 154 that is labeled "map"; • The weather widget 149-1 has icon 438 labeled "weather"; • The alarm clock widget 149-4 has an icon 440 labeled "clock"; • The icon 442 of the fitness support module 142 is labeled "fitness support"; • The icon 444 labeled "Memo" in the Memo module 153; and • The icon 446, labeled "Settings," is used to set up an application or module that provides access to settings for the device 100 and its various applications 136.

[0142] It should be pointed out that, Figure 4A The illustrated icon labels are merely exemplary. For example, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels may be used for various application icons. In some embodiments, the label of a particular application icon includes the name of the application corresponding to that particular application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to that particular application icon.

[0143] Figure 4B An example is illustrated having a touch-sensitive surface 451 (e.g., separate from the display 450 (e.g., touchscreen display 112)). Figure 3A Devices such as tablets or touchpads (e.g., 355) Figure 3A An exemplary user interface on the device 300. The device 300 may also optionally include one or more contact intensity sensors (e.g., one or more sensors in sensor 359) for detecting the intensity of contact on the tactile surface 451 and / or one or more tactile output generators 357 for generating tactile output for the user of the device 300.

[0144] While some examples of inputs on a reference touchscreen display 112 (which combines a touch-sensitive surface and a display) are given below, in some embodiments the device detects inputs on a touch-sensitive surface separate from the display, such as... Figure 4B As shown. In some implementations, the touch-sensitive surface (e.g., Figure 4B 451) has a spindle (e.g., on the display (e.g., 450) corresponding to the main axis on the display (e.g., Figure 4B The main shaft of 453 in the middle (e.g., Figure 4B (452 in the example). According to these embodiments, the device detects the position corresponding to a specific location on the display (e.g., in...). Figure 4B In the diagram, 460 corresponds to 468 and 462 corresponds to 470) is in contact with the touch-sensitive surface 451 (e.g., Figure 4B (460 and 462 in the text). Thus, when the touch-sensitive surface (e.g., ...) Figure 4B 451 in the middle) and the display of a multi-functional device (e.g., Figure 4B When 450 is separated from the touch-sensitive surface, user input detected by the device on that touch-sensitive surface (e.g., touches 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods can be optionally used for other user interfaces described herein.

[0145] Additionally, while the examples below are given primarily with reference to finger input (e.g., finger touch, finger tap, finger swipe), it should be understood that in some implementations, one or more of these finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of a touch), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the touch). Similarly, a tap gesture may optionally be replaced by a mouse click while the cursor is over the location of the tap gesture (e.g., instead of detection of touch, followed by cessation of touch detection). Likewise, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice may optionally be used simultaneously, or mouse and finger touch may optionally be used simultaneously.

[0146] Additionally, while the examples below are given primarily with reference to finger input (e.g., finger touch, single-finger tap gesture, finger swipe gesture), it should be understood that in some implementations, one or more of these finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of a touch), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the touch). Similarly, a tap gesture may optionally be replaced by a mouse click while the cursor is over the location of the tap gesture (e.g., instead of detection of touch, followed by cessation of touch detection). Likewise, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice may optionally be used simultaneously, or mouse and finger touch may optionally be used simultaneously.

[0147] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some specific implementations that include a cursor or other position marker, the cursor acts as a "focus selector," such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the cursor is positioned on a touch-sensitive surface (e.g., a...). Figure 3A The touchpad 355 or Figure 4B When an input (e.g., a press input) is detected on the touch-sensitive surface 451 of the display, the specific user interface element is adjusted according to the detected input. This applies to touchscreen displays that enable direct interaction with user interface elements on the touchscreen display. Figure 1AIn some embodiments of the touch-sensitive display system 112, a touch detected on the touchscreen acts as a "focus selector," causing the specific user interface element to be selected based on the detected input when input (e.g., a press input via a touch) is detected at the location of a specific user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display. In some embodiments, focus moves from one area of ​​the user interface to another without corresponding movement of the cursor or movement of a touch on the touchscreen display (e.g., moving focus from one button to another using tab keys or arrow keys); in these embodiments, the focus selector moves based on the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user-controlled user interface element (or a touch on the touchscreen display) that transmits the user's expected interaction with the user interface (e.g., by indicating to the device the user interface element with which the user expects to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of the focus selector (e.g., a cursor, touch, or selection box) above the corresponding button will indicate to the user that they expect to activate the corresponding button (rather than other user interface elements shown on the device's display).

[0148] As used in the specification and claims, the term "characteristic strength" of a contact refers to a characteristic of the contact based on one or more strengths of the contact. In some embodiments, the characteristic strength is based on multiple strength samples. The characteristic strength may optionally be based on a predefined number of strength samples or a set of strength samples collected over a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted, before or after contact begins to move, before contact ends, before or after contact strength is detected to increase, and / or before or after contact strength is detected to decrease). The characteristic strength of the contact may optionally be based on one or more of the following: the maximum value of the contact strength, the mean value of the contact strength, the average value of the contact strength, the value at the top 10% of the contact strength, the half maximum value of the contact strength, or the 90% maximum value of the contact strength, etc. In some embodiments, the duration of the contact is used when determining the characteristic strength (e.g., when the characteristic strength is the average value of the contact strength over time). In some implementations, the characteristic intensity is compared to a set of one or more intensity thresholds to determine whether a user has performed an action. For example, the set of one or more intensity thresholds may optionally include a first intensity threshold and a second intensity threshold. In this example, contact with a characteristic intensity not exceeding the first threshold results in a first action, contact with a characteristic intensity exceeding the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a characteristic intensity exceeding the second threshold results in a third action. In some implementations, a comparison between the characteristic intensity and one or more thresholds is used to determine whether one or more actions should be performed (e.g., whether to perform the corresponding action or abandon performing the corresponding action) rather than to determine whether to perform the first action or the second action.

[0149] In some embodiments described herein, one or more operations are performed in response to detecting a gesture including a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the intensity of the contact (or multiple contacts) increases to above a press input intensity threshold. In some embodiments, the corresponding operation is performed in response to detecting that the intensity of the corresponding contact increases to above a press input intensity threshold (e.g., a "downward stroke" of the corresponding press input). In some embodiments, the press input includes the intensity of the corresponding contact increasing to above a press input intensity threshold and the intensity of the contact subsequently decreasing to below the press input intensity threshold, and the corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the press input threshold (e.g., an "upward stroke" of the corresponding press input).

[0150] In some implementations, the device employs intensity hysteresis to avoid unintended inputs sometimes referred to as "jitter," wherein the device defines or selects a hysteresis intensity threshold that has a predefined relationship with a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the press input intensity threshold). Therefore, in some implementations, a press input includes an increase in the intensity of the corresponding contact above the press input intensity threshold and a subsequent decrease in the intensity of that contact below the hysteresis intensity threshold corresponding to the press input intensity threshold, and an operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases below the hysteresis intensity threshold (e.g., the "upstroke" of the corresponding press input). Similarly, in some implementations, a press input is detected only when the device detects that the intensity of the contact increases from an intensity equal to or below a hysteresis intensity threshold to an intensity equal to or above a press input intensity threshold and optionally the intensity of the contact subsequently decreases to an intensity equal to or below the hysteresis intensity, and corresponding operations are performed in response to the detection of a press input (e.g., depending on the environment, the intensity of the contact increases or decreases).

[0151] For ease of explanation, optionally, the description of an operation triggered in response to a press input associated with a press input strength threshold or in response to a gesture including a press input may be provided in response to detecting any of the following conditions: the contact strength increases to above the press input strength threshold, the contact strength increases from below a hysteresis strength threshold to above the press input strength threshold, the contact strength decreases to below the press input strength threshold, and / or the contact strength decreases to below the hysteresis strength threshold corresponding to the press input strength threshold. Additionally, in the example where the operation is described as being performed in response to detecting a decrease in contact strength below the press input strength threshold, the operation may optionally be performed in response to detecting a decrease in contact strength below a hysteresis strength threshold corresponding to and less than the press input strength threshold.

[0152] Figure 5A A block diagram illustrating an exemplary architecture for device 500 according to some embodiments of this disclosure is shown. Figure 5AIn this implementation, media content or other content may optionally be received by device 500 via network interface 502, which may optionally be a wireless or wired connection. One or more processors 516 may optionally execute any number of programs stored in memory 506 or a storage device, the programs optionally including instructions for performing one or more of the methods and / or processes described herein (e.g., methods 700 and / or 900). A computer-readable storage medium may be any medium that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include, but is not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical discs based on CD, DVD, or Blu-ray technology, and persistent solid-state storage such as flash memory, solid-state drives, etc. The personal electronic device 500 is not limited to the components and configurations of FIG. 5, but may include other components or additional components in a variety of configurations.

[0153] Furthermore, in a method described herein where one or more steps depend on the satisfaction of one or more conditions, it should be understood that the method may be repeated in multiple repetitions such that, during the repetitions, all conditions determining the steps in the method are satisfied in different repetitions of the method. For example, if the method requires performing a first step (if the conditions are satisfied) and a second step (if the conditions are not satisfied), those skilled in the art will know that the stated steps are repeated until both conditions are satisfied and not satisfied (in no particular order). Thus, a method described as having one or more steps depending on the satisfaction of one or more conditions can be rewritten as a method that repeats until each condition described in the method is satisfied. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing discretionary operations based on the satisfaction of the corresponding one or more conditions, and thus to determine whether possible conditions have been satisfied without explicitly repeating the steps of the method until all conditions determining the steps in the method are satisfied. Those skilled in the art will also understand that, similar to a method having discretionary steps, a system or computer-readable storage medium may repeat the steps of the method multiple times as needed to ensure that all discretionary steps have been performed.

[0154] As used herein, the term "power indication" refers to the ability to indicate power in devices 100, 300, and / or 500 (…). Figure 1A Figure 3 and Figures 5A to 5B A user-interactive graphical user interface object displayed on a screen. For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) may each constitute a functional representation.

[0155] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some specific implementations that include a cursor or other position marker, the cursor acts as a "focus selector," such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the cursor is positioned on a touch-sensitive surface (e.g., a...). Figure 3A The touchpad 355 or Figure 4B When an input (e.g., a press input) is detected on the touch-sensitive surface 451 of the display, the specific user interface element is adjusted according to the detected input. This applies to touchscreen displays (e.g., those capable of direct interaction with user interface elements on a touchscreen display) that enable direct interaction with user interface elements on the touchscreen display. Figure 1A The touch-sensitive display system 112 or Figure 4A In some embodiments of the touchscreen 112, a touch detected on the touchscreen acts as a "focus selector," such that when input (e.g., a press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, that particular user interface element is adjusted according to the detected input. In some embodiments, focus moves from one area of ​​the user interface to another without corresponding movement of the cursor or movement of a touch on the touchscreen display (e.g., moving focus from one button to another using tab keys or arrow keys); in these embodiments, the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user-controlled user interface element (or a touch on the touchscreen display) that conveys the user's expected interaction with the user interface (e.g., by indicating to the device the elements of the user interface that the user expects to interact with). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of the focus selector (e.g., a cursor, touch, or selection box) above the corresponding button will indicate to the user that they expect to activate the corresponding button (rather than other user interface elements shown on the device's display).

[0156] As used in the specification and claims, the term "characteristic strength" of a contact refers to a characteristic of the contact based on one or more strengths of the contact. In some embodiments, the characteristic strength is based on multiple strength samples. The characteristic strength may optionally be based on a predefined number of strength samples or a set of strength samples collected over a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted, before or after contact begins to move, before contact ends, before or after contact strength is detected to increase, and / or before or after contact strength is detected to decrease). The characteristic strength of the contact may optionally be based on one or more of the following: the maximum value of the contact strength, the mean value of the contact strength, the average value of the contact strength, the value at the top 10% of the contact strength, the half maximum value of the contact strength, or the 90% maximum value of the contact strength, etc. In some embodiments, the duration of the contact is used when determining the characteristic strength (e.g., when the characteristic strength is the average value of the contact strength over time). In some implementations, the characteristic intensity is compared to a set of one or more intensity thresholds to determine whether a user has performed an action. For example, the set of one or more intensity thresholds may optionally include a first intensity threshold and a second intensity threshold. In this example, contact with a characteristic intensity not exceeding the first threshold results in a first action, contact with a characteristic intensity exceeding the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a characteristic intensity exceeding the second threshold results in a third action. In some implementations, a comparison between the characteristic intensity and one or more thresholds is used to determine whether one or more actions should be performed (e.g., whether to perform the corresponding action or abandon performing the corresponding action) rather than to determine whether to perform the first action or the second action.

[0157] Figure 5C An example is shown using multiple intensity sensors 524A to 524D to detect multiple contacts 552A to 552E on a touch-sensitive display screen 504. Figure 5C It also includes an intensity map, which shows the current intensity measurement of intensity sensors 524A to 524D relative to intensity units. In this example, the intensity measurements of intensity sensors 524A and 524D are both 9 intensity units, and the intensity measurements of intensity sensors 524B and 524C are both 7 intensity units. In some embodiments, the cumulative intensity is the sum of the intensity measurements of the multiple intensity sensors 524A to 524D, which is 32 intensity units in this example. In some embodiments, each contact is assigned a corresponding intensity, i.e., a portion of the cumulative intensity. Figure 5DAn example is illustrated of assigning cumulative intensity to contacts 552A through 552E based on their distance from the center of force 554. In this example, each of contacts 552A, 552B, and 552E is assigned a contact intensity of 8 intensity units of the cumulative intensity, and each of contacts 552C and 552D is assigned a contact intensity of 4 intensity units of the cumulative intensity. More generally, in some specific implementations, each contact j is assigned a corresponding intensity Ij according to a predefined mathematical function Ij = A·(Dj / ΣDi), which is a portion of the cumulative intensity A, where Dj is the distance of the corresponding contact j from the center of force, and ΣDi is the sum of the distances of all corresponding contacts (e.g., i=1 to the last) from the center of force. Reference can be performed using electronic devices similar to or equivalent to devices 100, 300, or 500. Figures 5C to 5D The described operation. In some embodiments, the characteristic intensity of the contact is based on one or more intensities of the contact. In some embodiments, an intensity sensor is used to determine the intensity of a single characteristic (e.g., the single characteristic intensity of a single contact). It should be noted that the intensity map is not part of the displayed user interface, but is included within... Figures 5C to 5D This is intended to assist readers.

[0158] In some implementations, a portion of the gesture is identified for determining the characteristic intensity. For example, a touch-sensitive surface may optionally receive a series of swipe contacts that transition from a starting position to an ending position, where the contact intensity increases. In this example, the characteristic intensity of the contact at the ending position may optionally be based only on a portion of the series of swipe contacts, rather than the entire swipe contact (e.g., only the portion of the swipe contact at the ending position). In some implementations, a smoothing algorithm may optionally be applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm may optionally include one or more of the following: unweighted moving average smoothing algorithm, triangular smoothing algorithm, median filter smoothing algorithm, and / or exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow spikes or dips in the intensity of the swipe contact to achieve the purpose of determining the characteristic intensity.

[0159] Optionally, the contact intensity on a touch-sensitive surface can be characterized relative to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device performs an operation typically associated with clicking a button on a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device performs an operation different from the operation typically associated with clicking a button on a physical mouse or touchpad. In some embodiments, when a contact with an intensity lower than the light press intensity threshold (e.g., and higher than the nominal contact detection intensity threshold, where contacts lower than the nominal contact detection intensity threshold are no longer detected) is detected, the device will move the focus selector based on the movement of the contact on the touch-sensitive surface without performing the operation associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise stated, these intensity thresholds are consistent across different groups of user interface figures.

[0160] An increase in contact intensity from below a light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in contact intensity from below a deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in contact intensity from below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting a contact on the touch surface. A decrease in contact intensity from above a contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting a contact being lifted off the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0161] In some embodiments described herein, one or more operations are performed in response to detecting a gesture including a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the intensity of the contact (or multiple contacts) increases to above a press input intensity threshold. In some embodiments, the corresponding operation is performed in response to detecting that the intensity of the corresponding contact increases to above a press input intensity threshold (e.g., a "downward stroke" of the corresponding press input). In some embodiments, the press input includes the intensity of the corresponding contact increasing to above a press input intensity threshold and the intensity of the contact subsequently decreasing to below the press input intensity threshold, and the corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the press input threshold (e.g., an "upward stroke" of the corresponding press input).

[0162] Figures 5E to 5HAn example of gesture detection is given, where the gesture includes the intensity of contact 562 ranging from less than... Figure 5E The light press intensity threshold (e.g., "IT") L The intensity of ) increases to higher than Figure 5H The deep press intensity threshold (e.g., "IT") D The intensity of the press input corresponds to the strength of the press. On the user interface 570, which includes application icons 572A to 572D displayed in the predefined area 574, when a cursor 576 is displayed above the application icon 572B corresponding to application 2, a gesture performed using contact 562 is detected on the touch-sensitive surface 560. In some embodiments, the gesture is detected on the touch-sensitive display 504. The intensity sensor detects the intensity of the contact on the touch-sensitive surface 560. The device determines the intensity of the contact 562 within a deep press intensity threshold (e.g., "IT"). D The intensity reaches its peak at 562. Contact 562 is maintained on the touch-sensitive surface 560. In response to a detected gesture, and based on the intensity increasing to a deep press intensity threshold during the gesture (e.g., "IT"), the intensity is increased to a peak value. D The above contact 562 displays the scaled representation 578A to 578C (e.g., thumbnails) of the document recently opened for application 2, such as... Figures 5F to 5H As shown. In some embodiments, the intensity is the characteristic intensity of the contact compared to one or more intensity thresholds. It should be noted that the intensity map for contact 562 is not part of the displayed user interface, but is included in... Figures 5E to 5H This is intended to assist readers.

[0163] In some implementations, the display of icons 578A through 578C includes animation. For example, icon 578A might initially appear near application icon 572B, such as... Figure 5F As shown, as the animation progresses, icon 578A moves upwards, and icon 578B appears near application icon 572B, as shown. Figure 5G As shown in the diagram. Then, representation 578A moves upward, 578B moves upward toward representation 578A, and representation 578C is displayed near application icon 572B, as shown. Figure 5H As shown in the diagram. This indicates that 578A to 578C form an array above icon 572B. In some implementations, the animation progresses according to the intensity of contact 562, such as... Figures 5F to 5G As shown, where 578A to 578C appear and increase with the intensity of contact 562 to a deeper pressing strength threshold (e.g., "IT") D "" and move upwards. In some implementations, the intensity upon which the animation progresses is based is the characteristic intensity of the contact. Reference can be performed using electronic equipment similar to or equivalent to devices 100, 300, or 500. Figures 5E to 5H The described operation.

[0164] In some implementations, the device employs intensity hysteresis to avoid unintended inputs sometimes referred to as "jitter," wherein the device defines or selects a hysteresis intensity threshold that has a predefined relationship with a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the press input intensity threshold). Therefore, in some implementations, a press input includes an increase in the intensity of the corresponding contact above the press input intensity threshold and a subsequent decrease in the intensity of that contact below the hysteresis intensity threshold corresponding to the press input intensity threshold, and an operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases below the hysteresis intensity threshold (e.g., the "upstroke" of the corresponding press input). Similarly, in some implementations, a press input is detected only when the device detects that the intensity of the contact increases from an intensity equal to or below a hysteresis intensity threshold to an intensity equal to or above a press input intensity threshold and optionally the intensity of the contact subsequently decreases to an intensity equal to or below the hysteresis intensity, and corresponding operations are performed in response to the detection of a press input (e.g., depending on the environment, the intensity of the contact increases or decreases).

[0165] For ease of explanation, optionally, the description of an operation triggered in response to a press input associated with a press input strength threshold or in response to a gesture including a press input may be provided in response to detecting any of the following conditions: the contact strength increases to above the press input strength threshold, the contact strength increases from below a hysteresis strength threshold to above the press input strength threshold, the contact strength decreases to below the press input strength threshold, and / or the contact strength decreases to below the hysteresis strength threshold corresponding to the press input strength threshold. Additionally, in the example where the operation is described as being performed in response to detecting a decrease in contact strength below the press input strength threshold, the operation may optionally be performed in response to detecting a decrease in contact strength below a hysteresis strength threshold corresponding to and less than the press input strength threshold.

[0166] As used herein, "installed application" refers to a software application that has been downloaded to an electronic device (e.g., device 100, 300, and / or 500) and is ready to be launched on the device (e.g., become open). In some implementations, the downloaded application becomes an installed application using an installer that extracts program portions from the downloaded software package and integrates the extracted portions with the computer system's operating system.

[0167] As used herein, the terms "open application" or "running application" refer to a software application that maintains state information (e.g., as part of device / global internal state 157 and / or application internal state 192). An open or running application can optionally be any of the following types of applications: • Active app, which is currently displayed on the screen of the device that is using the app; • Background applications (or background processes) that are not currently displayed but whose one or more processes are being handled by one or more processors; and • Suspended or hibernating applications that are not currently running but have state information stored in memory (either volatile or non-volatile) that can be used to resume the application's execution.

[0168] As used herein, the term "closed application" refers to a software application that does not retain state information (e.g., the state information of a closed application is not stored in the device's memory). Therefore, closing an application includes: stopping and / or removing the application's process and removing the application's state information from the device's memory. Generally, opening a second application while the first application is running does not close the first application. When the second application is displayed and the first application stops displaying, the first application becomes a background application.

[0169] Now let’s turn our attention to the implementation of the user interface (“UI”) on electronic devices (such as device 100, device 300 or device 500) and the associated processes.

[0170] Figure 5I An example of an exemplary device for implementing techniques for pre-field paths is provided. Figures 6A to 6J An exemplary user interface for on-site paths is illustrated according to some implementation schemes. Figure 7 This is a flowchart illustrating a method for on-site paths based on some implementation schemes. Figures 6A to 6J The user interface in this document is used to illustrate the processes described below, including... Figure 7 The process in.

[0171] The process described below describes various techniques for making user interfaces and / or human-computer interaction more efficient (e.g., by helping users provide input quickly and easily and preventing user errors when operating the device). These techniques sometimes reduce the amount of input required for users (e.g., people and / or users) to perform actions, provide users with clear and / or meaningful feedback (e.g., visual, acoustic, and / or haptic feedback) so that users know what is happening or what is expected, provide additional information and controls without cluttering the user interface, and / or perform certain actions without further input from the user. Because users can use the device more quickly and easily, these techniques sometimes extend battery life and / or reduce the device's power consumption.

[0172] In a method described herein where one or more steps depend on the satisfaction of one or more conditions, it should be understood that the described method may be repeated in multiple repetitions such that, during the repetitions, all the conditions determining the steps in the method have been satisfied in different repetitions of the method. For example, if the method requires performing a first step (if the conditions are satisfied) and a second step (if the conditions are not satisfied), it should be understood that these steps are repeated until both conditions are satisfied and not satisfied (in no particular order). Thus, a method described as having one or more steps depending on the satisfaction of one or more conditions can be rewritten as a method that repeats until each of the conditions described in the method has been satisfied. However, system or computer-readable medium claims do not require such multiple repetitions, wherein the system or computer-readable medium contains instructions for performing conditional operations that require the satisfaction of one or more conditions prior to the operation. Those skilled in the art will also understand that, similar to a method having conditional steps, a system or computer-readable storage medium may repeat the steps of the method multiple times as needed to ensure that all conditional steps have been performed.

[0173] The terminology used in the description of the various implementation schemes is for the purpose of describing a particular implementation scheme only and is not intended to be limiting.

[0174] The following describes user interfaces for electronic devices and associated processes for using these devices. In some embodiments, the device may be a desktop computer with a touch-sensitive surface (e.g., a touchscreen display and / or touchpad). In other embodiments, the device is a portable, mobile, and / or mobile electronic device (e.g., a processor, smartphone, smartwatch, tablet, fitness tracker, laptop, head-mounted display (HMD) device, public facilities, vehicles, media devices, smart speakers, smart displays, robots, televisions, and / or personal computing devices).

[0175] In some embodiments, the electronic device is a computer system that communicates with the display component (e.g., via wireless or wired communication). The display component may be integrated into the computer system or may be separate from the computer system. Additionally, the display component may be configured to provide visual output to a display (e.g., a liquid crystal display, an OLED display, or a CRT display). As used herein, "display" content includes content that causes a display (e.g., video data rendered or decoded by a display controller) by sending data (e.g., image data or video data) to an integrated or external display component via a wired or wireless connection to visually generate content. In some embodiments, visual output is any output that can be perceived by the human eye, including but not limited to images, videos, graphics, charts, and other graphical representations of data.

[0176] In some implementations, the electronic device is a computer system that communicates with the audio generation component (e.g., via wireless or wired communication). The audio generation component may be integrated into the computer system or may be separate from it. Additionally, the audio generation component may be configured to provide audio output. Examples of audio generation components include speakers, home theater systems, soundbars, headphones, earphones, in-ear headphones, television speakers, augmented reality headset speakers, audio jacks, optical audio outputs, Bluetooth audio outputs, and / or HDMI audio outputs. In some implementations, the audio output is any output perceptible to the human ear, including but not limited to sound waves, music, speech, and / or other audible representations of data.

[0177] In the following discussion, an electronic device including specific input and output devices is described. However, it should be understood that the electronic device may optionally include one or more other input and / or output devices, such as physical user interface devices (e.g., physical keyboard, mouse, and / or joystick).

[0178] Figure 5I An example system 100 for implementing the techniques described herein is illustrated. System 100 is executable. Figure 7 Any of the methods described herein (e.g., method 700) and / or a portion thereof.

[0179] exist Figure 5I In this system 100, various components are included, such as processor 592, RF circuitry 105, memory 107, sensors 586 (e.g., image sensors, orientation sensors, position sensors, heart rate monitors, temperature sensors), input components 588 (e.g., cameras (e.g., periscope cameras, telephoto cameras, wide-angle cameras, and / or ultra-wide-angle cameras), depth sensors, microphones, touch-sensitive surfaces, hardware input mechanisms, and / or rotatable input mechanisms), mobility components (e.g., actuators (e.g., pneumatic actuators, hydraulic actuators, and / or electric actuators), motors, wheels, movable bases, rotatable components, translational components, and / or rotatable bases), and output components 590 (e.g., speakers, display components, audio generation components, haptic output devices, displays, projectors, and / or touch-sensitive displays). These components optionally communicate via the system's communication bus 123. Although shown as separate components, in some specific implementations, various components may be combined and used as a single component; for example, a sensor may be an input component.

[0180] In some implementations, system 100 is a mobile and / or mobile device (e.g., a tablet, smartphone, laptop, head-mounted display (HMD) device, and / or smartwatch). In other implementations, system 100 is a desktop computer, embedded computer, and / or server.

[0181] In some embodiments, processor 592 includes one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some embodiments, memory 107 is one or more non-transitory computer-readable storage media (e.g., flash memory and / or random access memory) storing computer-readable instructions configured to be executed by processor 592 to perform the techniques described herein.

[0182] In some embodiments, RF circuit 105 includes circuitry for communicating with electronic devices and / or networks (e.g., the Internet, intranets, and / or wireless networks such as cellular networks and wireless local area networks (LANs)). In some embodiments, RF circuit 105 includes circuitry for using near-field communication and / or short-range communication such as Bluetooth. ® Circuits for communication, or ultra-wideband communication.

[0183] In some embodiments, display 121 includes one or more monitors, projectors, and / or screens. In some embodiments, display 121 includes a first display for displaying an image to a user's first eye and a second display for displaying an image to a user's second eye. In such embodiments, corresponding images may be displayed simultaneously on the first and second displays. Optionally, the corresponding images include representations of the same virtual objects and / or the same physical objects viewed from different viewpoints, thereby creating a parallax effect that provides the user with the illusion that objects have depth on the display. In some embodiments, display 121 is a single display. In such embodiments, for each of the user's eyes, corresponding images are simultaneously displayed in a first and a second area of ​​a single display. Optionally, the corresponding images include representations of the same virtual objects and / or the same physical objects viewed from different viewpoints, thereby creating a parallax effect that provides the user with the illusion that objects have depth on a single display.

[0184] In some embodiments, system 100 includes one or more touch-sensitive surfaces 115 for receiving user input such as tap and swipe inputs. In some embodiments, display 121 and touch-sensitive surfaces 115 form a touch-sensitive display.

[0185] In some embodiments, sensor 586 includes sensors for detecting various conditions. In some embodiments, sensor 586 includes orientation sensors (e.g., orientation sensor 111) for detecting the orientation and / or movement of platform 150. For example, system 100 uses orientation sensors to track changes in the position and / or orientation (sometimes collectively referred to as positioning) of system 100, such as relative to physical objects in the physical environment. In some embodiments, sensor 586 includes one or more gyroscopes, one or more inertial measurement units, and / or one or more accelerometers. In some embodiments, sensor 586 includes a Global Positioning Sensor (GPS) for detecting the GPS position of platform 150. In some embodiments, sensor 586 includes a radar system, a lidar system, a sonar system, an image sensor (e.g., image sensor 109, a visible light image sensor, and / or an infrared sensor), a depth sensor, a rangefinder, and / or a motion detector. In some embodiments, sensor 586 includes sensors located within internal portions of system 100 and / or sensors located externally to system 100. In some embodiments, system 100 uses sensors 586 (e.g., internal sensors) to detect the presence and / or state (e.g., location and / or orientation) of passengers within the interior of system 100. In some embodiments, system 100 uses sensors 586 (e.g., external sensors) to detect the presence and / or state of objects outside system 100. In some embodiments, system 100 uses sensors 586 to receive user input, such as gestures and / or other air gestures. In some embodiments, system 100 uses sensors 586 to detect the location and / or orientation of system 100 in the physical environment. In some embodiments, system 100 uses sensors 586 to navigate system 100 along a planned route, around obstacles, and / or to a destination location. In some embodiments, sensors 586 include one or more sensors for identifying and / or authenticating users of system 100, such as fingerprint sensors and / or facial recognition sensors.

[0186] In some embodiments, the image sensor includes one or more visible light image sensors, such as charge-coupled device (CCD) sensors and / or complementary metal-oxide-semiconductor (CMOS) sensors, capable of acquiring images of a physical object. In some embodiments, the image sensor includes one or more infrared (IR) sensors, such as passive or active IR sensors, for detecting infrared light. For example, an active IR sensor may include an IR emitter, such as an IR point emitter, for emitting infrared light. In some embodiments, the image sensor includes one or more cameras configured to capture movement of a physical object. In some embodiments, the image sensor includes one or more depth sensors configured to detect the distance of a physical object from system 100. In some embodiments, system 100 uses a combination of CCD sensors, cameras, and depth sensors to detect the physical environment surrounding system 100. In some embodiments, the image sensor includes a first image sensor and a second image sensor different from the first image sensor. In some embodiments, system 100 uses the image sensor to receive user input, such as gestures and / or other air gestures. In some embodiments, system 100 uses the image sensor to detect the position and / or orientation of system 100 in the physical environment.

[0187] In some implementations, system 100 uses an orientation sensor to detect the orientation and / or movement of system 100. For example, system 100 may use an orientation sensor to track changes in the position and / or orientation of system 100, such as relative to a physical object in the physical environment. In some implementations, the orientation sensor includes one or more gyroscopes, one or more inertial measurement units, and / or one or more accelerometers.

[0188] In some embodiments, system 100 uses a microphone to detect sound from one or more users and / or the physical environment of those users. In some embodiments, the microphone includes a microphone array (comprising multiple microphones) that optionally operates in cooperation, such as to identify ambient noise or to locate sound sources in a space of the physical environment (e.g., inside and / or outside system 100).

[0189] In some embodiments, input device 588 includes one or more mechanical and / or electrical devices for detecting input, such as buttons, sliders, knobs, switches, remote controls, joysticks, touch-sensitive surfaces, keypads, microphones, and / or cameras. In some embodiments, input device 588 includes one or more input devices internal to system 100. In some embodiments, input device 588 includes one or more input devices (e.g., touch-sensitive surfaces and / or keypads) external to system 100.

[0190] In some embodiments, output device 590 includes one or more devices such as a display, monitor, projector, speaker, lamp, and / or haptic output device. In some embodiments, output device 590 includes one or more external output devices such as an external display screen, external lamp, and / or external speaker. In some embodiments, output device 590 includes one or more internal output devices such as an internal display screen, internal lamp, and / or internal speaker.

[0191] In some embodiments, environmental controls 584 include mechanical and / or electrical systems for monitoring and / or controlling the condition of internal parts (e.g., compartments) of system 100. In some embodiments, environmental controls 584 include fans, heaters, air conditioners, and / or thermostats for controlling temperature and / or airflow within the internal parts of system 100.

[0192] In some embodiments, the mobility component includes mechanical and / or electrical components that enable and / or assist the platform in moving. In some embodiments, the mobility system 582 includes a power system, drivetrain, motor (e.g., an electric motor), engine, power source (e.g., a battery), transmission, suspension system, speed control system, and / or steering system. In some embodiments, one or more elements of the mobility component are configured for autonomous or manual control (e.g., via system 100 and / or input device 588).

[0193] In some implementations, system 100 performs monetary transactions with or without another computer system. For example, system 100, or another computer system associated with and / or communicating with system 100 (e.g., via a user account described below), is associated with a user's payment account, such as a credit card account or checking account. To complete the transaction, system 100 may send a key to the entity purchasing goods and / or services from it, enabling the entity to charge the payment account for the transaction. Alternatively, system 100 may store encrypted payment account information and send that information to the entity purchasing goods and / or services from it to complete the transaction.

[0194] System 100 may optionally engage in other transactions with other systems, computers, and / or devices. For example, system 100 may engage in transactions to unlock another system, computer, and / or device, and / or be unlocked by another system, computer, and / or device. Unlocking transactions may optionally include using, for example, RF circuitry 105 to transmit and / or receive one or more secure cryptographic keys.

[0195] In some implementations, system 100 is capable of communicating with other computer systems and / or electronic devices. For example, system 100 may use RF circuitry 105 to access a network connection that enables the transmission of data between systems for communication purposes. Example communication sessions include telephone calls, emails, SMS messages, and / or video conferencing communication sessions.

[0196] In some implementations, a video conferencing communication session includes the transmission and / or reception of video and / or audio data between systems participating in the video conferencing communication session (including system 100). In some implementations, system 100 uses sensor 586 to capture video and / or audio content, which is then transmitted to other systems in the video conferencing communication session using RF circuitry 105. In some implementations, system 100 uses RF circuitry 105 to receive video and / or audio from other systems in the video conferencing communication session and uses output components 590 (such as display 121 and / or speakers) to present the video and / or audio. In some implementations, the transmission of audio and / or video between systems is near real-time, such as presenting it to other systems with a delay of less than 0.1 seconds, 0.5 seconds, 1 second, or 3 seconds from the time the corresponding portion of the audio and / or video was captured.

[0197] In some embodiments, system 100 uses output component 590 to generate haptic (e.g., touch) output. In some embodiments, output component 590 generates haptic output by displacing a movable mass block relative to a neutral position. In some embodiments, the haptic output is inherently periodic, optionally including the frequency and / or amplitude of two-dimensional or three-dimensional movement. In some embodiments, system 100 generates a variety of different haptic outputs that differ in the frequency, amplitude, and / or duration / number of cycles of the included movement. In some embodiments, the haptic output pattern includes a start buffer and / or an end buffer during which the movable mass block gradually accelerates and / or decelerates at the beginning and / or end of the haptic output, respectively.

[0198] In some implementations, the haptic output has a corresponding characteristic frequency that affects the "pitch" of the haptic sensation perceived by the user. For example, a higher frequency corresponds to faster movement of the movable mass, while a lower frequency corresponds to slower movement of the movable mass. In some implementations, the haptic output has a corresponding characteristic amplitude that affects the "intensity" of the haptic sensation perceived by the user. For example, a higher amplitude corresponds to movement of the movable mass over a greater distance, while a lower amplitude corresponds to movement of the movable mass over a smaller distance. In some implementations, the "pitch" and / or "intensity" of the haptic output varies over time.

[0199] In some embodiments, the tactile output is distinct from the movement of system 100. For example, system 100 may include a tactile output device that moves a movable mass to generate the tactile output, and may include other moving parts that control the movement of system 100, such as motors, wheels, axles, control arms, and / or brakes. Although in some cases the movement and / or cessation of movement of system 100 generates vibrations and / or other physical sensations, these vibrations and / or other physical sensations are distinct from the tactile output. In some embodiments, system 100 generates a tactile output independent of the movement of system 100. For example, system 100 may generate a tactile output without causing system 100 to accelerate, decelerate, and / or move to a new position.

[0200] In some embodiments, system 100 detects gesture input made by a user. In some embodiments, gesture input includes touch gestures and / or air gestures, as described herein. In some embodiments, touch-sensitive surface 115 identifies touch gestures based on contact patterns (e.g., varying intensities, timings, and / or movements of an object touching or nearly touching touch-sensitive surface 115). Thus, touch-sensitive surface 115 detects gestures by detecting the corresponding contact patterns. For example, detecting a finger press event and then detecting a finger lift-off (e.g., lift-off) event at the same location as the finger press event (e.g., at the location of a user interface element), which could correspond to detecting a tap gesture on a user interface element. As another example, detecting a finger press event, then detecting movement of the contact, and subsequently detecting a finger lift-off (e.g., lift-off) event, which could correspond to detecting a swipe gesture. Additional and / or alternative touch gestures are possible.

[0201] In some embodiments, an air gesture is a gesture performed by a user without touching the input component 588. In some embodiments, an air gesture is based on the detected movement of a portion of the user (e.g., hand, fingers, and / or body) in the air. In some embodiments, an air gesture includes the movement of that portion of the user relative to a reference object. Example reference objects include the distance of the user's hand relative to a physical object (such as the ground), the angle of the user's arm relative to a physical object, and / or the movement of a first portion of the user (e.g., hand or fingers) relative to a second portion of the user (e.g., shoulder, another hand, or another finger). In some embodiments, detecting an air gesture includes detecting the absolute movement of that portion of the user, such as a tapping gesture including the hand moving in a predetermined pose at a predetermined amount and / or speed, or a shaking gesture including a portion of the user at a predetermined speed or amount of rotation.

[0202] In some implementations, detecting one or more inputs includes detecting the user's speech. In some implementations, system 100 uses one or more microphones of input component 588 to detect when the user utters one or more words. In some implementations, system 100 parses information and / or communicates information to one or more other systems to determine the content of the user's speech, including identifying words and / or obtaining a semantic understanding of those words. For example, system processor 592 may be configured to perform natural language processing to detect one or more words and / or determine the possible meaning of one or more words in a sequence spoken by the user. Additionally or alternatively, in some implementations, system 100 determines the meaning of one or more words in a sequence spoken based on the context determined by system 100.

[0203] In some embodiments, system 100 outputs spatial audio via output component 590. In some embodiments, spatial audio is output at a specific location. For example, system 100 may play a notification ringtone having one or more characteristics that cause the notification ringtone to be generated as if it were emitted from a first location relative to the user's current viewpoint (e.g., "spatializing" and / or "spatializing" includes modifying the audio in amplitude, filtering, and / or delaying it to provide the user with perceived spatial quality).

[0204] In some embodiments, system 100 presents visual and / or audio feedback indicating the user's current position relative to another user's current viewpoint, thereby informing the other user of the user's updated position. In some embodiments, playing audio corresponding to the user includes altering one or more characteristics of audio acquired from another computer system to simulate the effect of placing an audio source generating the audio playback within the user's position (such as a position the user moves to, appears in, and / or is assigned to within a 3D environment). In some embodiments, the relative magnitude of the audio at one or more frequencies and / or frequency groups is altered, one or more filters are applied to the audio (e.g., directional audio filters), and / or the magnitude of the audio provided via one or more channels is altered (e.g., increased or decreased) to produce the perceived effect of a physical audio source. In some embodiments, the simulated position of the audio source relative to the floor of the 3D environment is matched to the height of the participant's head, or one or more predetermined heights relative to the floor of the 3D environment, when the user's position is determined to correspond to a second position different from a first position and one or more first criteria are met, and this feedback includes generating audio as if it were emanating from the second position.

[0205] In some embodiments, system 100 communicates with one or more accessory devices. In some embodiments, one or more accessory devices are integrated with system 100. In some embodiments, one or more accessory devices are external to system 100. In some embodiments, system 100 communicates with accessory devices using RF circuitry 105 and / or using wired connections. In some embodiments, system 100 controls the operation of accessory devices such as doors, windows, locks, speakers, lights, and / or cameras. For example, system 100 can control the operation of a motorized door of system 100. As another example, system 100 can control the operation of a motorized window included in system 100. In some embodiments, accessory devices used as input devices, such as remote controls and / or other computer systems (e.g., smartphones, media players, tablets, computers, and / or wearable devices), control the operation of system 100. For example, a wearable device (e.g., a smartwatch) serves as a key to initiate the operation of the actuation system of system 100. In some implementations, system 100 acts as an input device to control the operation of another system, device, and / or computer, such as platform 100 acting as a key to initiate the operation of an actuation system of a platform associated with another system, device, and / or computer.

[0206] In some implementations, the digital assistant assists users in performing various functions using System 100. For example, the digital assistant may provide weather updates, set alarms, and perform searches locally and / or using a network connection (e.g., the Internet) via a natural language interface. In some implementations, the digital assistant accepts requests that are at least partially in the form of natural language commands, narrations, requests, statements, and / or inquiries. In some implementations, the user uses the digital assistant to request informational answers and / or the execution of tasks. For example, in response to receiving the question “What is the current temperature?”, the digital assistant answers “30 degrees.” As another example, in response to receiving a request to perform a task, such as “Please invite my family to dinner tomorrow,” the digital assistant may confirm the request by playing spoken words (such as “Okay, right away”) and then, on behalf of the user, send the requested calendar invitations to each family member listed in the user’s contact list. In some implementations, the digital assistant engages in a continuous dialogue with the user during the execution of a user-requested task, which involves multiple exchanges of information over a period of time. Other ways of interacting with the digital assistant may involve requesting the execution of tasks and / or requesting information. For example, the digital assistant may respond to the user in other forms, such as displayed alerts, text, video, animation, music, etc. In some embodiments, the digital assistant includes a client-side portion executing on system 100 and a server-side portion executing on a server communicating with system 100. The client-side portion may communicate with the server via a network connection using RF circuitry 105. For example, the client-side portion may provide client-side functionality, input and / or output processing, and / or communication with the server. In some embodiments, the server-side portion provides server-side functionality for any number of client-side portions across multiple systems.

[0207] In some implementations, system 100 is associated with one or more user accounts. In some implementations, system 100 stores and / or encrypts user data, including files, settings, and / or preferences associated with a specific user account. In some implementations, user accounts are password protected, and system 100 requires user authentication before accessing user data associated with an account. In some implementations, user accounts are associated with other systems, devices, and / or servers. In some implementations, associating a user account with multiple systems enables these systems to access, update, and / or synchronize user data associated with that user account. For example, systems associated with a user account may access purchased media content, contact lists, communication sessions, payment information, stored passwords, and other user data. Therefore, in some implementations, user accounts provide security mechanisms for a customized user experience.

[0208] Users interact with electronic devices in a variety of different ways, including through transportation. In some implementations, the electronic device navigates or travels from a starting point to a destination using a predetermined path included in a map and / or a path determined (or partially determined) by the electronic device in areas where predetermined paths are unavailable. The implementations described below provide ways for the electronic device to navigate or travel on paths other than predetermined paths (e.g., paths included in a map and / or paths designated by lane markings).

[0209] Figures 6A to 6J Exemplary methods by which electronic devices according to some embodiments of this disclosure navigate or travel on predetermined paths and / or pre-defined paths are illustrated. The embodiments in these figures are used to illustrate the processes described below, including references to… Figure 7 The process described. Although Figures 6A to 6J Examples of electronic devices capable of performing the following references are provided. Figure 7 Various examples of the methods described are provided, but it should be understood that these examples are not intended to be limiting, and electronic devices can be used in accordance with the examples provided. Figures 6A to 6J For methods not explicitly described, please refer to the following text. Figure 7 One or more processes described.

[0210] Figure 6A A block diagram illustrating a first electronic device 500a operating in environment 600 is shown. In some embodiments, electronic device 500a communicates with (e.g., includes) a display component 504, a sensor 602, a position sensor 604, a processor 606, a controller 608, a memory 610, and a transceiver 612.

[0211] In some embodiments, sensor 602 includes a camera, a distance sensor, an accelerometer, a speedometer, and / or a thermometer. In some embodiments, electronic device 500a uses sensor 602 to sense data 614 corresponding to the environment 600 of the first device 500a. As described in more detail below, for example, electronic device 500a uses data 614 to generate a pre-defined path for navigation in an area that does not include a predetermined path, based on various characteristics of the environment 600.

[0212] In some embodiments, the position sensor 604 is a GPS (Global Positioning Satellite) or other satellite-based navigation unit configured to sense the position 616 of the electronic device 500a. In some embodiments, the electronic device 500a uses the position sensor 604 to navigate along a predetermined path included in map information and to determine whether the predetermined path is available at the starting position 616 of the electronic device 500a.

[0213] In some embodiments, one or more controllers 608 control subsystems of electronic device 500a and / or other devices communicating with electronic device 500a, such as a second electronic device 500b and / or a vehicle communicating with electronic device 500a. For example, one or more controllers 608 control actuators of the vehicle, such as an engine or motor, brakes, steering, lights, windows, doors, climate control, and / or media playback. In some embodiments, navigation along a predetermined path and / or a nearby path includes autonomous driving and / or driver assistance functions controlled by controller 608. In some embodiments, navigation along a predetermined path and / or a nearby path includes presenting navigation guidance to the user when the user is driving the vehicle.

[0214] In some implementations, electronic device 500a uses memory 610 to store instructions for performing the methods disclosed herein and / or to store map or path information.

[0215] In some embodiments, electronic device 500a communicates with second electronic device 500b using transceiver 612. In some embodiments, electronic device 500a communicates with multiple second electronic devices 500b. For example, electronic device 500a communicates with vehicles in its environment 600, a map server, other devices associated with a user account of electronic device 500a, and / or electronic devices associated with the starting location 616 of electronic device 500a. In some embodiments, electronic device 500a obtains information from second electronic devices 500b for generating and / or selecting pre-existing paths, as described in more detail below.

[0216] Figure 6B An example map user interface 618 is illustrated, optionally displayed by electronic device 500a when navigating to a destination. In some embodiments, electronic device 500a displays map user interface 618 using display component 504 when navigating to a destination. In some embodiments, electronic device 500a omits displaying map user interface 618 when navigating to a destination. In some embodiments, additionally or alternatively, electronic device 500a outputs an audio user interface, which includes audio features... Figure 6B The map information corresponding to the map user interface 618 illustrated in the example. For example, in some embodiments, the electronic device 500a abandons the display of... Figure 6B The graphical user interface in, and Figure 6B The graphical user interface in the diagram represents the current position 622 of the electronic device relative to a predetermined path 620 and navigation path 624 in the environment of the electronic device.

[0217] In some implementations, the map user interface 618 represents a physical area including the starting location of the electronic device 500a. For example... Figure 6B As shown, the map user interface 618 includes indications 620 of a predetermined path included in map information accessible to the electronic device 500a, indications 622 of the starting position of the electronic device 500a, and indications 624 of the path the electronic device 500a is navigating. In some embodiments, the predetermined path includes paths included in public map data, paths marked with lane markings and / or road signs, such as streets, highways, toll roads, bridges, bicycle lanes, roads distinguished by their surface material and surrounding environment, hiking trails, tunnels, driving lanes, and / or walking paths. For example, in Figure 6B In this implementation, the path that electronic device 500a is navigating is a predetermined path, therefore the path indication 624 of electronic device 500a overrides the indication of the predetermined path. In some embodiments, the audio map user interface includes an audio description of map information (such as roads and landmarks in the area including the starting location of electronic device 500a) corresponding to the physical area including the starting location of electronic device 500a. In some embodiments, the audio map user interface includes an audio description of the path that electronic device 500a is navigating.

[0218] Figure 6CAn example map user interface 618, optionally displayed by electronic device 500a, is illustrated. This example map user interface includes locations that do not include predetermined paths. For example, the starting location of electronic device 500a does not include roads and / or highways or other map information used by electronic device 500a to determine navigation from the initial point to the destination. In some embodiments, when navigating to the destination and when electronic device 500a is in a location without predetermined paths or other map information, electronic device 500a generates a pre-existing path according to method 700. In some embodiments, the user may input a path that electronic device 500a can use for navigation (e.g., drawing a path from one point to another on the user interface). In some embodiments, the pre-existing path is an intermediate location in the direction from the initial location of electronic device 500a to the destination and / or a location that is the navigation destination itself. In some embodiments, the pre-existing path is a predetermined path included on the route from the starting location of electronic device 500a to the destination. For example, electronic device 500 uses a pre-existing path to move from a temporary parking lot on a site that does not include predetermined paths to a road included in map data accessible to electronic device 500. In this example, once on a road, electronic device 500 navigates to its destination using one or more additional pre-defined paths included in the road and optional location map information. Alternatively, electronic device 500 may use ad-hoc paths in wilderness areas, such as when traveling within a campsite, where pre-defined paths are not included in the map information. Ad-hoc paths may be included at the start, end, or somewhere in between of navigation from the starting point to the destination. In some embodiments, when electronic device 500a is in a different location, it displays a map of the area without pre-defined paths. For example, a user of electronic device 500a uses it to plan, browse, and / or share ad-hoc paths that may be used in the future by electronic device 500a and / or by different devices. In some embodiments, electronic device 500a omits the display of... Figure 6C The graphical user interface in, and Figure 6C The graphical user interface in the diagram represents the current location 622 of the electronic device relative to a predetermined path 620 in the environment of the electronic device.

[0219] In some implementations, in response to detecting the location of electronic device 500a or the location that the electronic device is navigating to does not include a predetermined path, electronic device 500a generates one or more possible pre-defined paths to the destination. In some cases, electronic device 500a prompts the user for input to select or create a pre-defined path, as described in the following reference. Figures 6D to 6JDescribed in more detail. In some embodiments, electronic device 500a requests varying amounts of feedback from the user based on the level of confidence that one or more predetermined paths generated by electronic device 500a are suitable for reaching the destination. In some embodiments, electronic device 500a uses one or more of machine learning, scene understanding, computer vision, and / or algorithmic techniques to generate and / or evaluate possible paths.

[0220] In some implementations, electronic device 500a uses multiple factors and / or criteria (optionally with different weights) to generate and / or evaluate pre-existing paths for navigation in areas where no pre-determined path exists. Examples of these factors include: The size of the vehicle communicating with the electronic device 500a that is navigating to its destination and the size of the environment surrounding that vehicle; Weather conditions at the starting point or when the electronic device 500a will proceed with the journey; The presence path previously used by the user of electronic device 500a, the vehicle communicating with electronic device 500a, or another electronic device; Information provided by the managers of the locations included in the on-site path; Guidance provided by people in the environment of electronic device 500a; The behavior of vehicles in an environment with electronic device 500a; The presence and / or arrangement of temporary traffic objects (such as traffic cones, traffic signs, temporary road closures or detours, and / or law enforcement personnel) in the environment of electronic device 500a; and / or The type of environment in which electronic device 500a is currently located or will be located based on navigation guidance, navigation destination, and / or travel itinerary; User preferences.

[0221] In some implementations, electronic device 500a generates a precognitive path based on one or more of the factors described above. For example, if a corresponding factor has a first value, electronic device 500a generates one or more first precognitive paths, and if the corresponding factor has a second value different from the first value, electronic device 500a generates one or more second precognitive paths different from the one or more first precognitive paths. For example, when the temperature of the physical environment of electronic device 500a is above a threshold level (e.g., 25 degrees Celsius, 28 degrees Celsius, 30 degrees Celsius, 32 degrees Celsius, or 35 degrees Celsius), one or more precognitive paths include greater use of shaded areas of the physical environment than when the temperature of the physical environment of electronic device 500a is below the threshold temperature. Additional examples and details regarding these factors are described below with reference to method 700. In some implementations, electronic device 500a uses data 614 sensed by one or more sensors 602, position data 616 sensed by position sensor 604, and / or information provided by a second electronic device 500b to determine the nature of these factors.

[0222] Additionally or alternatively, in some examples, electronic device 500a uses a pre-determined path provided by another electronic device. In some embodiments, an electronic device associated with a geographic location provides a pre-defined path for use at that location. For example, an event organizer uses an electronic device to provide a pre-defined path for use in a temporary event parking lot. As another example, a business owner uses an electronic device to provide a pre-defined path for use in their business parking lot. As another example, a property owner uses an electronic device to provide a pre-defined path for use on their property. In some embodiments, other electronic devices transmit the pre-defined path to other devices, such as electronic device 500a, located in an area including the pre-defined path. As another example, other electronic devices make the pre-defined path available to electronic devices (including electronic device 500a) that view a map of a geographic area including the pre-defined path, regardless of the current location of the electronic device viewing the map.

[0223] Figure 6D An example of an electronic device 500a outputting a user interface is illustrated, which includes a representation 630a of a presence path. In some embodiments, electronic device 500a displays... Figure 6D The user interface is shown. Additionally or alternatively, in some embodiments, the electronic device 500a outputs an audio description 634a of the precognitive path. In some embodiments, the electronic device 500a presents the precognitive path based on a relatively high confidence level. Figure 6D The graphical user interface and / or audio 634a.

[0224] like Figure 6DAs shown, the graphical user interface includes an instruction 630a for the pre-defined path, an instruction 622 for the initial positioning of the electronic device 500a, and / or a prompt 626 that instructs the user that the electronic device 500a will continue navigating according to the pre-defined path after a predetermined time threshold, optionally including autonomous driving along the pre-defined path, unless the electronic device 500a receives user input requesting that the electronic device 500a not navigate the pre-defined path. Figure 6D As shown, prompt 626 includes a selectable option 628 that, when selected, causes electronic device 500a to abandon navigation of the present path. In some embodiments, prompt 626 includes an animation having a duration equal to a predetermined time threshold to visually or otherwise indicate to the user the remaining time before electronic device 500a continues navigation according to the present path. For example, selectable option 628 is animated as a shadow that gradually fills selectable option 628 over a predefined threshold amount of time.

[0225] In some implementations, electronic device 500a presents a similar context to planning its upcoming journey along the field path 630a. Figure 6D The user interface. For example, electronic device 500a displays the user interface in the context of viewing, planning, sharing, and / or saving a pre-existing path for future use by electronic device 500a and / or different electronic devices. For example, in a context other than the upcoming movement, electronic device 500a abandons displaying prompt 626, and electronic device 500a does not initiate movement along the pre-existing path 630a if no input is received within a time threshold. In some embodiments, outside the context of the upcoming movement, the starting location 622 may differ from the current location of electronic device 500a. For example, electronic device 500a uses a location selected by the user as the starting location 622, regardless of the current location of electronic device 500a.

[0226] In some embodiments, electronic device 500a generates a precognitive path based on one or more of the factors listed above and described in more detail below with respect to method 700. Additionally or alternatively, in some embodiments, electronic device 500a senses an object 632a in its environment and generates a precognitive path based at least in part on avoiding collisions with the object 632a. In some embodiments, for illustrative purposes, the object 632a is included... Figure 6D In some implementations, if object 632a is not included in the map data, the electronic device 500a does not display a representation of object 632a in the user interface.

[0227] In some implementations, electronic device 500a senses objects in its environment and updates map data to include information about the size, shape, and / or location of objects based on whether the objects meet one or more criteria. For example, electronic device 500a updates map data to include objects, such as buildings, walls, curbs, and other long-term objects, that are more likely to remain in their current location than to move from it. In some implementations, electronic device 500a forgoes updating map data to include information about objects that do not meet one or more criteria. For example, electronic device 500a does not update map data to include information about objects (such as vehicles, people, animals, temporary traffic objects, and other short-term objects) that are more likely to move from their current location than to remain in it. In some implementations, electronic device 500a generates a pre-existing path to avoid collisions with objects, regardless of whether electronic device 500a updates map information to include information about objects.

[0228] In some implementations, when detecting an object, electronic device 500a detects a moving object, such as a person, animal, or vehicle. In some implementations, electronic device 500a tracks the movement of the object to predict its future movement and / or future location, thereby avoiding collisions with objects that appear to be moving into the path of electronic device 500a. For example, if electronic device 500a detects a person moving toward the road that electronic device 500a is navigating, and then no longer detects the person because the person has moved behind another object, electronic device 500a predicts that the person will move from behind the object toward the road, and may modify the path and / or the speed of the vehicle to avoid a collision.

[0229] Figure 6E Another example of an electronic device 500a that outputs a user interface is illustrated, which includes a representation 630b of the presence path. In some embodiments, Figure 6E The on-site path shown is different from Figure 6D The on-site path shown is different because one or more of the factors listed above and / or described below with reference to method 700 are different. For example, as Figure 6E As shown, the environment of electronic device 500a includes... Figure 6D Object 632a is located at a different position than object 632b. One or more other factors in Figure 6D and Figure 6E The circumstances may differ between implementations. In some implementations, if object 632b is not included in the map information, the graphical user interface does not include an indication of object 632b, but the electronic device 500a generates the pre-existing path 630b in part based on the location, size, and / or shape of object 632b.

[0230] In some implementation schemes, Figure 6E The graphical user interface in the reference includes the above. Figure 6D The described prompt 626 is similar to prompt 626. In some embodiments, if electronic device 500a has a relatively high degree of confidence in the present path 630b, electronic device 500a presents the user with an indication 630b of the present path and / or an audio description 634b of the present path, and continues to navigate and / or drive autonomously on the present path unless electronic device 500a receives user input corresponding to a request to abandon navigation and / or autonomous driving on the present path. For example, electronic device 500a detects input pointing to selectable option 628 and / or voice input corresponding to a request not to navigate and / or drive autonomously on the present path. In some embodiments, in response to this input, electronic device 500a abandons navigation and / or autonomous driving on the present path. In some embodiments, if electronic device 500a does not receive input after a predetermined time threshold since the presentation of the visual indication 630b of the present path, electronic device 500a continues to navigate and / or drive autonomously on the present path.

[0231] Figure 6F An example of an electronic device 500a is illustrated, which outputs visual indications 630c and 630d and / or audio indications 634c for a precognitive path. In some embodiments, based on a moderate level of confidence that the electronic device 500a has in the generated precognitive path, the electronic device 500a presents multiple path options to the user. In some embodiments, the electronic device 500a generates additional path options not presented to the user because the path options presented to the user have a higher multi-criteria ranking than other generated precognitive paths. In some embodiments, the electronic device 500a presents a graphical user interface and / or an audio user interface that includes indications of the path options.

[0232] For example, in Figure 6FIn this system, the graphical user interface includes visual indications 630c and 630d for route options, a prompt 636 requesting the user to select a route option, and selectable options 638a and 638b corresponding to the displayed route options. For example, in response to detecting a selection of option 638a, electronic device 500a continues navigation and / or autonomous driving along path A corresponding to visual indication 630c. Similarly, in response to detecting a selection of option 638b, electronic device 500a continues navigation and / or autonomous driving along path B corresponding to visual indication 630d. In some embodiments, electronic device 500a displays route indications 630c and 630d outside the context of an upcoming journey along one of the paths (such as when electronic device 500a is not at the starting point 622), and displays route indications 630c and 630d for planning, browsing, saving, and / or sharing purposes. In some implementations, in response to receiving input to select option 638a or 638b, electronic device 500a saves and / or shares path A or path B, respectively.

[0233] Additionally or alternatively, in some embodiments, the output of electronic device 500a includes audio 634c describing the pre-existing path. In some embodiments, in response to receiving voice input 640a from the user selecting a pre-existing path, electronic device 500a continues navigation and / or autonomous driving on the path selected by the user. In some embodiments, if electronic device 500a does not receive input selecting a pre-existing path, electronic device 500a abandons navigation and / or autonomous driving.

[0234] In some implementations, electronic device 500a generates and selects based on one or more of the factors and / or criteria listed above and / or described in more detail below with reference to method 700. Figure 6F The pre-existing paths included. In some implementations, if the environmental factors and / or criteria of the electronic device 500a are different, the electronic device 500a will generate and / or select different pre-existing paths.

[0235] Figure 6G An example is illustrated where electronic device 500a requests user input to create a precognitive path. For instance, if electronic device 500a has relatively low confidence in the precognitive path generated by it, electronic device 500a outputs... Figure 6G The graphical user interface and / or audio output shown is 634d. (Example) Figure 6G As shown, the graphical user interface includes prompts 642 guiding the user to provide input for creating a presence path, and indications 622 for the starting position of the electronic device 500a. (As shown...) Figure 6GAs shown, electronic device 500a receives input including movement of contact 603a, which draws a path in a graphical user interface. In some embodiments, the input includes a contact similar to 603a identifying the start position of the precognitive path, and a second contact indicating the end position of the precognitive path, instead of including movement of the contact along the desired precognitive path. In some embodiments, additionally or alternatively, electronic device 500a receives voice input 640b from a user describing the precognitive path. For example, the voice input includes guidance on how long to continue in a straight line, the location and / or direction of a turn to be made, and / or instructions for creating a path toward objects in the environment of electronic device 500a. In some embodiments, in response to receiving Figure 6G The illustrated input and / or voice input 640b to the graphical user interface, and the electronic device 500a output instructions for the pre-existing path provided by the user, such as... Figure 6H As shown.

[0236] Figure 6H An example is given of the output pair of electronic device 500a. Figure 6G The visual indication 630e and / or audio indication 634e of the user-provided presence path are included. In some embodiments, the electronic device 500a responds to pointing... Figure 6G The graphical user interface (GUI) input is used to output visual instructions 630e. For example, a precognitive path is a path drawn by the user as the contact 603g moves. Alternatively, a precognitive path is a path generated by electronic device 101 based on a start point and an end point provided by the user based on one or more other factors described herein. In some embodiments, electronic device 500a responds to... Figure 6G The voice input in the device is used to output audio instructions 634e. In some implementations, the electronic device 500a responds to... Figure 6G The audio instruction 634e is output to the graphical user interface input. In some embodiments, the electronic device 500a responds to... Figure 6G The electronic device 500a responds to voice input to output visual instructions 630e. In some implementations, the electronic device 500a responds to... Figure 6G The device outputs both visual indication 630e and audio indication 634e via input to the graphical user interface or voice input. In some embodiments, the electronic device 500a abandons the output of visual indication 630e and audio indication 634e and continues navigation according to the path.

[0237] Figure 6I This is another example of an electronic device 500a requesting user input to provide a presence path, similar to the reference above. Figure 6G The output of the described electronic device 500a. For example, in Figure 6IIn the electronic device 500a, the output includes a graphical user interface with prompts 642 and an audio output 634f, which requests user input to create a presence path, as referenced above. Figure 6G As described. Figure 6I As shown, electronic device 500a receives input and / or voice input 640c directed to a graphical user interface having contact 603b, thereby creating a presence path. In response to... Figure 6I One or more of the exemplified inputs, the electronic device 500a outputs an indication of the pre-field path, such as... Figure 6J As shown. In some embodiments, the electronic device 500a abandons the output of visual indicators 630f and audio indicators 634g, and continues to navigate according to the path.

[0238] Figure 6J An example is given of the output pair of electronic device 500a. Figure 6I The visual indication 630f and / or audio indication 634g are provided by the user in the presence path. In some embodiments, the electronic device 500a outputs the visual indication 630f and / or audio indication 634g, as referenced above. Figure 6H As described.

[0239] Figure 7 This is a flowchart illustrating a method for an electronic device according to some embodiments of the present disclosure to navigate or travel on a predetermined path and / or a pre-defined path. Method 700 may optionally be implemented in a first device and / or electronic device (such as those referenced above). Figure 5I The operation is performed at the described device 100 or device 500. Some operations in method 700 may be optionally combined, and / or the order of some operations may be optionally changed.

[0240] As described below, method 700 provides a way for electronic devices to navigate and / or travel using immersive paths and / or pre-defined paths. Efficient navigation and / or travel using immersive paths and / or pre-defined paths enhances the user experience by simplifying user interaction with the electronic device, thus reducing the burden of attempting to reach a destination.

[0241] In some embodiments, method 700 is performed at an electronic device (e.g., 500a) that communicates with one or more input devices and one or more output devices. In some embodiments, the electronic device is a mobile device (e.g., a tablet, media player, smartphone, or wearable device) that includes wireless communication circuitry and optionally communicates with one or more of the following: a touchpad (optionally integrated or external), a mouse (e.g., external), a remote control device (e.g., external), a handheld device (e.g., external), another mobile device (e.g., separate from the electronic device), and / or a controller (e.g., external). In some embodiments, one or more output devices include audio output devices (e.g., one or more speakers, headsets, and / or headphones), a display component, and / or a haptic output device. In some embodiments, the display component is integrated with the electronic device (e.g., a touchscreen display), is external to the electronic device (e.g., a monitor, television, projector), or is a hardware component (optionally integrated or external) for making the user interface visible or projecting the user interface. In some implementations, electronic devices communicate with or are integrated with vehicles (e.g., cars, airplanes, and / or ships).

[0242] In some implementations, when navigating to a physical location (702), the current location of the electronic device (e.g., 500) being navigated to includes a pre-determined path (e.g., 620) (included in a map being used for navigation), such as in... Figure 6B In this process, the electronic device navigates along the path (704) while simultaneously outputting a user interface via one or more output devices for selecting a corresponding (optionally user-defined) path along which the electronic device will navigate, including its current location. In some embodiments, the predetermined path is an official road included in a map accessible to the electronic device. In some embodiments, navigation to a physical location includes presenting navigational instructions to the user using one or more output devices. For example, the electronic device instructs the user on what maneuvers to perform when operating a vehicle. In some embodiments, navigation to a physical location includes controlling one or more actuators of the vehicle to autonomously travel along the navigation route. In some embodiments, the path is a road included in a map.

[0243] In some implementations, when navigating to a physical location (702), depending on whether the current location of the electronic device (e.g., 500a) is being navigated does not include a predetermined path (e.g., 620) (which is included in the map being used for navigation) (706), the electronic device (e.g., 500) outputs (708) via one or more output devices a user interface for selecting a corresponding path that includes the current location of the electronic device to be navigated along, such as in Figure 6F In some embodiments, the electronic device navigates in physical areas that do not include road or lane markings, such as remote areas not marked on a map, parking lots (e.g., without parking space markings, lane markings, etc.), private real estate, and other open areas. In some embodiments, the output user interface includes displaying a graphical user interface using a display component (e.g., included in the vehicle). In some embodiments, the output user interface includes playing audio, such as prompts providing guidance to the user on where to navigate. In some embodiments, as described in more detail below, the user interface includes two or more options corresponding to user-selectable routes available for the vehicle, chosen by the electronic device. In some embodiments, as described in more detail below, the user interface includes an interface for the user to provide a customized route for navigating the vehicle.

[0244] In some implementations, when navigating to a physical location (702), depending on whether the current location of the electronic device (e.g., 500a) is being navigated does not include a predetermined path (e.g., 620) (which is included in the map being used for navigation) (706), the electronic device (e.g., 500a) receives (710) user input for selecting the appropriate path via one or more input devices, such as in Figure 6G In some embodiments, user input is the selection of a chosen path or the definition of a custom path presented by an electronic device. In some embodiments, user input is voice input, input provided using a touchscreen, input provided using hardware buttons or switches, or gestures performed by a part of the user (e.g., hand, arm, and / or head) detected by one or more cameras.

[0245] In some implementations, when navigating to a physical location (702), based on the determination that the current location of the electronic device (e.g., 500a) being navigated does not include a predetermined path (e.g., 620) (included in the map being used for navigation) (706), in response to receiving an input to select a corresponding path (712), based on the determination that the input corresponds to a request to select a first path, the electronic device (e.g., 500a) uses the first path for navigation (714), such as in Figure 6G In some embodiments, in response to receiving input selecting a first route, the electronic device autonomously drives along the first route. In some embodiments, in response to receiving input selecting a first route, the electronic device presents navigation guidance instructing the user to drive the vehicle along the first route.

[0246] In some implementations, when navigating to a physical location (702), based on the determination that the current location of the electronic device (e.g., 500a) being navigated does not include a predetermined path (e.g., 620) (which is included in the map being used for navigation) (706), in response to receiving an input to select a corresponding path (712), based on the determination that the input corresponds to a request to select a second path different from the first path, such as in Figure 6I In this embodiment, the electronic device (e.g., 500a) uses a second path for navigation (716) (e.g., navigating along the second path). In some embodiments, in response to receiving input selecting a second path, the electronic device autonomously drives along the second path. In some embodiments, in response to receiving input selecting a second path, the electronic device presents navigation guidance instructing the user to drive the vehicle along the second path. In some embodiments, selecting the appropriate path is different from real-time vehicle operation. In some embodiments, the electronic device uses input devices other than vehicle control input devices (such as a steering wheel, accelerator pedal, and / or brake pedal) to receive input selecting the appropriate path. In some embodiments, the input selecting the appropriate path includes input corresponding to a series of maneuvers to be performed along the appropriate path (this input is received before navigation is performed based on these maneuvers), rather than receiving input for one maneuver at a time for real-time driving. Navigating along paths included in a map when a mapped path is available, and outputting a user interface for selecting the appropriate path when a mapped path is unavailable, enhances user interaction with the electronic device by reducing the amount of input required to navigate along a predetermined path and providing the user with improved control when a predetermined path is unavailable.

[0247] In some implementations, the user interface for selecting a corresponding path includes: determining, based on a first relationship between the size of the vehicle communicating with the electronic device and the size of the environment surrounding that vehicle, an indication of a third path corresponding to the first relationship (e.g., 630a), such as in... Figure 6D In some embodiments, the dimensions of the vehicle include one or more of its height, width, length, and / or ground clearance. In some embodiments, the dimensions of the environment include the dimensions of the area through which the vehicle will travel to follow a corresponding path. In some embodiments, the electronic equipment and / or the vehicle uses one or more sensors (such as lidar, distance sensors, radar, and / or cameras) to sense the dimensions of the environment. In some embodiments, the electronic equipment obtains information about the dimensions of the environment from map-based information corresponding to the current position of the electronic equipment and / or the vehicle. For example, the electronic equipment senses the vertical height between the ground and an object under which the vehicle will drive (such as a bridge, overpass, arch, or canopy). As another example, the electronic equipment senses the width of a path between two or more of walls, buildings, railings, and / or other vehicles. As another example, the electronic equipment senses the height of an object that the vehicle will drive over (such as a speed bump, ditch, and / or hillside). As another example, the electronic equipment senses the turning radius of turns included in the corresponding path. In some embodiments, the third path is the path through which the vehicle will physically adapt. In some implementations, the electronic device receives input indicating the selection of a third route and, in response, uses the third route for navigation, as described in more detail above. In some implementations, based on the determination that the relationship between the size of the vehicle and the size of the environment is a primary relationship, the electronic device abandons displaying instructions for a fourth route as described below.

[0248] In some implementations, the user interface for selecting a corresponding route includes: based on a second relationship determined to be different from a first relationship, indicating a fourth route corresponding to the second relationship (e.g., 630b), which is different from a third route, such as in... Figure 6EIn some embodiments, the fourth path is a path that the vehicle will physically adapt to. In some embodiments, the electronic device waives the presentation of indications for paths that the vehicle will not adapt to. In some embodiments, the user interface also includes one or more indications of additional path options that the vehicle will adapt to. In some embodiments, the indications for the third path and the fourth path are visual and / or audio indications. In some embodiments, the electronic device receives input selecting the fourth path and, in response, navigates using the fourth path, as described in more detail above. In some embodiments, the electronic device waives the display of indications for the third path described above, based on the determination that the relationship between the size of the vehicle and the size of the environment is a second relationship. Presenting indications of paths selected based on the size of the vehicle and the size of the environment enhances user interaction with the electronic device by efficiently providing the user with paths that they might expect based on the size of the vehicle and the environment.

[0249] In some implementations, the user interface for selecting the appropriate path includes: determining a first weather condition based on the weather conditions at the current location of the electronic device, and selecting a third path corresponding to the first weather condition (e.g., 630a), such as... Figure 6D In some embodiments, weather conditions include temperature, precipitation, sunshine and / or cloud cover and / or humidity at the current location of the electronic device. For example, on hot and sunny days, the path presented by the electronic device has more shade than the path presented on cloudy and / or cool days. Similarly, on cold days, the path presented by the electronic device has less shade than the path presented on hot and sunny days. Also, if there is snow at the current location of the electronic device, the electronic device presents a path that has been cleared of snow. In some embodiments, the electronic device receives input to select a third path and, in response, uses the third path for navigation, as described in more detail above. In some embodiments, based on the determination that the weather conditions are first weather conditions, the electronic device abandons the display of indications for a fourth path as described below.

[0250] In some implementations, the user interface for selecting the appropriate path includes: Based on the determination that the weather conditions at the current location of the electronic device are a second weather condition different from the first weather condition, a fourth path (e.g., 630b) corresponding to the second weather condition is determined. This fourth path differs from the first path, such as in... Figure 6EIn some implementations, the electronic device presents different routes based on different weather conditions at the device's current location. In some implementations, the electronic device receives input indicating the selection of a fourth route and, in response, navigates using the fourth route, as described in more detail above. In some implementations, if the weather conditions are determined to be secondary weather conditions, the electronic device abandons displaying indications for the third route described above. Presenting indications for routes selected based on weather conditions at the electronic device's current location enhances user interaction with the electronic device by efficiently providing the user with the route they might expect based on current weather conditions.

[0251] In some implementation schemes, such as in Figure 6G In this embodiment, the user interface for selecting the appropriate path includes prompts for the user to create the appropriate path (e.g., 642). In some embodiments, the electronic device presents audio prompts for user input requesting the creation of the appropriate path. In some embodiments, the electronic device displays visual instructions for user input requesting the creation of the appropriate path. In some embodiments, the user interface does not include a portion of the appropriate path, and the request for user input is a request for user input to create a path that starts from the location of the electronic device without defining an end point.

[0252] In some implementation schemes, such as in Figure 6G In this context, user input includes input corresponding to the creation of a corresponding path (e.g., via contact 603a). In some embodiments, user input includes spoken and / or voice input corresponding to a verbal guide for the corresponding path, as described in more detail below. In some embodiments, user input is input for “drawing” a path on a representation of the environment of the electronic device included in the displayed user interface. For example, when the user interface is displayed on a touchscreen, the electronic device receives input including movement of nearby objects (e.g., the user’s finger or stylus) for drawing the corresponding path (e.g., from the current location of the vehicle and / or electronic device to the suggested final location of the vehicle and / or electronic device). Presenting the user interface to the user for creating the corresponding path enhances the user’s interaction with the electronic device by providing the user with an efficient way to control the path used for navigation (e.g., driving) to the destination.

[0253] In some implementation schemes, such as in Figure 6GIn this embodiment, user input for selecting a route includes voice input (640b). In some embodiments, the voice input selects one of several route options presented by the electronic device. In some embodiments, the voice input confirms the use of the route presented by the electronic device. In some embodiments, the voice input creates the route by providing voice guidance to the electronic device (e.g., when the electronic device does not present a suggested route). For example, the voice input includes straight-ahead distance, landmarks to proceed to before turning and / or stopping, and turns and / or stops to make to follow the route. Accepting voice input to select a route enhances user interaction with the electronic device by providing a hands-free and / or eyes-free interface for providing user feedback, thereby improving user safety when using the electronic device in a vehicle.

[0254] In some implementations, the user interface for selecting a corresponding route includes: indicating a fourth route based on the third route (e.g., 630a), such as in [the context of the third route]. Figure 6D In some embodiments, the fourth route is identical to the third route. In some embodiments, a portion of the fourth route includes a portion of the third route. In some embodiments, the fourth route is additionally based on one or more other factors, such as those described above and / or below. For example, if one or more of the other factors and / or conditions described herein differ from the factors and / or conditions when the third route was adopted, the electronic device modifies one or more portions of the third route based on the current factors and / or conditions. In some embodiments, the third route is traveled by the user of the electronic device. In some embodiments, the third route is traveled by a user different from the user of the electronic device. In some embodiments, the third route is traveled by an electronic device different from the electronic device. In some embodiments, the electronic device receives input selecting the fourth route and, in response, uses the fourth route for navigation, as described in more detail above. In some embodiments, based on the determination that the vehicle previously followed the third route, the electronic device abandons displaying instructions for the sixth route described below.

[0255] In some implementations, the user interface for selecting a corresponding route includes: an indication of a sixth route based on the fifth route (e.g., 630b), which is different from the fourth route, such as in [the context of the third route], based on a fifth route determined to have been previously followed by the vehicle including its current location. Figure 6EIn some embodiments, the fifth path is the same as the sixth path. In some embodiments, a portion of the sixth path includes a portion of the fifth path. In some embodiments, the sixth path is additionally based on one or more other factors, such as those described above and / or below. For example, if one or more of the other factors and / or conditions described herein differ from the factors and / or conditions when the fifth path was adopted, the electronic device modifies one or more portions of the fifth path based on the current factors and / or conditions. In some embodiments, the fifth path is traveled by the user of the electronic device. In some embodiments, the fifth path is traveled by a user different from the user of the electronic device. In some embodiments, the fifth path is traveled by an electronic device different from the electronic device. In some embodiments, the electronic device receives input selecting the sixth path and, in response, navigates using the sixth path, as described in more detail above. In some embodiments, based on the determination that the vehicle previously followed the fifth path, the electronic device abandons displaying indications for the fourth path described above. Presenting a path based on a path previously traveled by the vehicle enhances user interaction with the electronic device by reducing the amount of input required to generate a path similar to a previously used path.

[0256] In some implementations, based on a determination that the user of the electronic device previously followed a third path including the current location of the electronic device, the user interface for selecting the appropriate path includes an indication of a fourth path based on the third path (e.g., 630a), such as in... Figure 6D In some embodiments, the fourth path is identical to the third path. In some embodiments, a portion of the fourth path includes a portion of the third path. In some embodiments, the fourth path is additionally based on one or more other factors, such as those described above and / or below. For example, if one or more of the other factors and / or conditions described herein differ from the factors and / or conditions when the third path was adopted, the electronic device modifies one or more portions of the third path according to the current factors and / or conditions. In some embodiments, the third path is traveled using the same vehicle as the vehicle communicating with the electronic device. In some embodiments, the third path is traveled using a different vehicle than the vehicle communicating with the electronic device. In some embodiments, the third path is traveled by the same electronic device. In some embodiments, the third path is traveled using a different electronic device. In some embodiments, the electronic device receives input selecting the fourth path and, in response, navigates using the fourth path, as described in more detail above. In some embodiments, based on the determination that the user previously followed the third path, the electronic device abandons displaying instructions for the sixth path described below.

[0257] In some implementations, based on determining that the user has previously followed a fifth path, including the current location of the electronic device, which differs from the third path, the user interface for selecting the appropriate path includes an indication of a sixth path based on the fifth path (e.g., 630b), which differs from the fourth path, such as in... Figure 6E In some embodiments, the fifth path is the same as the sixth path. In some embodiments, a portion of the sixth path includes a portion of the fifth path. In some embodiments, the sixth path is additionally based on one or more other factors, such as those described above and / or below. For example, if one or more of the other factors and / or conditions described herein differ from the factors and / or conditions when the fifth path was adopted, the electronic device modifies one or more portions of the fifth path according to the current factors and / or conditions. In some embodiments, the fifth path is traveled using the same vehicle as the vehicle communicating with the electronic device. In some embodiments, the fifth path is traveled using a different vehicle than the vehicle communicating with the electronic device. In some embodiments, the fifth path is traveled by the same electronic device. In some embodiments, the fifth path is traveled using a different electronic device. In some embodiments, the electronic device receives input selecting the sixth path and, in response, navigates using the sixth path, as described in more detail above. In some embodiments, based on the determination that the user previously followed the fifth path, the electronic device abandons displaying instructions for the fourth path described above. Paths are presented based on the path the user has previously traveled, which enhances user interaction with electronic devices by reducing the amount of input required to generate a path similar to the previously used path.

[0258] In some implementations, based on the determination that the second electronic device previously followed a third path including the current location of the electronic device, the user interface for selecting the appropriate path includes an indication of a fourth path (e.g., 630a) based on the third path, such as in... Figure 6DIn some embodiments, the fourth path is identical to the third path. In some embodiments, a portion of the fourth path includes a portion of the third path. In some embodiments, the fourth path is additionally based on one or more other factors, such as those described above and / or below. For example, if one or more of the other factors and / or conditions described herein differ from the factors and / or conditions when the third path was adopted, the electronic device modifies one or more portions of the third path according to the current factors and / or conditions. In some embodiments, the third path is traveled by the user of the electronic device. In some embodiments, the third path is traveled by a user different from the user of the electronic device. In some embodiments, the third path is traveled by multiple users, electronic devices, and / or vehicles. In some embodiments, the electronic device receives input selecting the fourth path and, in response, uses the fourth path for navigation, as described in more detail above. In some embodiments, based on the determination that a second electronic device previously followed the third path, the electronic device abandons displaying instructions for a sixth path as described below.

[0259] In some implementations, based on the determination that the second electronic device previously followed a fifth path including the current location of the electronic device, which is different from the third path, the user interface for selecting the appropriate path includes an indication of a sixth path based on the fifth path (e.g., 630b), which is different from the fourth path, such as in... Figure 6E In some embodiments, the fifth path is identical to the sixth path. In some embodiments, a portion of the sixth path includes a portion of the fifth path. In some embodiments, the sixth path is additionally based on one or more other factors, such as those described above and / or below. For example, if one or more of the other factors and / or conditions described herein differ from the factors and / or conditions when the fifth path was adopted, the electronic device modifies one or more portions of the fifth path based on the current factors and / or conditions. In some embodiments, the fifth path is traveled by the user of the electronic device. In some embodiments, the fifth path is traveled by a user different from the user of the electronic device. In some embodiments, the fifth path is traveled by multiple users, electronic devices, and / or vehicles. In some embodiments, the electronic device receives input selecting the sixth path and, in response, navigates using the sixth path, as described in more detail above. In some embodiments, based on the determination that a second electronic device previously followed the fifth path, the electronic device abandons displaying indications for the fifth path described above. Presenting a path based on a path previously traveled by a second electronic device enhances user interaction with the electronic device by reducing the amount of input required to generate a path similar to a previously used path.

[0260] In some implementations, the electronic device (e.g., 500a) receives path information of the current location of the electronic device from a second electronic device (e.g., 500b) associated with the current location of the electronic device, such as in... Figure 6A middle.

[0261] In some implementations, when navigating to a physical location, the current location of the electronic device (e.g., 500a) that is being navigated does not include a predetermined path (e.g., ...). Figure 6B (620 in the example), based on path information determined from the second electronic device, including first path information, the user interface for selecting the appropriate path includes an indication of a third path based on the first path information (e.g., 630a), such as in Figure 6D In some embodiments, the second electronic device is operated by the owner and / or manager of the property located at the current location of the electronic device. For example, a shopping mall manager configures the second electronic device to provide appropriate routes for navigating the shopping mall's parking lot. In some embodiments, based on determining that the route information from the second electronic device includes first route information, the electronic device abandons presenting an indication of a fourth route as described below. In some embodiments, the electronic device receives input selecting an indication of a third route and, in response to receiving such input, navigates using the third route.

[0262] In some implementations, when navigating to a physical location, the current location of the electronic device (e.g., 500a) that is being navigated does not include a predetermined path (e.g., ...). Figure 6B In 620), based on path information determined from a second electronic device, including second path information different from the first path information, a user interface for selecting a corresponding path includes an indication of a fourth path based on the second path information (e.g., 630b), which is different from the third path, such as in... Figure 6EIn some embodiments, the path information provided by the second electronic device is not included in a map accessed by the electronic device that includes predetermined paths. In some embodiments, the path information includes one or more paths provided by the owner and / or manager of the property located at the current location of the electronic device for navigating the environment of the electronic device. In some embodiments, the path information includes one or more areas of the environment that should not be included in the paths used for navigating the location. In some embodiments, based on the determination that the path information from the second electronic device includes second path information, the electronic device waives the presentation of instructions for the third path described above. In some embodiments, the electronic device receives input selecting instructions for a fourth path and, in response to receiving such input, navigates using the fourth path. Presenting instructions for paths based on path information provided by the second electronic device associated with the current location of the electronic device enhances interaction with the electronic device by efficiently providing the preferred path for authorized personnel at the current location, thereby improving security and / or efficiency when navigating the location.

[0263] In some implementations, when navigating to a physical location, the electronic device (e.g., 500a) communicates via one or more input devices (e.g., Figure 6A Sensor 602 in the electronic device detects objects in the environment that are not included in map information accessible to the electronic device. In some embodiments, the object is one that a vehicle communicating with the electronic device should avoid, and the object's location should not be included in the corresponding path. In some embodiments, a predetermined path is included in the map information. In some embodiments, multiple other electronic devices have access to the map information.

[0264] In some implementations, when navigating to a physical location, the electronic device (e.g., 500a) updates map information to include information associated with the object, based on the determination that the object meets one or more criteria. In some implementations, meeting one or more criteria corresponds to long-term characteristics of the object as a location, such as buildings, walls, fences, curbs, traffic signs (e.g., stop signs, no-entry signs, or one-way signs), making the object more likely to remain in its location rather than be removed from it. For example, the electronic device uses machine learning, scene understanding, and / or computer vision techniques to classify objects and determine whether the object's category meets one or more criteria. In some implementations, the electronic device stores the size and location of the object in the map information and uses the information associated with the object to plan a path in the current location in the future. In some implementations, the electronic device shares the size and location of the object with one or more other electronic devices that have access to map information, which optionally allows those other electronic devices to use the size and location of the object when navigating using the map information. For example, if the electronic device has already planned a path that will intersect with the object and must develop a new path to avoid the object, the next time the electronic device navigates in that location, the electronic device may optionally develop a different path to avoid the object in a more efficient manner.

[0265] In some implementations, when navigating to a physical location, if an object is determined not to meet one or more criteria, the electronic device (e.g., 500a) refrains from updating map information to include information associated with the object. In some implementations, not meeting one or more criteria corresponds to short-term characteristics of the object as a location, such as a person, animal, vehicle, or temporary traffic object (e.g., traffic cones, road barriers, or law enforcement closures of the road), which makes the object more likely to move from its location rather than remain in it. In some implementations, the electronic device avoids the object (or more generally, considers the object for route planning purposes) but does not store the object's size and / or location as map information. Selectively updating the map information to include objects that meet one or more criteria, while not updating the map information to include objects that do not meet one or more criteria, enhances user interaction with the electronic device by developing routes based on the updated information, thereby providing more efficient navigation paths.

[0266] In some implementations, upon navigating to a physical location, the electronic device (e.g., 500a) immediately initiates navigation via one or more input devices (e.g., ...). Figure 6ASensor 602 in the electronic device detects objects in the environment of the electronic device that are not included in map information accessible to the electronic device. In some embodiments, the objects are people or animals. In some embodiments, the electronic device uses machine learning, computer vision, and / or scene understanding technologies to identify the objects. In some embodiments, the objects are moving. In some embodiments, a predetermined path is included in the map information.

[0267] In some implementations, when navigating to a physical location, at a second time different from the first time, the electronic device (e.g., 500) transmits data via one or more input devices (e.g., ...). Figure 6A The electronic device detects an object using sensor 602; and in some embodiments, the electronic device does not detect an object between a first time and a second time. For example, the object moves behind another object that occludes the object from the field of view of one or more sensors (e.g., a camera) of the electronic device.

[0268] In some implementations, when navigating to a physical location, the electronic device (e.g., 500a) tracks objects in its environment based on object detection in a first and second time. In some implementations, tracking object movement includes predicting the object's future movement. For example, the electronic device senses an upcoming portion of an object moving toward the electronic device's path, after which the object is occluded by another object. In this example, the electronic device predicts the object will emerge from behind another object and continue toward the upcoming portion of the path, and slows down, stops, or modifies its path to avoid a collision with that object. Object tracking enhances user interaction with the electronic device by enabling the electronic device to modify its path to avoid collisions with objects, thereby enhancing the operational safety of the electronic device.

[0269] In some implementations, the user interface for selecting a corresponding path includes a representation of one or more path options (e.g., 630a), such as in Figure 6D In some embodiments, the user interface is a graphical user interface (GUI) that includes a representation of the electronic device's environment and visual indications of one or more paths. For example, the user interface includes a map from a bird's-eye view that includes path indications. In some embodiments, the user interface is an audio-based user interface that includes audio descriptions of one or more path options.

[0270] In some implementations, based on a set of criteria that determine one or more first path options (e.g., 630a) satisfy and a set of one or more second path options that differ from the one or more first path options do not satisfy the criteria, the user interface for selecting a corresponding path includes a representation of one or more first path options (e.g., 630a), such as in Figure 6D In some embodiments, the electronic device generates one or more first path options and one or more second path options, and sorts the path options according to a set of criteria, and presents a subset of the path options with the highest sorting using one or more criteria. In some embodiments, the criteria include the criteria and / or factors described above and below. In some embodiments, the electronic device weights the criteria when evaluating the path options, thereby assigning more weight to some criteria and less weight to others. In some embodiments, the user interface excludes one or more second path options based on determining that one or more first path options satisfy a set of criteria and one or more second path options do not satisfy a set of criteria. In some embodiments, the electronic device abandons presenting one or more second path options as described below based on determining that one or more first path options satisfy a set of criteria and one or more second path options do not satisfy a set of criteria. In some embodiments, the electronic device receives input selecting one of the first path options, and in response to receiving the input, navigates using the appropriate path corresponding to the selected first path option.

[0271] In some implementations, based on a set of criteria that determine one or more second path options (e.g., 630b) satisfy and a set of criteria that determine one or more first path options do not satisfy, the user interface for selecting a corresponding path includes a representation of one or more second path options (e.g., 630b), such as in Figure 6E In some embodiments, the user interface excludes one or more first path options based on determining that one or more second path options satisfy a set of criteria and one or more first path options do not satisfy a set of criteria. In some embodiments, the electronic device abandons presenting one or more first path options described above based on determining that one or more second path options satisfy a set of criteria and one or more first path options do not satisfy a set of criteria. In some embodiments, the electronic device receives input selecting one of the second path options and, in response to receiving the input, navigates using the appropriate path corresponding to the selected second path option. Selecting path options based on a set of criteria to include in the user interface enhances user interaction with the electronic device by reducing the amount of input required to select a suitable path and saving battery life.

[0272] In some implementations, the user interface for selecting a corresponding path includes a representation of one or more path options (e.g., 630c and / or 630d), such as in Figure 6F In some implementations, path options are presented as described above.

[0273] In some implementations, based on the determination that the vehicle in the environment of the electronic device has a first behavior, the user interface for selecting the appropriate route includes a representation of one or more first route options (e.g., 630a), such as in Figure 6D In some embodiments, vehicle behavior includes speed, direction of travel, merging behavior, and / or the vehicle's position within the electronic device's environment. In some embodiments, one or more first path options include path options that, depending on the situation, avoid other vehicles, follow other vehicles, or give way to other vehicles. In some embodiments, based on determining that a vehicle in the environment exhibits a first behavior, the electronic device abandons presenting one or more second path options as described below. In some embodiments, the electronic device receives input selecting one of the first path options and, in response to receiving that input, navigates using the appropriate path corresponding to the selected first path option.

[0274] In some implementations, based on the determination that the vehicle in the environment of the electronic device has a second behavior different from the first behavior, the user interface for selecting the appropriate path includes a representation of one or more second path options different from one or more first path options (e.g., 630b), such as in Figure 6DIn some embodiments, one or more second route options differ from one or more first route options because the one or more second route options avoid other vehicles, while one or more first route options follow or give way to other vehicles. In some embodiments, one or more second route options differ from one or more first route options because the one or more second route options follow other vehicles, while the first route options avoid or give way to other vehicles. In some embodiments, one or more second route options differ from one or more first route options because the one or more second route options give way to other vehicles, while one or more first route options avoid or follow other vehicles. In some embodiments, one or more first route options and one or more second route options avoid, follow, or give way to other vehicles, but do so in different ways due to differences in the behavior of the other vehicles. In some embodiments, based on determining that the vehicles in the environment have a second behavior, the electronic device abandons presenting one or more first route options as described above. In some embodiments, the electronic device receives input selecting one of the second route options and, in response to receiving that input, navigates using the appropriate path corresponding to the selected second route option. Presenting route options based on the behavior of other vehicles in the electronic device environment enhances user interaction with electronic devices by improving the safety and efficiency of the routes presented by the electronic devices.

[0275] In some implementations, when navigating to a physical location and when the electronic device's current location does not include a predetermined path, the electronic device (e.g., 500a) uses one or more input devices (e.g., Figure 6A Sensor 602 in the electronic device detects a person in the environment providing route guidance. In some embodiments, the electronic device detects a person providing route guidance using gestures (e.g., waving and / or holding traffic signs) and / or voice commands. In some embodiments, the guidance includes one or more of the following: stop, go forward, turn, merge, drive to a lane or location, and / or follow another vehicle or object. In some embodiments, the person is located outside the vehicle the user is using to navigate the route. In some embodiments, the person is different from the user of the electronic device.

[0276] In some implementation schemes, such as in Figure 6D In this context, the user interface for selecting a path includes a representation of one or more path options (e.g., 630a). In some embodiments, the electronic device presents path options as described in more detail above.

[0277] In some implementations, the user interface for selecting a path includes a first path guide based on a path provided by a person, and includes a representation of one or more first path options corresponding to the first path guide (e.g., 630a), such as in Figure 6D In some embodiments, one or more first route options follow a first route guide provided by a person. In some embodiments, based on the determination that the human-provided route guide includes the first route guide, the electronic device abandons presenting one or more second route options as described below. In some embodiments, the electronic device receives input selecting one of the first route options, and in response to receiving the input, navigates using the appropriate path corresponding to the selected first route option.

[0278] In some implementations, based on the determination that the path guidance provided by a person includes a second path guidance different from the first path guidance, the user interface for selecting the corresponding path includes a representation of one or more second path options corresponding to the second path guidance, different from the first path options (e.g., 630b), such as in Figure 6E In some embodiments, one or more second route options follow second route guidance provided by a person. In some embodiments, route options are included in the path to the destination. In some embodiments, if it is determined that the human-provided route guidance is incompatible with reaching the destination, the route options include the option to leave the area and reach the destination using an alternative route. In some embodiments, if it is determined that the human-provided route guidance includes second route guidance, the electronic device abandons presenting one or more first route options described above. In some embodiments, the electronic device receives input selecting one of the second route options and, in response to receiving the input, navigates using the appropriate path corresponding to the selected second route option. Presenting route options based on human-provided guidance in the environment of the electronic device enhances user interaction with the electronic device by efficiently and automatically incorporating these guidelines into navigation to the destination.

[0279] In some implementations, when navigating to a physical location and when the current location of the electronic device does not include a predetermined path, based on the determination that the path guidance provided by a person fails to meet one or more criteria, the electronic device (e.g., 500a) outputs a user interface for selecting an appropriate path. This user interface includes requests for user input for selecting the appropriate path (e.g., 636 and / or 634c), such as in... Figure 6FIn some implementations, one or more criteria are associated with the confidence level of the electronic device in its interpretation of human-provided route guidance. In some implementations, failure to meet one or more criteria corresponds to the electronic device's confidence level in its interpretation of human-provided route guidance being less than a predetermined threshold level. In some implementations, the electronic device compares the confidence level to multiple predefined threshold levels and presents different user interfaces for selecting an appropriate route based on whether the confidence level meets or does not meet the threshold, as described in more detail below. For example, the higher the confidence level of the electronic device in its interpretation of human-provided route guidance, the less user intervention will be requested. In some implementations, the electronic device selects an appropriate route without user input based on the determination that human-provided guidance meets one or more criteria. In some implementations, the electronic device additionally presents an indication of the appropriate route. Requesting user input for selecting an appropriate route based on the determination that human-provided route guidance fails to meet one or more criteria enhances user interaction by improving security and reducing errors when the electronic device's confidence level in human-provided guidance is low and improving efficiency when the electronic device's confidence level in human-provided guidance is high.

[0280] In some implementations, based on the determination that human-provided route guidance fails to meet one or more criteria (and the electronic device has a relatively high level of confidence in its interpretation of the guidance compared to the level of confidence described below), a user interface for selecting a route is presented, including a request for user input to select the route. This includes presenting options (e.g., 628) and indications of the route (e.g., 630b) using one or more output devices. When an option is selected, the electronic device abandons navigation along the route, such as in... Figure 6E In some embodiments, the electronic device presents a representation of the corresponding path, as described in more detail above. In some embodiments, the option is a choice displayed by the electronic device in a graphical user interface, and detecting a selection of an option includes detecting input pointing to the displayed option. In some embodiments, the option is an audio prompt, and detecting a selection of an option includes detecting voice input corresponding to a request to abandon navigation along the path.

[0281] In some implementations, when navigating to a physical location, if the human-provided route guidance fails to meet one or more criteria (and the electronic device has a relatively high level of confidence in its interpretation of the guidance compared to the confidence levels described below), the appropriate route is selected based on the user interface (such as in...) used to select the corresponding route. Figure 6EIf a predetermined time threshold has been exceeded (e.g., no selection of an optional option is detected), the electronic device (e.g., 500) navigates along the corresponding path. In some embodiments, the predetermined time threshold is 1 second, 2 seconds, 3 seconds, 5 seconds, 10 seconds, 15 seconds, or 30 seconds. In some embodiments, in response to detecting a selection of an optional option, the electronic device abandons navigation along the corresponding path. Additionally or alternatively, in some embodiments, the electronic device presents one or more alternative path options in the user interface. In some embodiments, the electronic device presents options and indications of the corresponding path based on a relatively high confidence level for the corresponding path compared to other confidence thresholds. Concurrently presenting indications of the corresponding path and the option to abandon navigation along the corresponding path, and navigating along the corresponding path after the time threshold, enhances user interaction with the electronic device by allowing the user to prevent the electronic device from navigating the path, thereby improving security, or automatically navigating the path if the user does not prevent the electronic device from navigating the corresponding path, thereby improving efficiency.

[0282] In some implementations, based on the determination that the human-provided route guidance fails to meet one or more criteria (and the electronic device has moderate confidence in its interpretation of the guidance compared to the confidence levels described above and below), the electronic device (e.g., 500a) presents a user interface for selecting a route, including a request for user input for selecting the appropriate route, comprising presenting multiple route options based on the human-provided route guidance (e.g., 630c and / or 630d), such as in Figure 6F In some implementations, in response to a moderate level of confidence in multiple route options compared to other confidence thresholds, the electronic device presents multiple route options based on human-provided route guidance. In some implementations, the electronic device abandons navigation according to a given route option unless and until it receives user input selecting one of the route options. In some implementations, the electronic device presents multiple route options, as described in more detail above. Presenting multiple route options based on human-provided route guidance enhances user interaction with the electronic device by improving security and reducing the time and input required to follow a route according to human-provided route guidance.

[0283] In some embodiments, based on the determination that the human-provided path guidance fails to meet one or more criteria (and the electronic device has relatively low confidence in its interpretation of the guidance compared to the confidence levels described above), the electronic device (e.g., 500a) presents a user interface for selecting a path, including a request for user input for selecting the path, comprising presenting a prompt to the user to provide the path (e.g., 642 and / or 634d). In some embodiments, the electronic device presents the prompt to the user to provide the path based on the electronic device's low confidence in the human-provided path guidance compared to other confidence thresholds. In some embodiments, the electronic device abandons navigation along the path unless and until the electronic device receives user input to provide the path. In some embodiments, the user input refers to a graphical user interface, such as the user drawing the path on a representation of the electronic device's environment. In some embodiments, the user input is voice input, such as a voice instruction for following a path. In some embodiments, in response to receiving the user input, the electronic device navigates along the path. In some embodiments, the electronic device abandons navigation along the path unless and until the electronic device receives user input. Requesting the user to provide a corresponding path when one or more criteria are not met based on human-provided path guidance enhances user interaction with electronic devices by increasing security through additional user oversight.

[0284] In some implementations, when navigating to a physical location, the current location of the electronic device is determined to be outside a predetermined path, and the vehicle in the environment of the electronic device is determined to travel along a path that meets one or more criteria (e.g., based on data 614 from one or more sensors 602 or communication with a second device 500b associated with the vehicle, such as in...). Figure 6A In some embodiments, the electronic device (e.g., 500a) navigates along the path that the vehicle is traveling along (e.g., without outputting a user interface for selecting a corresponding path). In some embodiments, when the vehicle's path is included in a path to a destination, the vehicle's path satisfies one or more criteria. In some embodiments, navigating along the path that the vehicle is traveling along includes following the vehicle. In some embodiments, the electronic device continues to navigate along the path that the vehicle is traveling along (e.g., following the vehicle) while the path continues to satisfy one or more criteria. In some embodiments, the electronic device waives the output of a user interface. In some embodiments, the electronic device outputs a user interface that includes instructions on the vehicle's path.

[0285] In some implementations, when navigating to a physical location, the current location of the electronic device is determined not to include a predetermined path, and the vehicle in the environment of the electronic device is determined to travel along a path that fails to meet one or more criteria (e.g., based on data 614 from one or more sensors 602 or communication with a second device 500b associated with the vehicle, such as in...). Figure 6A (in the middle), outputs a user interface for selecting the corresponding path, such as in Figures 6D to 6J In some embodiments, when navigating along a path that a vehicle is traveling on (e.g., following a vehicle), if the path is determined to no longer meet one or more criteria, the electronic device stops following the vehicle and presents a user interface for selecting an appropriate path. In some embodiments, the user interface for selecting an appropriate path includes one or more path options different from the path of the vehicle. Navigating along a path that a vehicle is traveling on, based on whether the path meets one or more criteria, enhances user interaction with the electronic device by efficiently and safely navigating to the destination without interfering with other vehicles in the electronic device's environment.

[0286] In some implementations, when navigating to a physical location, the current location of the electronic device is determined to include a predetermined path, while the environment of the electronic device is determined to include one or more objects that indicate the path (e.g., by...). Figure 6AThe electronic device (e.g., 500a) navigates along a path indicated by one or more objects (e.g., without outputting a user interface for selecting the appropriate path), as determined by sensor 602. In some embodiments, the one or more objects are not included in the information used to map the device. In some embodiments, the objects are temporary traffic objects, such as traffic cones, road barriers, temporary traffic signs, and / or law enforcement personnel directing traffic. In some embodiments, the electronic device uses machine learning, scene understanding, computer vision, and / or one or more algorithms to determine the path indicated by the one or more objects. In some embodiments, the electronic device uses one or more sensors to detect the one or more objects. In some embodiments, based on the determination that the environment of the electronic device includes one or more objects indicating the path, the electronic device outputs a user interface with indications of the path indicated by the one or more objects. In some embodiments, based on the determination that the environment of the electronic device includes one or more objects indicating the path, the electronic device abandons the output of the user interface. In some embodiments, based on the determination that the path indicated by the one or more objects is included in the path to the destination, the electronic device navigates along the path indicated by the one or more objects. If it is determined that the path indicated by one or more objects is not included in the path to the destination, the electronic device abandons navigation along the path indicated by one or more objects and presents a user interface for selecting the appropriate path.

[0287] In some implementations, when navigating to a physical location, based on determining that the current location of the electronic device does not include a predetermined path, and based on determining that the environment of the electronic device does not include one or more objects indicating the path (e.g., based on data 614 sensed by sensor 602), the electronic device (e.g., 500a) outputs a user interface for selecting the appropriate path, such as in Figures 6D to 6J In some implementations, the user interface for selecting an appropriate path includes one or more indications of one or more paths that differ from those indicated by one or more objects. Navigating along a path indicated by one or more objects based on whether the environment includes one or more objects and outputting a user interface for selecting an appropriate path based on whether the environment does not include one or more objects enhances user interaction with the electronic device by efficiently navigating along the path when one or more objects indicate it, thereby reducing input and saving battery life, and improving safety when one or more objects do not indicate a path.

[0288] In some embodiments, the user interface for selecting a corresponding path includes displaying a visual indication of the corresponding path via a display component included in one or more output devices (e.g., 630a), such as in Figure 6DIn some embodiments, visual indications of the corresponding path are displayed in a user interface representing the environment of the electronic device, such as a map. In some embodiments, the electronic device generates the representation based on map information and / or data collected by one or more sensors communicating with the electronic device. Displaying visual indications of the corresponding path enhances user interaction with the electronic device by providing improved visual feedback to the user.

[0289] In some implementations, the user interface for selecting a corresponding path includes a representation of one or more path options (e.g., 630c and / or 630d), such as in Figure 6F In some implementations, the electronic device outputs one or more path options, as described in more detail above.

[0290] In some implementations, based on determining that the environment of the electronic device is a first type of environment, one or more path options are selected according to a first set of one or more criteria (e.g., 630a), such as in Figure 6D In some implementations, the electronic device uses machine learning, scene understanding, and / or computer vision techniques to determine the type of environment. Example types of environments include parking lots, drive-thru windows, curbside pickups, event parking areas, and / or private real estate. In some implementations, the electronic device receives input of selecting one of the route options and, in response, navigates using the route corresponding to the selected route option, as described in more detail above. In some implementations, based on determining that the environment is a first type, the electronic device abandons displaying one or more route options selected according to a second set of one or more criteria described in more detail below.

[0291] In some implementations, one or more path options are selected based on a second set of criteria (e.g., 630b) that differs from a first set of criteria, depending on whether the environment of the electronic device is a second type of environment different from a first type of environment (e.g., in...). Figure 6EIn some embodiments, depending on whether the environment of the electronic device is determined to be a first type of environment, the user interface includes a representation of one or more first paths selected and / or generated based on a first set of criteria. In some embodiments, depending on whether the environment of the electronic device is determined to be a second type of environment, the user interface includes a representation of one or more second paths selected and / or generated based on a second set of criteria, different from the first paths. In some embodiments, the electronic device prioritizes different characteristics when selecting path options to output based on the type of environment. In some embodiments, prioritizing characteristics is based on differences in appropriate driving behavior in different situations. For example, following other vehicles may have high priority for a drive-thru window at a restaurant, but not for a parking lot. In some embodiments, the electronic device receives input of a path option to be selected and, in response, navigates using the path corresponding to the selected path option, as described in more detail above. In some embodiments, depending on whether the environment is determined to be second type, the electronic device abandons displaying one or more path options selected based on a first set of criteria described in more detail above. Selecting path options with different criteria based on the type of environment in which the electronic device is located enhances the user's interaction with the electronic device by improving safety and efficiency.

[0292] It should be understood that, Figure 7 The specific order in which the operations described herein are presented is merely exemplary and is not intended to indicate that the described order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein.

[0293] The operations in the methods described above may optionally include running one or more functional modules in the device, such as a general-purpose processor (e.g., such as...). Figure 5I (As described) and / or application-specific circuitry. Additionally, the above references... Figure 7 The described operation may optionally be performed by Figure 5I The components described herein are used to implement this. When a corresponding predefined event or sub-event is detected, the event recognizer activates the event handler associated with the detection of that event or sub-event. The event handler may optionally use or invoke a data updater or object updater to update the application's internal state. In some implementations, the event handler accesses the corresponding GUI updater to update the user interface displayed associated with the application. Similarly, those skilled in the art will readily understand how this can be achieved based on... Figure 5I The components described herein are used to implement other processes.

[0294] For illustrative purposes, this disclosure is described by reference to specific embodiments. These discussions are not intended to be exhaustive or to limit this disclosure and / or claims to particular embodiments. Modifications and / or variations are possible according to this disclosure. Some embodiments have been selected and described to explain the principles of these techniques and their practical application. Others skilled in the art will thus be able to utilize these techniques and various embodiments with modifications and / or variations suited to the intended particular use.

[0295] While this disclosure and its embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and / or modifications will become apparent to those skilled in the art. It should be understood that such changes and / or modifications are considered to be included within the scope of this disclosure and its embodiments as defined by the claims.

[0296] The purpose of this disclosure is to collect, manage, and process any personal information of users in a manner that minimizes the risk of unintentional and / or unauthorized access and / or use.

[0297] Therefore, while this disclosure broadly covers the use of personal information to implement one or more embodiments, it is also contemplated that embodiments can also be implemented without access to such personal information.

Claims

1. A method, the method comprising: At an electronic device that communicates with one or more input devices and one or more output devices: When navigating to a physical location: Based on the current location of the electronic device being navigated, which includes a predetermined path, navigation is performed along the path, while abandoning the output of a user interface via the one or more output devices for selecting the appropriate path to be navigated along, which includes the current location of the electronic device. as well as Based on the determination that the current location of the electronic device, which is navigating, does not include the predetermined path: The user interface is output via the one or more output devices for selecting the corresponding path along which the electronic device will be navigated, including the current location of the electronic device; as well as When the user interface is used to select the corresponding path, user input for selecting the corresponding path is received via the one or more input devices; as well as In response to receiving the input that selects the appropriate path: Based on the determination that the input corresponds to a request to select a first path, navigation is performed using the first path; as well as Based on the determination that the input corresponds to a request to select a second path different from the first path, navigation is performed using the second path.

2. The method according to claim 1, wherein: Based on the determination that the relationship between the size of the vehicle communicating with the electronic device and the size of the vehicle's environment is a first relationship, the user interface for selecting the corresponding path includes an indication of a third path corresponding to the first relationship, and Based on the determination that the relationship between the size of the vehicle and the size of the environment of the vehicle is a second relationship different from the first relationship, the user interface for selecting the corresponding path includes an indication of a fourth path corresponding to the second relationship, the fourth path being different from the third path.

3. The method according to any one of claims 1 to 2, wherein: Based on the determination that the weather conditions at the current location of the electronic device are a first weather condition, the user interface for selecting the corresponding path includes a third path corresponding to the first weather condition. Based on the determination that the weather conditions at the current location of the electronic device are a second weather condition different from the first weather condition, the user interface for selecting the corresponding path includes a fourth path corresponding to the second weather condition, which is different from the first path.

4. The method according to any one of claims 1 to 3, wherein: The user interface for selecting the corresponding path includes prompts for the user to create the corresponding path, and The user input includes input corresponding to the creation of the corresponding path.

5. The method according to any one of claims 1 to 4, wherein the user input for selecting the corresponding path includes voice input.

6. The method according to any one of claims 1 to 5, wherein: Based on the determination that the vehicle communicating with the electronic device previously followed a third path including the vehicle's current location, the user interface for selecting the appropriate path includes an indication of a fourth path based on the third path, and Based on a determination that the vehicle previously followed a fifth path including the vehicle's current location, the user interface for selecting the corresponding path includes an indication of a sixth path based on the fifth path, which is different from the third path and different from the fourth path.

7. The method according to any one of claims 1 to 6, wherein: Based on a determination that the user of the electronic device previously followed a third path including the current location of the electronic device, the user interface for selecting the corresponding path includes an indication of a fourth path based on the third path, and Based on a fifth path determined to have been previously followed by the user including the current location of the electronic device, the user interface for selecting the corresponding path includes an indication of a sixth path based on the fifth path, which is different from the third path and different from the fourth path.

8. The method according to any one of claims 1 to 7, wherein: Based on the determination that the second electronic device previously followed a third path including the current location of the electronic device, the user interface for selecting the corresponding path includes an indication of a fourth path based on the third path, and Based on the determination that the second electronic device previously followed a fifth path including the current location of the electronic device, the user interface for selecting the corresponding path includes an indication of a sixth path based on the fifth path, which is different from the third path and the sixth path is different from the fourth path.

9. The method according to any one of claims 1 to 8, further comprising: Receive path information of the current location of the electronic device from a second electronic device associated with the current location of the electronic device, wherein when navigating to the physical location: Based on the determination that the current location of the electronic device, which is navigating, does not include the predetermined path: The user interface for selecting the corresponding path, based on the path information determined from the second electronic device, includes first path information and includes an indication of a third path based on the first path information. The user interface for selecting the corresponding path, based on the path information determined from the second electronic device, includes second path information that is different from the first path information, and the user interface includes an indication of a fourth path based on the second path information, the fourth path being different from the third path.

10. The method according to any one of claims 1 to 9, further comprising: When navigating to the physical location: Objects in the environment of the electronic device are detected via the one or more input devices, and these objects are not included in map information accessible to the electronic device. The map information is updated to include information associated with the object based on whether the object meets one or more criteria. as well as If it is determined that the object does not meet one or more of the criteria, the update of the map information to include the information associated with the object is abandoned.

11. The method according to any one of claims 1 to 10, further comprising: When navigating to the physical location: At the first moment, objects in the environment of the electronic device are detected via the one or more input devices, and these objects are not included in the map information accessible to the electronic device; The object is detected via the one or more input devices at a second time, different from the first time. as well as The object in the environment of the electronic device is tracked based on the detection of the object at the first time and the second time.

12. The method according to any one of claims 1 to 11, wherein: The user interface for selecting the corresponding path includes a representation of one or more path options. Based on the set of criteria that determine one or more first path options satisfy, and the set of one or more second path options that do not satisfy the criteria, the user interface for selecting the corresponding path includes a representation of the one or more first path options. The user interface for selecting the corresponding path includes a representation of the one or more second path options, based on the set of criteria that determine the one or more second path options satisfy and the set of criteria that determine the one or more first path options do not satisfy.

13. The method according to any one of claims 1 to 12, wherein: The user interface for selecting the corresponding path includes a representation of one or more path options. Based on the determination that the vehicle in the environment of the electronic device has a first behavior, the user interface for selecting the corresponding path includes a representation of one or more first path options, and Based on the determination that the vehicle in the environment of the electronic device has a second behavior different from the first behavior, the user interface for selecting the corresponding path includes a representation of one or more second path options different from the one or more first path options.

14. The method according to any one of claims 1 to 13, further comprising: When navigating to the physical location, and when the current location of the electronic device does not include the predetermined path, the one or more input devices are used to detect a person providing route guidance in the environment of the electronic device, wherein: The user interface for selecting the corresponding path includes a representation of one or more path options. The user interface for selecting the corresponding path, based on the path guidance provided by the person, includes a first path guidance and a representation of one or more first path options corresponding to the first path guidance. The user interface for selecting the corresponding path, based on the determination that the path guidance provided by the person includes a second path guidance different from the first path guidance, includes a representation of one or more second path options corresponding to the second path guidance, different from the first path options.

15. The method according to claim 14, further comprising: When navigating to the physical location, and when the current location of the electronic device does not include the predetermined path: If it is determined that the path guidance provided by the person fails to meet one or more criteria, a user interface for selecting the corresponding path is presented, including a request for user input to select the corresponding path.

16. The method of claim 15, wherein: Based on the determination that the route guidance provided by the person failed to meet one or more of the criteria: The user interface for selecting the corresponding path, presented in response to a user input request, includes options and indications of the corresponding path using the one or more output devices. When an option is selected, the electronic device abandons navigation along the corresponding path. The method further includes: when navigating to the physical location: Navigation proceeds along the corresponding path based on the determination that the user interface used to select the corresponding path has been presented for more than a predetermined time threshold.

17. The method according to any one of claims 15 to 16, wherein: Based on the determination that the route guidance provided by the person failed to meet one or more of the criteria: The user interface for selecting the corresponding path, which includes a request for user input to select the corresponding path, presents multiple path options based on the path guidance provided by the user.

18. The method according to any one of claims 15 to 17, wherein: Based on the determination that the route guidance provided by the person failed to meet one or more of the criteria: The user interface for selecting the corresponding path, which includes presenting a prompt to the user to provide the corresponding path, includes the user inputting the request for selecting the corresponding path.

19. The method according to any one of claims 1 to 18, the method further comprising: When navigating to the physical location, it is determined that the current location of the electronic device does not include the predetermined path: Navigation is performed along the path that the vehicle is traveling along, based on the determination that the vehicle in the environment of the electronic device is traveling along a path that meets one or more criteria; as well as Based on the determination that the vehicle in the environment of the electronic device is traveling along a path that fails to meet one or more of the criteria, the user interface is output for selecting the appropriate path.

20. The method according to any one of claims 1 to 19, the method further comprising: When navigating to the physical location, it is determined that the current location of the electronic device does not include the predetermined path: Navigation is performed along the path indicated by one or more objects, based on the determination that the environment of the electronic device includes one or more objects indicating a path; and Based on the determination that the environment of the electronic device does not include the one or more objects indicating the path, the user interface for selecting the corresponding path is output.

21. The method according to any one of claims 1 to 20, wherein outputting the user interface for selecting the corresponding path includes displaying a visual indication of the corresponding path via a display component included in the one or more output devices.

22. The method according to any one of claims 1 to 21, wherein: The user interface for selecting the corresponding path includes a representation of one or more path options. Based on the determination that the environment of the electronic device is a first type of environment, the one or more path options are selected according to a first set of one or more criteria, and Based on the determination that the environment of the electronic device is a second type of environment different from the first type of environment, the one or more path options are selected based on a second set of one or more criteria different from the first set of criteria.

23. An electronic device, the electronic device comprising: One or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: When navigating to a physical location: Based on the current location of the electronic device being navigated, which includes a predetermined path, navigation is performed along the path, while abandoning the output of a user interface via one or more output devices for selecting the corresponding path to be navigated along, which includes the current location of the electronic device. as well as Based on the determination that the current location of the electronic device, which is navigating, does not include the predetermined path: The user interface is output via the one or more output devices for selecting the corresponding path along which the electronic device will be navigated, including the current location of the electronic device; as well as When the user interface is used to select the corresponding path, user input for selecting the corresponding path is received via the one or more input devices; as well as In response to receiving the input that selects the appropriate path: Based on the determination that the input corresponds to a request to select a first path, navigation is performed using the first path; as well as Based on the determination that the input corresponds to a request to select a second path different from the first path, navigation is performed using the second path.

24. A non-transitory computer-readable storage medium storing one or more programs, said one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform a method comprising the following operations: When navigating to a physical location: Based on the current location of the electronic device being navigated, which includes a predetermined path, navigation is performed along the path, while abandoning the output of a user interface via one or more output devices for selecting the corresponding path to be navigated along, which includes the current location of the electronic device. as well as Based on the determination that the current location of the electronic device, which is navigating, does not include the predetermined path: The user interface is output via the one or more output devices for selecting the corresponding path along which the electronic device will be navigated, including the current location of the electronic device; as well as When the user interface is used to select the corresponding path, user input for selecting the corresponding path is received via the one or more input devices; as well as In response to receiving the input that selects the appropriate path: Based on the determination that the input corresponds to a request to select a first path, navigation is performed using the first path; as well as Based on the determination that the input corresponds to a request to select a second path different from the first path, navigation is performed using the second path.

25. An electronic device communicating with a display component and one or more input devices, the electronic device comprising: One or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any one of the methods according to claims 1 to 22.

26. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device in communication with a display component and one or more input devices, cause the electronic device to perform any one of the methods according to claims 1 to 22.

27. An electronic device for communicating with one or more output devices, the electronic device comprising: One or more processors; Memory; and Components used to perform the following operations when navigating to a physical location: Based on the current location of the electronic device being navigated, which includes a predetermined path, navigation is performed along the path, while abandoning the output of a user interface via the one or more output devices for selecting the appropriate path to be navigated along, which includes the current location of the electronic device. as well as Based on the determination that the current location of the electronic device, which is navigating, does not include the predetermined path: The user interface is output via the one or more output devices for selecting the corresponding path along which the electronic device will be navigated, including the current location of the electronic device; And when the user interface is output for selecting the corresponding path, user input for selecting the corresponding path is received via the one or more input devices; as well as In response to receiving the input that selects the appropriate path: Based on the determination that the input corresponds to a request to select a first path, navigation is performed using the first path; as well as Based on the determination that the input corresponds to a request to select a second path different from the first path, navigation is performed using the second path.

28. An electronic device for communicating with one or more output devices, the electronic device comprising: One or more processors; Memory; and Components for performing any one of the methods according to claims 1 to 22.

Images