User interface for translation
By automatically detecting and outputting spoken audio translations in electronic devices, combined with speech recognition and display transcription, the complex and inefficient translation management in existing technologies is solved, achieving more efficient translation management and extended battery life.
Patent Information
- Application Number
- CN202610673149.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2026-01-15
- Filing Date
- 2026-05-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing translation management methods for electronic devices are complex and inefficient, wasting user time and device energy, especially in battery-powered devices.
By automatically detecting spoken audio and outputting translations in a computer system, and utilizing audio devices and display generation components, real-time speech translation and transcription without additional user input is achieved. Speech recognition technology is combined to insert punctuation to optimize the translation interface.
It provides a faster and more efficient translation management method, reduces the cognitive burden on users, saves device power, extends battery life, and improves the user experience.
Smart Images

Figure CN123114602A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates generally to computer user interfaces, and more specifically to techniques for managing translations. Background Technology
[0002] Computer systems are used to translate text from one language to another, enabling individuals who do not understand the specific language in which the document was written to access a translated version of the document. Summary of the Invention
[0003] However, some technologies used to manage translations using electronic devices are often cumbersome and inefficient. For example, some existing technologies use complex and time-consuming user interfaces that may include multiple buttons or keystrokes. These technologies require more time than necessary, resulting in wasted user time and device energy. This latter consideration is particularly important in battery-powered devices.
[0004] Therefore, this technology provides electronic devices with faster and more efficient methods and interfaces for managing translations. Such methods and interfaces can optionally supplement or replace other methods used for managing translations. These methods and interfaces reduce the cognitive burden on users and result in more efficient human-computer interfaces. For battery-powered computing devices, such methods and interfaces save power and increase the time between battery charging cycles.
[0005] A method is described. The method is performed at a computer system communicating with one or more audio devices, one or more input devices, and one or more display generation components. The method includes: when not operating in a translation mode where spoken audio is automatically detected via one or more audio devices and a translation of the detected spoken audio is output; and in response to the detection of the set of one or more inputs for enabling the translation mode, operating in the translation mode, the operation including, without requiring additional user input: detecting speech of a conversation of two or more participants via one or more audio devices, the speech including a first speech in a first language and a second speech in a second language different from the first language; outputting an audio translation of the first speech and / or an audio translation of the second speech via one or more audio devices; and displaying a conversation transcript via one or more display generation components and concurrently with the output of the audio translation of the conversation, the conversation transcript including a transcript of the first speech and the second speech of the conversation of two or more participants.
[0006] A non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with one or more audio devices, one or more input devices, and one or more display generation components. The one or more programs include instructions for: detecting a set of one or more inputs via one or more input devices to enable the translation mode when not operating in a translation mode that automatically detects spoken audio via one or more audio devices and outputs a translation of the detected spoken audio; and operating in the translation mode in response to detecting the set of one or more inputs to enable the translation mode, the operation including, without requiring additional user input: detecting speech of a conversation of two or more participants via one or more audio devices, the speech including a first speech in a first language and a second speech in a second language different from the first language; outputting an audio translation of the first speech and / or an audio translation of the second speech via one or more audio devices; and displaying a conversation transcript via one or more display generation components and concurrently with the output of the audio translation of the conversation, the conversation transcript including a transcript of the first and second speech of the conversation of two or more participants.
[0007] A transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with one or more audio devices, one or more input devices, and one or more display generation components. The one or more programs include instructions for: detecting a set of one or more inputs via one or more input devices to enable the translation mode when not operating in a translation mode that automatically detects spoken audio via one or more audio devices and outputs a translation of the detected spoken audio; and operating in the translation mode in response to detecting the set of one or more inputs to enable the translation mode, the operation including, without requiring additional user input: detecting speech of a conversation of two or more participants via one or more audio devices, the speech including a first speech in a first language and a second speech in a second language different from the first language; outputting an audio translation of the first speech and / or an audio translation of the second speech via one or more audio devices; and displaying a conversation transcript via one or more display generation components and concurrently with the output of the audio translation of the conversation, the conversation transcript including a transcript of the first speech and the second speech of the conversation of two or more participants.
[0008] A computer system is described. The computer system is configured to communicate with one or more audio devices, one or more input devices, and one or more display generation components, and the computer system includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: detecting a set of one or more inputs via one or more input devices to enable the translation mode when not operating in a translation mode that automatically detects spoken audio via one or more audio devices and outputs a translation of the detected spoken audio; and operating in the translation mode in response to detecting the set of one or more inputs to enable the translation mode, the operation including, without requiring additional user input: detecting speech of a conversation of two or more participants via one or more audio devices, the speech including a first speech in a first language and a second speech in a second language different from the first language; outputting an audio translation of the first speech and / or an audio translation of the second speech via one or more audio devices; and displaying a conversation transcript via one or more display generation components and concurrently with the output of the audio translation of the conversation, the conversation transcript including a transcript of the first speech and the second speech of the conversation of two or more participants.
[0009] A computer system is described. The computer system is configured to communicate with one or more audio devices, one or more input devices, and one or more display generation components, and the computer system includes: components for detecting a set of one or more inputs via one or more input devices to enable the translation mode when not operating in a translation mode that automatically detects spoken audio via one or more audio devices and outputs a translation of the detected spoken audio; and components for operating in the translation mode in response to detecting the set of one or more inputs to enable the translation mode, the components including, without requiring additional user input: components for detecting speech of a conversation of two or more participants via one or more audio devices, the speech including a first speech in a first language and a second speech in a second language different from the first language; components for outputting an audio translation of the first speech and / or an audio translation of the second speech via one or more audio devices; and components for displaying a conversation transcript via one or more display generation components and concurrently with the output of the audio translation of the conversation, the conversation transcript including a transcript of the first speech and the second speech of a conversation of two or more participants.
[0010] A computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system communicating with one or more audio devices, one or more input devices, and one or more display generation components. The one or more programs include instructions for: detecting a set of one or more inputs via one or more input devices to enable the translation mode when not operating in a translation mode that automatically detects spoken audio via one or more audio devices and outputs a translation of the detected spoken audio; and operating in the translation mode in response to detecting the set of one or more inputs to enable the translation mode, the operation including, without requiring additional user input: detecting speech of a conversation of two or more participants via one or more audio devices, the speech including a first speech in a first language and a second speech in a second language different from the first language; outputting an audio translation of the first speech and / or an audio translation of the second speech via one or more audio devices; and displaying a conversation transcript via one or more display generation components and concurrently with the output of the audio translation of the conversation, the conversation transcript including a transcript of the first speech and the second speech of the conversation of two or more participants.
[0011] A method is described. The method is performed at a computer system communicating with one or more audio devices. The method includes: receiving an input audio stream including speech; and upon receiving the input audio stream including speech: performing speech processing on the input audio stream, including performing speech recognition to generate text based on the speech of the input audio stream; determining a position for inserting punctuation within the text, wherein the position is not the end of a sentence; inserting the punctuation at the determined position within the text; and before detecting the end of the sentence and based on the inserted punctuation at the determined position within the text, beginning to output spoken audio of a first portion of the text preceding the inserted punctuation via the one or more audio devices.
[0012] A non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with one or more audio devices, the one or more programs including instructions for: receiving an input audio stream including speech; and upon receiving the input audio stream including speech: performing speech processing on the input audio stream, including performing speech recognition to generate text based on the speech of the input audio stream; determining a position for inserting punctuation within the text, wherein the position is not the end of a sentence; inserting punctuation at the determined position within the text; and before detecting the end of a sentence and based on the inserted punctuation at the determined position within the text, beginning to output spoken audio of a first portion of the text preceding the inserted punctuation via the one or more audio devices.
[0013] A transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with one or more audio devices, the one or more programs including instructions for: receiving an input audio stream including speech; and upon receiving the input audio stream including speech: performing speech processing on the input audio stream, including performing speech recognition to generate text based on the speech of the input audio stream; determining a position for inserting punctuation within the text, wherein the position is not the end of a sentence; inserting punctuation at the determined position within the text; and before detecting the end of a sentence and based on the inserted punctuation at the determined position within the text, beginning to output spoken audio of a first portion of the text preceding the inserted punctuation via the one or more audio devices.
[0014] A computer system is described. The computer system is configured to communicate with one or more audio devices, and the computer system includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: receiving an input audio stream including speech; and upon receiving the input audio stream including speech: performing speech processing on the input audio stream, including performing speech recognition to generate text based on the speech of the input audio stream; determining a position for inserting punctuation within the text, wherein the position is not the end of a sentence; inserting punctuation at the determined position within the text; and before detecting the end of a sentence and based on the inserted punctuation at the determined position within the text, starting to output spoken audio of a first portion of the text preceding the inserted punctuation via the one or more audio devices.
[0015] A computer system is described. The computer system is configured to communicate with one or more audio devices, and the computer system includes: components for receiving an input audio stream including speech; and components for performing the following operations upon receiving the input audio stream including speech: performing speech processing on the input audio stream, including performing speech recognition to generate text based on the speech of the input audio stream; determining a position for inserting punctuation within the text, wherein the position is not the end of a sentence; inserting punctuation at the determined position within the text; and before detecting the end of the sentence and based on the inserted punctuation at the determined position within the text, starting to output spoken audio of a first portion of the text preceding the inserted punctuation via the one or more audio devices.
[0016] A computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system communicating with one or more audio devices, the one or more programs including instructions for: receiving an input audio stream including speech; and upon receiving the input audio stream including speech: performing speech processing on the input audio stream, including performing speech recognition to generate text based on the speech of the input audio stream; determining a position for inserting punctuation within the text, wherein the position is not the end of a sentence; inserting punctuation at the determined position within the text; and before detecting the end of a sentence and based on the inserted punctuation at the determined position within the text, beginning to output spoken audio of a first portion of the text preceding the inserted punctuation via the one or more audio devices.
[0017] Executable instructions for performing these functions may optionally be included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.
[0018] Therefore, faster and more efficient methods and interfaces are provided for managing translations on these devices, thereby improving their effectiveness, efficiency, and user satisfaction. These methods and interfaces can complement or replace other methods used for managing translations. Attached Figure Description
[0019] To better understand the various described embodiments, reference should be made to the following detailed description in conjunction with the accompanying drawings, wherein the same reference numerals in all the drawings indicate corresponding parts.
[0020] Figure 1A This is a block diagram illustrating a portable multi-functional device with a touch-sensitive display according to some implementation schemes.
[0021] Figure 1B This is a block diagram illustrating exemplary components for event handling according to some implementation schemes.
[0022] Figure 2 Examples of portable multi-functional devices with touchscreens according to some implementation schemes are shown.
[0023] Figure 3A This is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface according to some implementation schemes.
[0024] Figures 3B to 3G This example demonstrates how to perform an operation using an Application Programming Interface (API).
[0025] Figure 4A An exemplary user interface for a menu applied to a portable multi-functional device, according to some implementation schemes, is illustrated.
[0026] Figure 4B An exemplary user interface for a multifunctional device having a touch-sensitive surface separate from the display is illustrated according to some embodiments.
[0027] Figure 5A Examples of personal electronic devices according to some implementation schemes are shown.
[0028] Figure 5B This is a block diagram illustrating a personal electronic device according to some implementation schemes.
[0029] Figures 6A to 6Y An exemplary user interface for managing translations is illustrated according to some implementation schemes.
[0030] Figure 7 This is a flowchart illustrating a method for managing translation based on some implementation plans.
[0031] Figure 8 This is a flowchart illustrating a method for outputting audio according to some implementation schemes. Detailed Implementation
[0032] The following description illustrates exemplary methods, parameters, etc. However, it should be understood that such description is not intended to limit the scope of this disclosure, but is provided as a description of exemplary embodiments.
[0033] Electronic devices need to provide more efficient methods and interfaces for managing translations. For example, accessing both audio and visual versions of translated text or audio would be beneficial. Such technologies can reduce the cognitive burden on users accessing translations, thereby increasing productivity. Furthermore, such technologies can reduce processor and battery power wasted otherwise on redundant user input.
[0034] under Figures 1A to 1B , Figure 2 , Figures 3A to 3G , Figures 4A to 4B and Figures 5A to 5B A description of an exemplary device for performing techniques for managing translation is provided. Figures 6A to 6Y An exemplary user interface for managing translations is illustrated according to some implementation schemes. Figure 7 This is a flowchart illustrating a method for managing translation based on some implementation plans. Figure 8 This is a flowchart illustrating a method for outputting audio according to some implementation schemes. Figures 6A to 6Y The user interface in the document is used to illustrate the processes described below, including Figure 7 and Figure 8 The process in.
[0035] The processes described below enhance device operability and make user-device interfaces more efficient through various technologies (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with the device), including providing improved visual feedback to users, reducing the amount of input required to perform operations, providing additional control options without cluttering the user interface with additional displayed controls, performing operations without requiring further user input when a set of conditions are met, and / or additional technologies. These technologies also reduce power consumption and extend device battery life by enabling users to use the device faster and more efficiently.
[0036] Furthermore, in a method described herein where one or more steps depend on the satisfaction of one or more conditions, it should be understood that the described method can be repeated in multiple repetitions such that, during the repetitions, all conditions determining the steps in the method are satisfied in different repetitions of the method. For example, if the method requires performing a first step (if the conditions are satisfied) and a second step (if the conditions are not satisfied), those skilled in the art will know that the stated steps are repeated until both conditions are satisfied and not satisfied (in no particular order). Thus, a method described as having one or more steps depending on the satisfaction of one or more conditions can be rewritten as a method that repeats until each condition described in the method is satisfied. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing discretionary operations based on the satisfaction of the corresponding one or more conditions, and thus to determine whether possible conditions have been satisfied without explicitly repeating the steps of the method until all conditions determining the steps in the method are satisfied. Those skilled in the art will also understand that, similar to a method having discretionary steps, a system or computer-readable storage medium can repeat the steps of the method multiple times as needed to ensure that all discretionary steps have been performed.
[0037] Although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. In some embodiments, these terms are used to distinguish one element from another. For example, a first touch may be referred to as a second touch, and similarly, a second touch may be referred to as a first touch, without departing from the scope of the various described embodiments. In some embodiments, a first touch and a second touch are two separate references to the same touch. In some embodiments, both a first touch and a second touch are touches, but they are not the same touch.
[0038] As used herein, the phrase “one or more of A and / or B” is interpreted to include all combinations of A and B, including but not limited to: A alone without B; B alone without A; and combinations of A and B. The phrase “one or more of A, B, and / or C” is interpreted to include all combinations of A, B, and C, including but not limited to: A alone without B and C; B alone without A and C; C alone without A and B; and any combination of A, B, and / or C (e.g., A and B without C; A and C without B; B and C without A; and / or A, B, and C). Additionally, as used herein, the phrase “a group consisting of A, B, C, and combinations thereof” and the phrase “at least one of A, B, and C” should be interpreted to have the same meaning as the phrase “one or more of A, B, and / or C” as defined above. As used herein, the phrases “at least one of A, B, or C” and “one or more of A, B, or C” should be interpreted as having the same meaning as the phrase “one or more of A, B, and / or C” as defined above. As used herein, the phrase “including all combinations of A, B, and C” is interpreted as including combinations of all listed elements (e.g., combinations of A, B, and C).
[0039] The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and in the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “comprising” and / or “including” as used in this specification specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.
[0040] Depending on the context, the term "if" may optionally be interpreted as meaning "when," "in," or "in response to determination" or "in response to detection." Similarly, depending on the context, the phrases "if it is determined..." or "if [the stated condition or event] is detected" may optionally be interpreted as meaning "in response to determination..." or "in response to detection of [the stated condition or event]."
[0041] This document describes implementations of electronic devices, user interfaces for such devices, and associated processes for using such devices. In some implementations, the device is a portable communication device, such as a mobile phone, that also includes other functionalities such as PDA and / or music player functionality. Exemplary implementations of portable multi-functional devices include, but are not limited to, the iPhone from Apple Inc. of Cupertino, California. ® Devices, iPod Touch ® Devices and iPads ® Device. Alternatively, other portable electronic devices may be used, such as laptops or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). In some embodiments, the electronic device is a computer system that communicates (e.g., via wireless or wired communication) with a display generating component (e.g., a display device such as a head-mounted display (HMD), monitor, projector, touchscreen display; or other devices or components that present visual content to a user, such as visual content generated on or in or from the display generating component and visible elsewhere). The display generating component is configured to provide visual output, such as display via a CRT monitor, via an LED monitor, or via image projection. In some embodiments, the display generating component is integrated with the computer system. In some embodiments, the display generating component is separate from the computer system. As used herein, “display” content includes content (e.g., video data rendered or decoded by display controller 156) that is visually generated by sending data (e.g., image data or video data) to an integrated or external display generating component via a wired or wireless connection.
[0042] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device may optionally include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.
[0043] The device typically supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, game applications, telephone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or digital video player applications.
[0044] Various applications running on this device may optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device may optionally be adjusted and / or varied for different applications, and / or within the respective applications. In this way, the shared physical architecture of the device (such as the touch-sensitive surface) may optionally support a variety of applications using a user interface that is intuitive and clear to the user.
[0045] Now let’s turn our attention to implementation schemes for portable devices with touch-sensitive displays. Figure 1A This is a block diagram illustrating a portable multi-functional device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a “touchscreen” for convenience, and is sometimes referred to as or called a “touch-sensitive display system.” Device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of device 100). Device 100 may optionally include one or more haptic output generators 167 for generating haptic outputs on device 100 (e.g., generating haptic outputs on a haptic surface such as the haptic display system 112 of device 100 or the touchpad 355 of device 300). These components may optionally communicate via one or more communication buses or signal lines 103.
[0046] As used in this specification and claims, the term "intensity" of contact on a tactile surface refers to the force or pressure (force per unit area) of a contact (e.g., finger contact) on a tactile surface, or to a substitute (alternative) for the force or pressure of a contact on a tactile surface. The intensity of contact has a range of values that includes at least four different values and more typically hundreds of different values (e.g., at least 256). The intensity of contact may optionally be determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the tactile surface may optionally be used to measure the force at different points on the tactile surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted average) to determine the estimated contact force. Similarly, the pressure-sensitive tip of a stylus may optionally be used to determine the pressure of the stylus on the tactile surface. Alternatively, the size and / or change of the contact area detected on the touch-sensitive surface, the capacitance and / or change of the touch-sensitive surface adjacent to the contact, and / or the resistance and / or change of the touch-sensitive surface adjacent to the contact may be used as substitutes for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement result). In some embodiments, the substitute measurement of the contact force or pressure is translated into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of the contact as an attribute of user input allows users to access additional device functions that would otherwise be inaccessible to the user on a smaller device with limited physical space, which is used (e.g., on a touch-sensitive display) to display an indication and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls, such as knobs or buttons).
[0047] As used in this specification and claims, the term "tactile output" refers to a physical displacement of the device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device, detected by the user using the user's tactile sense. For example, when the device or a component of the device comes into contact with a touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or touchpad) may optionally be interpreted by the user as a "press-click" or "release-click" on a physically actuated button. In some cases, the user will feel a tactile sensation, such as a "press-click" or "release-click," even when a physically actuated button associated with the touch-sensitive surface, which has been physically pressed (e.g., displaced) by the user's movement, does not move. As another example, even when the smoothness of the tactile surface remains unchanged, the movement of the tactile surface can optionally be interpreted or sensed by the user as the "roughness" of the tactile surface. While such interpretations of touch by users will be limited by the individualized sensory perceptions of the user, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a specific sensory perception of the user (e.g., "release click", "press click", "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or its components that will generate the sensory perception described by a typical (or common) user.
[0048] It should be understood that device 100 is merely an example of a portable multifunctional device, and device 100 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of these components. Figure 1A The various components shown are implemented in hardware, software, or a combination of both, including one or more signal processing and / or application-specific integrated circuits.
[0049] Memory 102 may optionally include high-speed random access memory, and may also optionally include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 may optionally control access to memory 102 by other components of device 100.
[0050] Peripheral interface 118 can be used to couple input and output peripherals of the device to CPU 120 and memory 102. The one or more processors 120 run or execute various software programs (such as computer programs (e.g., including instructions)) and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data. In some embodiments, peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.
[0051] RF (Radio Frequency) circuit 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 108 translates electrical signals into electromagnetic signals / electromagnetic signals into electrical signals and communicates with communication networks and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, subscriber identity module (SIM) cards, memory, etc. RF circuit 108 optionally communicates wirelessly with networks (such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs))) and other devices. RF circuit 108 optionally includes well-known circuitry for detecting near-field communication (NFC) fields, such as via short-range communication radio components. Wireless communication may optionally use any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed Downlink Packet Access (HSDPA), High-Speed Uplink Packet Access (HSUPA), Evolution, Pure Data (EV-DO), HSPA, HSPA+, Dual-Unit HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), and Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Messaging Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence with Extended Utility (SIMPLE), Instant Messaging and Presence Service (IMPS)) and / or Short Message Service (SMS), or any other suitable communication protocol that has not been developed as of the date of this document submission.
[0052] Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and device 100. Audio circuitry 110 receives audio data from peripheral interface 118, translates the audio data into electrical signals, and sends the electrical signals to speaker 111. Speaker 111 translates the electrical signals into sound waves that are audible to humans. Audio circuitry 110 also receives electrical signals translated from sound waves by microphone 113. Audio circuitry 110 translates the electrical signals into audio data and sends the audio data to peripheral interface 118 for processing. Audio data may optionally be retrieved by peripheral interface 118 from and / or sent to memory 102 and / or RF circuitry 108. In some embodiments, audio circuitry 110 also includes a headset jack (e.g., ...). Figure 2 (212 in the text). The headset jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headphone or a headset with both outputs (e.g., a mono-ear or binaural headset) and inputs (e.g., a microphone).
[0053] I / O subsystem 106 couples input / output peripherals (such as touchscreen 112 and other input control devices 116) on device 100 to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, optical sensor controller 158, depth camera controller 169, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive electrical signals from / transmit electrical signals to the other input control device 116. Other input control devices 116 optionally include physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, joysticks, click dials, etc. In some embodiments, input controller 160 may optionally be coupled to (or not coupled to) any of the following: keyboard, infrared port, USB port, and pointing device such as mouse. One or more buttons (e.g., ... Figure 2 Optionally, 208 may include increase / decrease buttons for volume control of speaker 111 and / or microphone 113. These one or more buttons may optionally include a push-button (e.g., Figure 2(Ref. 206 in the original text). In some embodiments, the electronic device is a computer system that communicates with one or more input devices (e.g., via wireless communication, via wired communication). In some embodiments, the one or more input devices include a touch-sensitive surface (e.g., a touchpad, as part of a touch-sensitive display). In some embodiments, the one or more input devices include one or more camera sensors (e.g., one or more optical sensors 164 and / or one or more depth camera sensors 175), such as for tracking user gestures (e.g., hand gestures and / or air gestures) as input. In some embodiments, the one or more input devices are integrated with the computer system. In some embodiments, the one or more input devices are separate from the computer system. In some implementations, air gestures are gestures detected without the user touching an input element that is part of the device (or independently of an input element that is part of the device) and based on the detected movement of a part of the user's body through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another of the user's fingers or a part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tapping gesture that includes the hand moving a predetermined amount and / or speed in a predetermined pose, or a shaking gesture that includes a predetermined speed or amount of rotation of a part of the user's body)).
[0054] A rapid press of the push button may optionally disengage the touchscreen 112 from its lock or optionally initiate a process of unlocking the device using gestures on the touchscreen, as described in U.S. Patent Application 11 / 322,549 (i.e., U.S. Patent No. 7,657,849), filed December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," the entire contents of which are incorporated herein by reference. A long press of the push button (e.g., 206) may optionally power on or off the device 100. The function of one or more of these buttons may optionally be user-customizable. The touchscreen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.
[0055] The touch-sensitive display 112 provides input and output interfaces between the device and the user. The display controller 156 receives electrical signals from and / or transmits electrical signals to the touchscreen 112. The touchscreen 112 displays visual output to the user. Visual output may optionally include graphics, text, icons, video, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output may optionally correspond to user interface objects.
[0056] Touchscreen 112 has a touch-sensitive surface, sensor, or sensor array that accepts input from a user based on tactile and / or haptic contact. Touchscreen 112 and display controller 156 (along with any associated modules and / or instruction set in memory 102) detect contact on touchscreen 112 (and any movement or interruption of that contact) and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touchscreen 112. In an exemplary embodiment, the contact point between touchscreen 112 and the user corresponds to the user's finger.
[0057] Touchscreen 112 may optionally employ LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, but other display technologies may be used in other embodiments. Touchscreen 112 and display controller 156 may optionally use any of a variety of touch sensing technologies now known or to be developed hereafter, along with other proximity sensor arrays or other elements for determining one or more points of contact with touchscreen 112, to detect contact and any movement or interruption thereto. These various touch sensing technologies include, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. In an exemplary embodiment, projected mutual capacitance sensing technology, such as that used in the iPhone from Apple Inc. (Cupertino, California), is used. ® and iPod Touch ® The technology used.
[0058] In some embodiments of the touchscreen 112, the touch-sensitive display may optionally resemble a multi-touch-sensitive touchpad described in the following U.S. patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman) and / or U.S. Patent Publication 2002 / 0015024A1, each of which is incorporated herein by reference in its entirety. However, the touchscreen 112 displays visual output from the device 100, while the touch-sensitive touchpad does not provide visual output.
[0059] The touch-sensitive display in some embodiments of the touchscreen 112 is described in the following applications: (1) U.S. Patent Application No. 11 / 381,313, filed May 2, 2006, “Multipoint Touch Surface Controller”; (2) U.S. Patent Application No. 10 / 840,862, filed May 6, 2004, “Multipoint Touchscreen”; (3) U.S. Patent Application No. 10 / 903,964, filed July 30, 2004, “Gestures For Touch Sensitive Input Devices”; (4) U.S. Patent Application No. 11 / 048,264, filed January 31, 2005, “Gestures For Touch Sensitive Input Devices”; and (5) U.S. Patent Application No. 11 / 038,590, filed January 18, 2005, “Mode-Based Graphical User Interfaces For Touch Sensitive Input”. (6) U.S. Patent Application No. 11 / 228,758, filed September 16, 2005, “Virtual Input Device Placement On A Touch Screen User Interface”; (7) U.S. Patent Application No. 11 / 228,700, filed September 16, 2005, “Operation Of A Computer With A Touch Screen Interface”; (8) U.S. Patent Application No. 11 / 228,737, filed September 16, 2005, “Activating Virtual Keys Of ATouch-Screen Virtual Keyboard”; and (9) U.S. Patent Application No. 11 / 367,749, filed March 3, 2006, “Multi-Functional Hand-Held Device”. The full text of all these applications is incorporated herein by reference.
[0060] The touchscreen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 160 dpi. Users may optionally use any suitable object or accessory, such as a stylus, finger, etc., to interact with the touchscreen 112. In some embodiments, the user interface is designed to operate primarily through finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of a finger on the touchscreen. In some embodiments, the device translates coarse finger-based input into precise pointer / cursor positioning or commands for performing the user-desired actions.
[0061] In some embodiments, in addition to the touchscreen, device 100 may optionally include a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of the device that, unlike the touchscreen, does not display visual output. The touchpad may optionally be a touch-sensitive surface separate from the touchscreen 112, or an extension of the touch-sensitive surface formed by the touchscreen.
[0062] The device 100 also includes a power system 162 for supplying power to various components. The power system 162 may optionally include a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power in the portable device.
[0063] Device 100 may optionally include a secure element 163 for securely storing information. In some embodiments, secure element 163 is a hardware component (e.g., a secure microcontroller chip) configured to securely store data or algorithms. In some embodiments, secure element 163 provides (e.g., publishes) security information (e.g., payment information (e.g., account number and / or transaction-specific dynamic security code), identification information (e.g., credentials for nationally approved digital identifiers), and / or authentication information (e.g., data generated using a cryptographic engine and / or by performing asymmetric cryptographic operations)). In some embodiments, secure element 163 provides (or publishes) security information in response to device 100 receiving authorization, such authorization being user authentication (e.g., fingerprint authentication; password authentication; detection of a double press of a hardware button when device 100 is in an unlocked state, and optionally when device 100 has been continuously on the user's wrist since being unlocked by providing authentication credentials to device 100), wherein the continued presence of device 100 on the user's wrist is determined by periodically checking the contact between the device and the user's skin. For example, device 100 detects a fingerprint at its fingerprint sensor (e.g., a fingerprint sensor integrated into a button). Device 100 determines whether the detected fingerprint matches a registered fingerprint. Based on the determination that the fingerprint matches a registered fingerprint, security element 163 provides (e.g., publishes) security information. Based on the determination that the fingerprint does not match a registered fingerprint, security element 163 refrains from providing (e.g., publishing) security information.
[0064] The device 100 may optionally also include one or more optical sensors 164. Figure 1A An optical sensor 164 coupled to an optical sensor controller 158 in I / O subsystem 106 is shown. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with an imaging module 143 (also referred to as a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of device 100, facing away from a touchscreen display 112 on the front of the device, allowing the touchscreen display to be used as a viewfinder for still image and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device, allowing an image of the user to be optionally acquired for use in video conferencing while the user views other video conferencing participants on the touchscreen display. In some embodiments, the positioning of the optical sensor 164 can be changed by the user (e.g., by rotating the lenses and sensors within the device housing), allowing a single optical sensor 164 to be used in conjunction with the touchscreen display for both video conferencing and still image and / or video image acquisition.
[0065] The device 100 may optionally also include one or more depth camera sensors 175. Figure 1A A depth camera sensor is shown coupled to a depth camera controller 169 in I / O subsystem 106. Depth camera sensor 175 receives data from the environment to create a 3D model of an object (e.g., a face) within the scene from a viewpoint (e.g., the depth camera sensor). In some embodiments, in conjunction with imaging module 143 (also referred to as a camera module), depth camera sensor 175 may be used to determine depth maps of different portions of an image captured by imaging module 143. In some embodiments, the depth camera sensor is located at the front of device 100, such that user images with depth information can be optionally acquired for video conferencing while a user views other video conferencing participants on a touchscreen display, and selfies with depth map data can be captured. In some embodiments, depth camera sensor 175 is located at the rear of the device, or both the rear and front of device 100. In some embodiments, the positioning of depth camera sensor 175 can be changed by the user (e.g., by rotating a lens and sensor within the device housing), such that depth camera sensor 175 is used in conjunction with a touchscreen display for both video conferencing and still image and / or video image acquisition.
[0066] The device 100 may optionally also include one or more contact strength sensors 165. Figure 1A A contact strength sensor is shown coupled to a strength sensor controller 159 in I / O subsystem 106. The contact strength sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). The contact strength sensor 165 receives contact strength information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact strength sensor is arranged juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact strength sensor is located on the rear of device 100, opposite to the touchscreen display 112 located on the front of device 100.
[0067] The device 100 may optionally also include one or more proximity sensors 166. Figure 1AA proximity sensor 166 coupled to a peripheral device interface 118 is shown. Alternatively, the proximity sensor 166 may be coupled to an input controller 160 in an I / O subsystem 106. The proximity sensor 166 may be configured as described in the following U.S. patent applications: 11 / 241,839, entitled "Proximity Detector In Handheld Device"; 11 / 240,788, entitled "Proximity Detector In Handheld Device"; 11 / 620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; 11 / 586,862, entitled "Automated Response To And Sensing Of User Activity In Portable Devices"; and 11 / 638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals", the entire contents of which are incorporated herein by reference. In some implementations, when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call), the proximity sensor is turned off and the touchscreen 112 is disabled.
[0068] The device 100 may optionally also include one or more haptic output generators 167. Figure 1A A haptic output generator coupled to a haptic feedback controller 161 in I / O subsystem 106 is shown. The haptic output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components for converting electrical signals into haptic outputs on the device). A contact intensity sensor 165 receives haptic feedback generation instructions from a haptic feedback module 133 and generates a haptic output on device 100 that can be felt by a user of device 100. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a haptic surface (e.g., haptic display system 112) and optionally generates the haptic output by moving the haptic surface vertically (e.g., in / outward from the surface of device 100) or laterally (e.g., backward and forward in the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is located on the rear of the device 100, opposite to the touch screen display 112 located on the front of the device 100.
[0069] The device 100 may optionally also include one or more accelerometers 168. Figure 1A An accelerometer 168 coupled to a peripheral device interface 118 is shown. Alternatively, the accelerometer 168 may be coupled to an input controller 160 in an I / O subsystem 106. The accelerometer 168 may be configured as described in the following U.S. Patent Publications: 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and 20060017692, entitled "Methods And Apparatuses For Operating A Portable DeviceBased On An Accelerometer," both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from one or more accelerometers. Device 100 may optionally include, in addition to the accelerometer 168, a magnetometer and a GPS (or GLONASS or other global navigation system) receiver for obtaining information about the location and orientation (e.g., portrait or landscape) of device 100.
[0070] In some embodiments, the software components stored in memory 102 include an operating system 126, a biometric module 109, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, an authentication module 105, and an application (or instruction set) 136. Furthermore, in some embodiments, memory 102 ( Figure 1A ) or 370 ( Figure 3A Storage device / global internal state 157, such as Figure 1A and Figure 3A As shown in the figure. Device / global internal state 157 includes one or more of the following: active application state, which indicates which applications (if any) are currently active; display state, indicating what applications, views or other information occupy various areas of the touch screen display 112; sensor state, including information obtained from various sensors and input control devices 116 of the device; and position information relating to the device's position and / or orientation.
[0071] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.
[0072] The communication module 128 facilitates communication with other devices via one or more external ports 124 and includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. External ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) ® (etc.) Suitable for direct coupling to other devices, or indirect coupling via a network (e.g., the Internet, wireless LAN, etc.). In some implementations, the external port is for use with an iPod. ® (Trademark of Apple Inc.) The same or similar and / or compatible multi-pin (e.g., 30-pin) connectors used in Apple Inc. devices.
[0073] The biometric module 109 optionally stores information about one or more registered biometric features (e.g., fingerprint features, facial recognition features, eye and / or iris features) for verifying whether received biometric information matches the registered biometric features. In some embodiments, the stored information about one or more registered biometric features includes data that enables comparison between the stored information and the received biometric information, but does not include sufficient information to reproduce the registered biometric features. In some embodiments, the biometric module 109 stores information about registered biometric features in association with a user account of device 100. In some embodiments, the biometric module 109 compares the received biometric information with the registered biometric features to determine whether the received biometric information matches the registered biometric features.
[0074] The contact / motion module 130 optionally detects contact with the touchscreen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether a contact has occurred (e.g., detecting a finger press event), determining the contact intensity (e.g., the force or pressure of the contact, or an alternative to force or pressure), determining whether there is movement of the contact and tracking movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or a contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations can optionally be applied to single-point contact (e.g., single-finger contact) or multi-point simultaneous contact (e.g., "multi-touch" / multiple-finger contact). In some implementations, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.
[0075] In some implementations, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by the user (e.g., determining whether the user has “clicked” an icon). In some implementations, at least a subset of the intensity thresholds is determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a specific physical actuator and can be adjusted without changing the physical hardware of device 100). For example, the mouse “click” threshold of a touchpad or touchscreen can be set to any threshold in a wide range of predefined thresholds without changing the touchpad or touchscreen display hardware. Additionally, in some specific implementations, the user of the device is provided with software settings for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting the individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using system-level clicks on the “intensity” parameter).
[0076] The touch / motion module 130 optionally detects gesture input performed by the user. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different movements, timings, and / or intensities of the detected contact). Therefore, gestures can be detected optionally by detecting specific contact patterns. For example, detecting a finger tap gesture includes: detecting a finger press event, and then detecting a finger lift-off (lift-away) event at the same (or substantially the same) location as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes: detecting a finger press event, then detecting one or more finger drag events, and subsequently detecting a finger lift-off (lift-away) event.
[0077] The graphics module 132 includes various known software components for rendering and displaying graphics on the touchscreen 112 or other displays, including components for altering the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). As used herein, the term "graphics" includes any object that can be displayed to a user, including but not limited to text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.
[0078] In some implementations, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from applications, etc., to specify the graphics to be displayed, and also receives coordinate data and other graphic attribute data if necessary, and then generates screen image data to output to the display controller 156.
[0079] The haptic feedback module 133 includes various software components for generating instructions which are used by the haptic output generator 167 to generate haptic output at one or more locations on the device 100 in response to user interaction with the device 100.
[0080] Optionally, the text input module 134, a component of the graphics module 132, provides a soft keyboard for entering text in various applications, such as the contact module 137, email client module 140, IM module 141, browser module 147, and any other application that requires text input.
[0081] GPS module 135 determines the location of the device and provides that information for use in various applications (e.g., to telephone module 138 for use in location-based dialing; to camera module 143 as image / video metadata; and to applications that provide location-based services, such as weather widgets, local yellow pages widgets, and map / navigation widgets).
[0082] Authentication module 105 determines whether a requested operation (e.g., requested by an application in application 136) is authorized to be performed. In some implementations, authentication module 105 receives an operation to be performed that optionally requires authentication. Authentication module 105 determines whether the operation is authorized to be performed based on a range of factors, including the locked state of device 100, the location of device 100, whether a security delay has occurred, whether the received biometric information matches a registered biometric feature, and / or other factors. Once authentication module 105 determines that the operation is authorized to be performed, authentication module 105 triggers the execution of the operation.
[0083] Application 136 may optionally include the following modules (or instruction sets) or subsets or supersets thereof: • Contacts module 137 (sometimes called address book or contact list); • Telephone module 138; • Video conferencing module 139; • Email client module 140; • Instant Messaging (IM) module 141; • Fitness support module 142; • Camera module 143 for still images and / or video images; • Image management module 144; • Video player module; • Music player module; • Browser module 147; • Calendar module 148; • Widget module 149, which may optionally include one or more of the following: weather widget 149-1, stock market widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5 and other widgets obtained by the user, as well as user-created widget 149-6; • Widget creator module 150 for creating user-created widgets 149-6; • Search module 151; • Video and music player module 152, which combines a video player module and a music player module; • Memo module 153; • Map module 154; and / or • Online video module 155.
[0084] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.
[0085] In conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., in application internal state 192 of the contact module 137 stored in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers or email addresses to initiate and / or facilitate communications via the telephone module 138, video conferencing module 139, email client module 140, or IM module 141; and so on.
[0086] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, telephone module 138 is optionally used to input character sequences corresponding to telephone numbers, access one or more telephone numbers in contact module 137, modify input telephone numbers, dial corresponding telephone numbers, initiate conversations, and disconnect or hang up when a conversation is completed. As noted above, wireless communication may optionally use any of a variety of communication standards, protocols, and technologies.
[0087] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, text input module 134, contact module 137, and telephone module 138, video conferencing module 139 includes executable instructions to initiate, conduct, and terminate video conferences between the user and one or more other participants based on user instructions.
[0088] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. Combined with image management module 144, email client module 140 makes it very easy to create and send emails containing still images or video images captured by camera module 143.
[0089] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for performing the following operations: entering a character sequence corresponding to an instant message, modifying previously entered characters, sending a corresponding instant message (e.g., using Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocols for telephone-based instant messaging or using XMPP, SIMPLE, or IMPS for internet-based instant messaging), receiving an instant message, and viewing received instant messages. In some embodiments, the sent and / or received instant messages may optionally include graphics, photographs, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Services (EMS). As used herein, "instant message" means both telephone-based messages (e.g., messages delivered using SMS or MMS) and internet-based messages (e.g., messages delivered using XMPP, SIMPLE, or IMPS).
[0090] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module, fitness support module 142 includes executable instructions for performing the following operations: creating fitness activities (e.g., with time, distance, and / or calorie burning goals); communicating with fitness sensors (exercise equipment); receiving fitness sensor data; calibrating sensors used to monitor fitness; selecting and playing music for fitness activities; and displaying, storing, and transmitting fitness data.
[0091] In conjunction with the touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, and image management module 144, the camera module 143 includes executable instructions for performing the following operations: capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, or deleting still images or videos from memory 102.
[0092] Incorporating the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, the image management module 144 includes executable instructions for performing operations such as arranging, modifying (e.g., editing) or otherwise manipulating, marking, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.
[0093] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions for performing the following operations: browsing the Internet according to user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, as well as links to attachments and other files on web pages.
[0094] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions to create, display, modify, and store calendars and associated data (e.g., calendar entries, to-dos, etc.) according to user instructions.
[0095] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and browser module 147, widget module 149 may be a micro-application (e.g., weather widget 149-1, stock market widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or a user-created micro-application (e.g., user-created widget 149-6). In some implementations, the widget includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript. ® Files. In some implementations, the component includes XML (Extensible Markup Language) files and JavaScript. ® Files (e.g., Yahoo!) ® (widgets).
[0096] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and browser module 147, widget creator module 150 can optionally be used by the user to create widgets (e.g., turning a user-specified section of a webpage into a widget).
[0097] In conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the search module 151 includes executable instructions for performing the following operations: searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.
[0098] Incorporating touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions allowing users to download and play back recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touchscreen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).
[0099] Combining the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the memo module 153 includes executable instructions for creating and managing memos, to-do items, etc., according to user instructions.
[0100] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 may optionally be used to receive, display, modify, and store maps and map-related data (e.g., driving directions, data related to shops and other points of interest at or near a specific location, and other location-based data) according to user instructions.
[0101] Incorporating touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, the online video module 155 includes instructions for performing the following operations: allowing users to access, browse, receive (e.g., via streaming and / or downloading), play back (e.g., on the touchscreen or on an external display connected via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, an instant messaging module 141 is used instead of the email client module 140 to send links to specific online videos. Additional descriptions of the online video application can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11 / 968,067, filed December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are incorporated herein by reference in their entirety.
[0102] Each of the modules and applications described above corresponds to an executable set of instructions for performing one or more functions described above and the methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) need not be implemented as separate software programs (such as computer programs (e.g., including instructions)), processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. For example, a video player module may optionally be combined with a music player module into a single module (e.g., Figure 1A (Video and music player module 152). In some embodiments, memory 102 may optionally store subgroups of the modules and data structures identified above. Additionally, memory 102 may optionally store additional modules and data structures not described above.
[0103] In some implementations, device 100 is a device on which the operation of a predefined set of functions is performed solely via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for operating device 100, the number of physical input control devices (such as push-buttons and dial pads) on device 100 can be optionally reduced.
[0104] A predefined set of functions, uniquely performed via a touchscreen and / or touchpad, may optionally include navigation between user interfaces. In some implementations, the touchpad, when touched by a user, navigates device 100 from any user interface displayed on device 100 to the main menu, main desktop menu, or root menu. In such implementations, a "menu button" is implemented using the touchpad. In some other implementations, the menu button is a physical push-button or other physical input control device, rather than a touchpad.
[0105] Figure 1B This is a block diagram illustrating exemplary components for event handling according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3A This includes an event classifier 170 (e.g., in operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).
[0106] Event classifier 170 receives event information and determines the application 136-1 and application view 191 of application 136-1 to which the event information should be delivered. Event classifier 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or running. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information should be delivered.
[0107] In some implementations, the application internal state 192 includes additional information such as one or more of the following: recovery information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or ready to be displayed by the application 136-1, a state queue for enabling the user to return to the previous state or view of the application 136-1, and a repeat / undo queue for the user's previous actions.
[0108] Event monitor 171 receives event information from peripheral device interface 118. The event information includes information about sub-events, such as user touches on touch-sensitive display 112 as part of a multi-touch gesture. Peripheral device interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (via audio circuitry 110). The information received by peripheral device interface 118 from I / O subsystem 106 includes information from touch-sensitive display 112 or touch-sensitive surfaces.
[0109] In some implementations, the event monitor 171 sends requests to the peripheral device interface 118 at predetermined intervals. In response, the peripheral device interface 118 sends event information. In other implementations, the peripheral device interface 118 sends event information only when a significant event occurs (e.g., receiving input above a predetermined noise threshold and / or receiving input for a predetermined duration).
[0110] In some implementations, the event classifier 170 also includes a hit view determination module 172 and / or an activity event recognizer determination module 173.
[0111] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides a software process for determining where a sub-event has occurred within one or more views. A view consists of controls and other elements that the user can see on the display.
[0112] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected optionally corresponds to a procedural level within the application's procedural or view hierarchy. For example, the lowest-level view in which a touch is detected may optionally be called the hit view, and the set of events identified as correct input is optionally determined, at least in part, based on the hit view of the initial touch that initiates the touch-based gesture.
[0113] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchical structure, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchical structure from which the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events forming an event or potential event) occurs. Once the hit view is identified by the hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source to which it was identified as the hit view.
[0114] The activity event recognizer determination module 173 determines which views(s) within the view hierarchy should receive a specific sub-event sequence. In some embodiments, the activity event recognizer determination module 173 determines that only the hit view should receive the specific sub-event sequence. In other embodiments, the activity event recognizer determination module 173 determines that all views including the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive the specific sub-event sequence. In other embodiments, even if the touch sub-event is entirely confined to the area associated with a particular view, higher views in the hierarchy will still remain actively participating views.
[0115] Event assigner module 174 assigns event information to event identifiers (e.g., event identifier 180). In embodiments that include active event identifier determination module 173, event assigner module 174 delivers event information to the event identifier determined by active event identifier determination module 173. In some embodiments, event assigner module 174 stores event information in an event queue, which is retrieved by the corresponding event receiver 182.
[0116] In some implementations, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another implementation, event classifier 170 is a separate module or part of another module (such as contact / motion module 130) stored in memory 102.
[0117] In some implementations, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes a plurality of event recognizers 180. In other implementations, one or more of the event recognizers 180 are part of a separate module, such as a user interface toolkit or a higher-level object from which application 136-1 inherits methods and other properties. In some implementations, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handlers 190 may optionally utilize or invoke the data updater 176, the object updater 177, or the GUI updater 178 to update the application's internal state 192. Alternatively, one or more application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.
[0118] The corresponding event identifier 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies the event based on the event information. The event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event identifier 180 also includes at least one subset of metadata 183 and event delivery instructions 188 (which may optionally include sub-event delivery instructions).
[0119] Event receiver 182 receives event information from event classifier 170. The event information includes information about sub-events, such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves touch movement, the event information may optionally also include the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a lateral orientation, or vice versa), and the event information includes corresponding information about the device's current orientation (also referred to as device orientation).
[0120] Event comparator 184 compares event information with predefined event or sub-event definitions and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 contains definitions of events (e.g., predefined sequences of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in events (e.g., 187-1 and / or 187-2) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, event 1 (187-1) is defined as a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift-off of a predetermined duration (touch end), a second touch (touch start) of a predetermined duration on the displayed object, and a second lift-off of a predetermined duration (touch end). In another example, event 2 (187-2) is defined as a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on the displayed object, movement of the touch on the touch-sensitive display 112, and lifting the touch (end of touch). In some embodiments, the event also includes information for one or more associated event handlers 190.
[0121] In some implementations, event definition 186 includes definitions of events for corresponding user interface objects. In some implementations, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub-event and the object that triggered the hit test.
[0122] In some implementations, the definition of the corresponding event (187) also includes a delay action that delays the delivery of event information until it has been determined whether the sub-event sequence actually corresponds to or does not correspond to the event type of the event recognizer.
[0123] When the corresponding event recognizer 180 determines that the sub-event sequence does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events based on touch gestures are ignored. In this case, other event recognizers (if any) that remain active in the hit view continue to track and process the ongoing sub-events based on touch gestures.
[0124] In some embodiments, the corresponding event recognizer 180 includes metadata 183 having configurable attributes, flags, and / or lists instructing how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing how or how event recognizers can interact with each other. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing whether sub-events are delivered to different levels in a view or programmatic hierarchy.
[0125] In some implementations, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates the event handler 190 associated with the event. In some implementations, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from delivering (and deferred delivering) the sub-events to the corresponding hit view. In some implementations, the event recognizer 180 throws a flag associated with the identified event, and the event handler 190 associated with the flag acquires the flag and performs a predefined process.
[0126] In some implementations, event delivery instruction 188 includes a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or the actively participating view receives the event information and performs a predetermined process.
[0127] In some implementations, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some implementations, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the positioning of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and transmits that display information to graphics module 132 for display on a touch-sensitive display.
[0128] In some implementations, event handler 190 includes, or has access to, a data updater 176, an object updater 177, and a GUI updater 178. In some implementations, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other implementations, they are included in two or more software modules.
[0129] It should be understood that the above discussion regarding event handling for user touch on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, and not all user input is initiated on the touchscreen. For example, mouse movement and mouse button presses optionally in conjunction with single or multiple keyboard presses or holds; touch movements on the touchpad, such as taps, drags, scrolls, etc.; stylus input; device movement; verbal commands; detected eye movements; biometric input; and / or any combination thereof may optionally be used as input corresponding to sub-events that define the event to be identified.
[0130] Figure 2A portable multifunction device 100 with a touchscreen 112 is illustrated according to some embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more graphics by gesturing over the graphics, for example, using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, selection of one or more graphics occurs when the user breaks contact with one or more graphics. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or scrolling (from right to left, from left to right, up and / or down) of a finger already in contact with the device 100. In some specific embodiments or in some cases, unintentional contact with a graphic does not select the graphic. For example, a swipe gesture over an application icon may optionally not select the corresponding application when the gesture corresponding to selection is a tap.
[0131] Device 100 may optionally also include one or more physical buttons, such as a "main desktop" or menu button 204. As previously described, menu button 204 may optionally be used to navigate to any application 136 of a set of applications optionally executed on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on touchscreen 112.
[0132] In some embodiments, device 100 includes a touchscreen 112, a menu button 204, a push-button 206 for powering on / off and locking the device, one or more volume control buttons 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push-button 206 is optionally used to: power on / off the device by pressing the button and holding it in the pressed state for a predefined time interval; lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or unlock the device or initiate an unlocking process. In another embodiment, device 100 also accepts voice input via microphone 113 for activating or deactivating certain functions. Device 100 may also optionally include one or more contact strength sensors 165 for detecting the intensity of contact on the touchscreen 112, and / or one or more haptic output generators 167 for generating haptic output for the user of device 100.
[0133] Figure 3AThis is a block diagram of an exemplary multi-functional device with a display and a touch-sensitive surface according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (e.g., a home controller or industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuitry (sometimes referred to as a chipset) that interconnects system components and controls communication between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touchscreen display. The I / O interface 330 may also optionally include a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, and a haptic output generator 357 for generating haptic output on device 300 (e.g., similar to the reference above). Figure 1A The described tactile output generator 167), sensor 359 (e.g., optical sensor, accelerometer, proximity sensor, touch sensor and / or contact intensity sensor (similar to the one described above)) Figure 1A The described contact strength sensor 165). Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310. In some embodiments, memory 370 stores information related to portable multifunction device 100. Figure 1A The memory 370 stores programs, modules, and data structures similar to those in the memory 102 of the portable multifunction device 100, or subsets thereof. Additionally, the memory 370 may optionally store additional programs, modules, and data structures not present in the memory 102 of the portable multifunction device 100. For example, the memory 370 of the device 300 may optionally store a drawing module 380, a rendering module 382, a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while the portable multifunction device 100 ( Figure 1A The memory 102 may optionally not store these modules.
[0134] Figure 3AEach of the elements described above may optionally be stored in one or more memory devices of the previously mentioned memory devices. Each of the modules described above corresponds to a set of instructions for performing the functions described above. The modules or computer programs described above (e.g., instruction sets or including instructions) need not be implemented as separate software programs (such as computer programs (e.g., including instructions)), processes, or modules, and therefore various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 may optionally store a subset of the modules and data structures described above. In addition, memory 370 may optionally store additional modules and data structures not described above.
[0135] Specific embodiments within the scope of this disclosure may be implemented, in whole or in part, using a tangible computer-readable storage medium (or a plurality of tangible computer-readable storage media of one or more types) that encodes one or more computer-readable instructions. It should be understood that computer-readable instructions may be organized in any format, including applications, widgets, processes, software, and / or components.
[0136] Specific embodiments within the scope of this disclosure include computer-readable storage media that encode instructions organized as an application (e.g., application 3160) that, when executed by one or more processing units, control the execution of an electronic device (e.g., device 3150). Figure 3B Methods Figure 3C The methods and / or one or more other processes and / or methods described herein.
[0137] It should be recognized that the application of 3160 ( Figure 3D The application 3160 (as shown in the diagram) can be any suitable type of application, including one or more of the following: browser applications, applications used as execution environments for plugins, widgets, or other applications, fitness applications, health applications, digital payment applications, media applications, social networking applications, messaging applications, and / or map applications. In some embodiments, application 3160 is an application pre-installed on device 3150 at the time of purchase (e.g., a first-party application). In some embodiments, application 3160 is an application provided to device 3150 via operating system update files (e.g., a first-party or second-party application). In some embodiments, application 3160 is an application provided via an app store. In some embodiments, the app store can be an app store pre-installed on device 3150 at the time of purchase (e.g., a first-party app store). In some embodiments, the app store is a third-party app store (e.g., an app store provided by another app store, downloaded via a network, and / or read from a storage device).
[0138] refer to Figure 3B and Figure 3FApplication 3160 obtains information (e.g., 3010). In some embodiments, at 3010, information is obtained from at least one hardware component of device 3150. In some embodiments, at 3010, information is obtained from at least one software module of device 3150. In some embodiments, at 3010, information is obtained from at least one hardware component external to device 3150 (e.g., peripheral devices, accessory devices, and / or servers). In some embodiments, the information obtained at 3010 includes location information, time information, notification information, user information, environmental information, electronic device status information, weather information, media information, historical information, event information, hardware information, and / or motion information. In some embodiments, in response to obtaining information at 3010 and / or after this step, application 3160 provides the information to the system (e.g., 3020).
[0139] In some implementations, the system (e.g., Figure 3E The 3110 shown is the operating system hosted on the device 3150. In some implementations, the system (e.g., Figure 3E 3110 shown in the figure is an external device (e.g., a server, peripheral device, accessory and / or personal computing device) that includes an operating system.
[0140] refer to Figure 3C and Figure 3G Application 3160 obtains information (e.g., 3030). In some embodiments, the information obtained at 3030 includes location information, time information, notification information, user information, environmental information, electronic device status information, weather information, media information, historical information, event information, hardware information, and / or motion information. In response to obtaining information at 3030 and / or after that step, application 3160 performs an operation on the information (e.g., 3040). In some embodiments, the operation performed at 3040 includes: providing notifications based on the information, sending messages based on the information, displaying information, controlling the user interface of a fitness application based on the information, controlling the user interface of a health application based on the information, controlling focus mode based on the information, setting reminders based on the information, adding calendar entries based on the information, and / or calling the API of system 3110 based on the information.
[0141] In some implementations, execution is performed in response to a trigger. Figure 3B Methods and / or Figure 3C The method involves one or more steps. In some implementations, triggering includes the detection of an event, a notification received from system 3110, user input, and / or a response to a call to an API provided by system 3110.
[0142] In some implementations, when the instructions of application 3160 are executed, control device 3150 executes them by calling an application programming interface (API) (e.g., API 3190) provided by system 3110. Figure 3B Methods and / or Figure 3C The method. In some implementations, application 3160 executes without calling API 3190. Figure 3B Methods and / or Figure 3C At least a part of the method.
[0143] In some implementation schemes, Figure 3B Methods and / or Figure 3C One or more steps of the method involve calling the API (e.g., API 3190) using one or more parameters defined by the API. In some implementations, one or more parameters include constants, keys, data structures, objects, object classes, variables, data types, pointers, arrays, lists, or pointers to functions or methods and / or references to data or other items to be passed via the API in another way.
[0144] refer to Figure 3D Example 3150 is shown. In some embodiments, device 3150 is a personal computing device, smartphone, smartwatch, fitness tracker, head-mounted display (HMD) device, media device, public utility, speaker, television, and / or tablet computer. Figure 3D As illustrated, device 3150 includes application 3160 and operating system (e.g., Figure 3E System 3110 is shown in the diagram. Application 3160 includes application implementation module 3170 and API call module 3180. System 3110 includes API 3190 and implementation module 3100. It should be understood that device 3150, application 3160 and / or system 3110 may include... Figure 3D and Figure 3E The examples illustrate more, fewer, and / or different components.
[0145] In some implementations, application implementation module 3170 includes a set of one or more instructions corresponding to one or more operations performed by application 3160. For example, when application 3160 is a messaging application, application implementation module 3170 may include operations for receiving and sending messages. In some implementations, application implementation module 3170 communicates with API calling module 3180 via API 3190 (in... Figure 3E (As shown in the figure) communicates with system 3110.
[0146] In some implementations, API 3190 is a software module (e.g., a set of computer-readable instructions) that provides an interface allowing different modules (e.g., API calling module 3180) to access and / or use one or more functions, methods, procedures, data structures, classes, and / or other services provided by implementation module 3100 of system 3110. For example, API calling module 3180 can access features of implementation module 3100 through one or more API calls or enablements (e.g., embodied by function or method calls) exposed by API 3190 (e.g., software and / or hardware modules capable of receiving, responding to, and / or transmitting API calls), and can pass data and / or control information via API calls or enablements using one or more parameters. In some implementations, API 3190 allows application 3160 to use services provided by a software development kit (SDK) library. In some implementations, application 3160 combines calls to functions or methods provided by the SDK library and API 3190, or uses data types or objects defined in the SDK library and provided by API 3190. In some implementations, API calling module 3180 makes API calls via API 3190 to access and use features of implementation module 3100 specified by API 3190. In such implementations, implementation module 3100 may return a value to API calling module 3180 via API 3190 in response to an API call. This value may report to application 3160 the capabilities or status of hardware components of device 3150, including those capabilities or statuses related to aspects such as input capabilities and status, output capabilities and status, processing capabilities, power status, storage capacity and status, and / or communication capabilities. In some implementations, API 3190 is implemented in part by firmware, microcode, or other low-level logic executed in part on the hardware components.
[0147] In some implementations, API 3190 allows the developer of API calling module 3180 (which may be a third-party developer) to utilize features provided by implementation module 3100. In such implementations, one or more API calling modules (e.g., including API calling module 3180) may exist that communicate with implementation module 3100. In some implementations, API 3190 allows multiple API calling modules written in different programming languages to communicate with implementation module 3100 (e.g., API 3190 may include features for translating calls and returns between implementation module 3100 and API calling module 3180), and API 3190 is implemented in a specific programming language. In some implementations, API calling module 3180 calls APIs from different providers, such as a set of APIs from an OS provider, another set of APIs from a plugin provider, and / or another set of APIs from another provider (e.g., a software library provider) or the creator of another set of APIs.
[0148] Examples of API 3190 may include one or more of the following: pairing API (e.g., for establishing a secure connection, such as with an accessory), device detection API (e.g., for locating nearby devices, such as media devices and / or smartphones), payment API, UIKit API (e.g., for generating user interfaces), location detection API, locator API, map API, health sensor API, sensor API, messaging API, push notification API, streaming API, collaboration API, video conferencing API, app store API, advertising service API, web browser API (e.g., WebKit API), transportation API, networking API, WiFi API, Bluetooth API, NFC API, UWB API, fitness API, smart home API, contact transfer API, photo API, camera API, and / or image processing API. In some implementations, a sensor API is an API for accessing data associated with sensors of device 3150. For example, a sensor API may provide access to raw sensor data. Alternatively, a sensor API may provide data derived (and / or generated) from raw sensor data. In some implementations, sensor data includes temperature data, image data, video data, audio data, heart rate data, IMU (Inertial Measurement Unit) data, LiDAR data, location data, GPS data, and / or camera data. In some implementations, sensors include one or more of accelerometers, temperature sensors, infrared sensors, optical sensors, heart rate sensors, barometers, gyroscopes, proximity sensors, and / or biometric sensors.
[0149] In some embodiments, implementation module 3100 is a system (e.g., an operating system and / or server system) software module (e.g., a set of computer-readable instructions) configured to perform operations in response to receiving an API call via API 3190. In some embodiments, implementation module 3100 is configured to provide an API response (via API 3190) as a result of processing the API call. For example, implementation module 3100 and API call module 3180 can each be any of an operating system, library, device driver, API, application, or other module. It should be understood that implementation module 3100 and API call module 3180 can be the same or different types of modules. In some embodiments, implementation module 3100 is at least partially embodied in firmware, microcode, or hardware logic.
[0150] In some implementations, implementation module 3100 returns a value via API 3190 in response to an API call from API call module 3180. While API 3190 defines the syntax and results of the API call (e.g., how to enable the API call and what the API call does), API 3190 may not reveal how implementation module 3100 performs the functionality specified by the API call. Various API calls are passed via one or more application programming interfaces between API call module 3180 and implementation module 3100. Passing API calls may include issuing, initiating, enabling, calling, receiving, returning, and / or responding to function calls or messages. In other words, passing can describe the actions of API call module 3180 or implementation module 3100. In some implementations, function calls or other enablements of API 3190 transmit and / or receive one or more parameters via parameter lists or other structures.
[0151] In some implementations, implementation module 3100 provides more than one API, each API providing a different view or aspect of the functionality implemented by implementation module 3100. For example, one API of implementation module 3100 may provide a first set of functions and be exposed to third-party developers, while another API of implementation module 3100 may be hidden (e.g., not exposed) and provide a subset of the first set of functions, and also provide another set of functions, such as test or debug functions not in the first set of functions. In some implementations, implementation module 3100 calls one or more other components via lower-level APIs, thus acting as both an API calling module and an implementation module. It should be recognized that implementation module 3100 may include additional functions, methods, classes, data structures, and / or other features not specified through API 3190 and not available to API calling module 3180. It should also be recognized that API calling module 3180 may be on the same system as implementation module 3100, or may be remotely located and accessed via a network using API 3190. In some implementations, implementation module 3100, API 3190, and / or API calling module 3180 are stored in a machine-readable medium, which includes any means for storing information in a machine-readable (e.g., computer or other data processing system) form. For example, a machine-readable medium may include a magnetic disk, optical disk, random access memory, read-only memory, and / or flash memory devices.
[0152] An Application Programming Interface (API) is an interface between a first software process and a second software process, specifying the format for communication between the two processes. Limited APIs (e.g., private or partner APIs) are APIs accessible to a limited set of software processes (e.g., only software processes within the operating system or only software processes authorized to access the limited API). Public APIs are accessible to a broader set of software processes. Some APIs enable a software process to communicate or set the state of one or more input devices (e.g., one or more touch sensors, proximity sensors, vision sensors, motion / or orientation sensors, pressure sensors, intensity sensors, sound sensors, wireless proximity sensors, biometric sensors, buttons, switches, rotatable elements, and / or external controllers). Some APIs enable a software process to communicate and / or set the state of one or more output generation components (e.g., one or more audio output generation components, one or more display generation components, and / or one or more haptic output generation components). Some APIs enable specific capabilities (e.g., scrolling, handwriting, text input, image editing, and / or image creation) to be accessed, executed, and / or used by a software process (e.g., generating output for use by the software process based on input from the software process). Some APIs enable content from software processes to be inserted into templates and displayed in user interfaces with layouts and / or behaviors specified by the templates.
[0153] Many software platforms include a set of frameworks that provide core objects and behaviors that software developers need to build software applications that can be used on the platform. Software developers use these objects to display content on a screen, interact with that content, and manage interactions with the software platform. The basic behavior of a software application depends on this framework, and this framework provides software developers with numerous ways to customize the application's behavior to match the specific needs of the application. Many of these core objects and behaviors are accessed via APIs. APIs typically specify the format for communication between software processes, including specifying and grouping available variables, functions, and protocols. API calls (sometimes called API requests) are typically passed from a sending software process to a receiving software process as a way to achieve one or more of the following: the sending software process requests information from the receiving software process (e.g., for the sending software process to take an action); the sending software process provides information to the receiving software process (e.g., for the receiving software process to take an action); the sending software process requests an action from the receiving software process; or the sending software process provides information to the receiving software process about the action taken by the sending software process. In some cases, interaction with a device (e.g., using a user interface) will involve passing and / or receiving one or more API calls (e.g., multiple API calls) between multiple different software processes (e.g., different parts of an operating system, applications and operating systems, or different applications) via one or more APIs (e.g., via multiple different APIs). For example, when input is detected, direct sensor data is frequently processed into one or more input events, which are provided (e.g., via an API) to a receiving software process, which makes some determinations based on the input events and then (e.g., via an API) transmits information to the software process to perform an operation (e.g., change the device state and / or the user interface) based on the determinations. While the determinations and the operations performed in response can be made by the same software process, alternatively, the determinations can be made in a first software process and relayed (e.g., via an API) to a second software process different from the first software process, allowing the operation to be performed by the second software process. Alternatively, the second software process can relay instructions (e.g., via an API) to a third software process different from the first and / or second software processes to perform the operation. It should be understood that some or all user interactions with a computer system may involve one or more API calls within the steps of interacting with the computer system (e.g., between different software components of the computer system or between software components of the computer system and software components of one or more remote computer systems).It should be understood that some or all user interactions with a computer system may involve one or more API calls between steps of interaction with the computer system (e.g., between different software components of the computer system or between software components of the computer system and software components of one or more remote computer systems).
[0154] In some implementations, the application can be any suitable type of application, including one or more of the following: browser applications, applications used as execution environments for plugins, widgets or other applications, fitness applications, health applications, digital payment applications, media applications, social networking applications, messaging applications and / or map applications.
[0155] In some implementations, the application is an application pre-installed on the first computer system at the time of purchase (e.g., a first-party application). In some implementations, the application is an application provided to the first computer system via an operating system update file (e.g., a first-party application). In some implementations, the application is an application provided via an app store. In some implementations, the app store is pre-installed on the first computer system at the time of purchase (e.g., a first-party app store) and allows the download of one or more applications. In some implementations, the app store is a third-party app store (e.g., an app store provided by another device, downloaded via a network, and / or read from a storage device). In some implementations, the application is a third-party application (e.g., an application provided by an app store, downloaded via a network, and / or read from a storage device). In some implementations, the application controls the first computer system to execute methods 700 and / or 800 by calling an application programming interface (API) provided by a system process using one or more parameters (e.g., ...). Figure 7 and / or Figure 8 ).
[0156] In some implementations, exemplary APIs provided by system processes include one or more of the following: pairing API (e.g., for establishing a secure connection, such as with an accessory), device detection API (e.g., for locating nearby devices, such as media devices and / or smartphones), payment API, UIKit API (e.g., for generating user interfaces), location detection API, locator API, map API, health sensor API, sensor API, messaging API, push notification API, streaming API, collaboration API, video conferencing API, app store API, advertising service API, web browser API (e.g., WebKit API), transportation API, networking API, WiFi API, Bluetooth API, NFC API, UWB API, fitness API, smart home API, contact transfer API, photo API, camera API, and / or image processing API.
[0157] In some embodiments, at least one API is a software module (e.g., a set of computer-readable instructions) that provides an interface allowing different modules (e.g., API calling module 3180) to access and use one or more functions, methods, procedures, data structures, classes, and / or other services provided by an implementation module of a system process. The API may define one or more parameters passed between the API calling module and the implementation module. In some embodiments, API 3190 defines a first API call that can be provided by API calling module 3180. An implementation module is a system software module (e.g., a set of computer-readable instructions) configured to perform operations in response to receiving an API call via the API. In some embodiments, the implementation module is configured to provide an API response (via the API) as a result of processing the API call. In some embodiments, the implementation module is included in a device (e.g., 3150) running an application. In some embodiments, the implementation module is included in an electronic device separate from the device running the application.
[0158] Now let’s turn our attention to the implementation of the user interface, which may be implemented, for example, on a portable multifunction device 100.
[0159] Figure 4A An exemplary user interface for an application menu on a portable multifunction device 100 according to some embodiments is illustrated. A similar user interface may optionally be implemented on device 300. In some embodiments, user interface 400 includes the following elements or a subset or superset thereof: • Signal strength indicator 402 for wireless communications such as cellular signals and Wi-Fi signals; • Time 404; • Bluetooth indicator 405; • Battery status indicator 406; • Tray 408 features icons for frequently used applications, such as: o The telephone module 138 has an icon 416 labeled "telephone", which optionally includes an indicator 414 for the number of missed calls or voicemail messages; o An icon 418 labeled "Mail" in the email client module 140, which optionally includes an indicator 410 for the number of unread emails; o The browser module 147 has an icon 420 labeled "Browser"; and o The video and music player module 152 (also known as the iPod (Apple Inc. trademark) module 152) with an icon 422 labeled "iPod"; and • Icons of other applications, such as: o IM module 141's icon 424 labeled "Message"; o Calendar module 148 has an icon 426 labeled "Calendar"; o Image management module 144 icon 428 labeled "Photo"; o The icon 430 of the camera module 143 is labeled "camera"; o The icon 432 of the online video module 155 is labeled "Online Video"; o The icon 434 labeled "Stock Market" in the Stock Market widget 149-2; o Map module 154's icon 436 labeled "map"; o Weather widget 149-1 with icon 438 labeled "Weather"; o Alarm clock widget 149-4 with icon 440 labeled "Clock"; o The icon 442 of the fitness support module 142 is labeled "fitness support"; o The icon 444 labeled "Memo" in the Memo module 153; and o A settings icon 446 labeled "Settings" is used to set up an application or module, which provides access to settings for the device 100 and its various applications 136.
[0160] It should be pointed out that, Figure 4A The illustrated icon labels are merely exemplary. For example, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels may be used for various application icons. In some embodiments, the label of a particular application icon includes the name of the application corresponding to that particular application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to that particular application icon.
[0161] Figure 4B An example is illustrated having a touch-sensitive surface 451 (e.g., separate from the display 450 (e.g., touchscreen display 112)). Figure 3A Devices such as tablets or touchpads (e.g., 355) Figure 3A An exemplary user interface on the device 300. The device 300 may also optionally include one or more contact intensity sensors (e.g., one or more sensors in sensor 359) for detecting the intensity of contact on the touch-sensitive surface 451 and / or one or more haptic output generators 357 for generating haptic output for the user of the device 300.
[0162] While some examples of input on a touchscreen display 112 (which combines a touch-sensitive surface and a display) are given below, in some implementations the device detects input on a touch-sensitive surface separate from the display, such as... Figure 4B As shown in the diagram. In some embodiments, the touch-sensitive surface (e.g., Figure 4B The touch-sensitive surface 451 in the middle has a spindle (e.g., on the display 450) corresponding to the main axis on the display (e.g., Figure 4B The main shaft of 453 in the middle (e.g., Figure 4B (452 in the example). According to these embodiments, the device detects the position corresponding to a specific location on the display (e.g., in...). Figure 4B In the middle, contact 460 corresponds to 468 and contact 462 corresponds to 470) is in contact with the touch-sensitive surface 451 (e.g., Figure 4B Contacts 460 and 462 in the middle). Thus, when the touch-sensitive surface (e.g., Figure 4B The touch-sensitive surface 451 in the middle) and the display of the multi-functional device (e.g., Figure 4B When the display (450) is separated, user input detected by the device on the touch-sensitive surface (e.g., touches 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods may be used alternatively for other user interfaces described herein.
[0163] Additionally, while the examples below are given primarily with reference to finger input (e.g., finger touch, single-finger tap gesture, finger swipe gesture), it should be understood that in some implementations, one or more of these finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of a touch), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the touch). As another example, a tap gesture may optionally be replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detection of touch, followed by cessation of touch detection). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice may optionally be used simultaneously, or mouse and finger touch may optionally be used simultaneously.
[0164] Figure 5A An exemplary personal electronic device 500 is illustrated. Device 500 includes a body 502. In some embodiments, device 500 may include components relative to device 100 and device 300 (e.g., Figures 1A to 4BThe device 500 may include some or all of the features described herein. In some embodiments, the device 500 has a touch-sensitive display 504, referred to below as a touchscreen 504. Alternatively, or in addition to the touchscreen 504, the device 500 may also have a display and a touch-sensitive surface. Similar to the cases of devices 100 and 300, in some embodiments, the touchscreen 504 (or touch-sensitive surface) may optionally include one or more intensity sensors for detecting the intensity of an applied contact (e.g., a touch). The one or more intensity sensors of the touchscreen 504 (or touch-sensitive surface) may provide output data representing the intensity of the touch. The user interface of the device 500 may respond to touches based on the intensity of the touch, meaning that touches of different intensities may invoke different user interface operations on the device 500.
[0165] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related applications: International Patent Application Serial No. PCT / US2013 / 040061, filed May 8, 2013, entitled “Device, Method, and Graphical User Interface for Displaying UserInterface Objects Corresponding to an Application,” published as WIPO Publication No. WO / 2013 / 169849; and International Patent Application Serial No. PCT / US2013 / 069483, filed November 11, 2013, entitled “Device, Method, and Graphical User Interface for Transitioning Between TouchInput to Display Output Relationships,” published as WIPO Publication No. WO / 2014 / 105276, each of which is incorporated herein by reference in its entirety.
[0166] In some embodiments, device 500 has one or more input mechanisms 506 and 508. Input mechanisms 506 and 508, if included, can be physical. Examples of physical input mechanisms include push-buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. Such attachment mechanisms, if included, allow device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watch straps, bangles, trousers, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow a user to wear device 500.
[0167] Figure 5BAn exemplary personal electronic device 500 is depicted. In some embodiments, device 500 may include information about... Figure 1A , Figure 1B and Figure 3A Some or all of the components described. Device 500 has a bus 512 that operatively couples I / O portion 514 to one or more computer processors 516 and memory 518. I / O portion 514 may be connected to display screen 504, which may have touch-sensitive component 522 and optionally have intensity sensor 524 (e.g., contact intensity sensor). Furthermore, I / O portion 514 may be connected to communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. For example, input mechanism 506 may optionally be a rotatable input device or a pressable and rotatable input device. In some examples, input mechanism 508 may optionally be a button.
[0168] In some examples, the input mechanism 508 may optionally be a microphone. The personal electronic device 500 may optionally include various sensors, such as a GPS sensor 532, an accelerometer 534, an orientation sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or combinations thereof, all of which may be operatively connected to the I / O section 514.
[0169] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which, when executed by one or more computer processors 516, may cause the computer processors to perform, for example, the techniques described below, including processes 700 to 800 (e.g., Figures 7 to 8 A computer-readable storage medium can be any medium that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media can include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, CD-based, DVD-based, or Blu-ray-based storage devices. ® Optical discs and persistent solid-state storage such as flash memory and solid-state drives. Personal electronic devices 500 are not limited to... Figure 5B It can be the components and configurations, or it can include other components or additional components in a variety of configurations.
[0170] As used herein, the term "power indication" refers to the ability to indicate power in devices 100, 300, and / or 500 (…). Figure 1A , Figure 3A and Figures 5A to 5B A user-interactive graphical user interface object displayed on a screen. For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) may each constitute a functional representation.
[0171] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some specific implementations that include a cursor or other position marker, the cursor acts as a "focus selector," such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the cursor is positioned on a touch-sensitive surface (e.g., a...). Figure 3A The touchpad 355 or Figure 4B When an input (e.g., a press input) is detected on the touch-sensitive surface 451 of the display, the specific user interface element is adjusted according to the detected input. This applies to touchscreen displays (e.g., those capable of direct interaction with user interface elements on a touchscreen display) that enable direct interaction with user interface elements on the touchscreen display. Figure 1A The touch-sensitive display system 112 or Figure 4A In some embodiments of the touchscreen 112, a touch detected on the touchscreen acts as a "focus selector," such that when input (e.g., a press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, that particular user interface element is adjusted according to the detected input. In some embodiments, focus moves from one area of the user interface to another without corresponding movement of the cursor or movement of a touch on the touchscreen display (e.g., moving focus from one button to another using tab keys or arrow keys); in these embodiments, the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user-controlled user interface element (or a touch on the touchscreen display) that delivers the user's expected interaction with the user interface (e.g., by indicating to the device the element of the user interface that the user expects to interact with). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of the focus selector (e.g., a cursor, touch, or selection box) above the corresponding button will indicate to the user that they expect to activate the corresponding button (rather than other user interface elements shown on the device's display).
[0172] As used in the specification and claims, the term "characteristic strength" of a contact refers to a characteristic of the contact based on one or more strengths of the contact. In some embodiments, the characteristic strength is based on multiple strength samples. The characteristic strength may optionally be based on a predefined number of strength samples or a set of strength samples collected over a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted, before or after contact begins to move, before contact ends, before or after contact strength is detected to increase, and / or before or after contact strength is detected to decrease). The characteristic strength of the contact may optionally be based on one or more of the following: the maximum value of the contact strength, the mean value of the contact strength, the average value of the contact strength, the value at the top 10% of the contact strength, the half maximum value of the contact strength, or the 90% maximum value of the contact strength, etc. In some embodiments, the duration of the contact is used when determining the characteristic strength (e.g., when the characteristic strength is the average value of the contact strength over time). In some implementations, the characteristic intensity is compared to a set of one or more intensity thresholds to determine whether a user has performed an action. For example, the set of one or more intensity thresholds may optionally include a first intensity threshold and a second intensity threshold. In this example, contact with a characteristic intensity not exceeding the first threshold results in a first action, contact with a characteristic intensity exceeding the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a characteristic intensity exceeding the second threshold results in a third action. In some implementations, a comparison between the characteristic intensity and one or more thresholds is used to determine whether one or more actions should be performed (e.g., whether to perform the corresponding action or abandon performing the corresponding action) rather than to determine whether to perform the first action or the second action.
[0173] In some implementations, the computer system is in a locked or unlocked state. In the locked state, the computer system is powered on and operable, but it is prevented from performing a predefined set of operations in response to user input. This predefined set of operations may optionally include navigation between user interfaces, activation or deactivation of a predefined set of functions, and activation or deactivation of certain applications. The locked state can be used to prevent unintentional or unauthorized use of a function of the computer system, or to activate or deactivate some functions on the computer system. In some implementations, in the unlocked state, the computer system is powered on and operable, and is not prevented from performing at least some of the predefined set of operations that are not possible when in the locked state. When the computer system is in a locked state, it is said to be locked. When the computer system is in an unlocked state, it is said to be unlocked. In some implementations, a locked computer system may optionally respond to a limited set of user inputs, including inputs corresponding to an attempt to unlock the computer system or inputs corresponding to shutting down the computer system.
[0174] Now let’s turn our attention to the implementation of user interfaces (“UIs”) on electronic devices, such as portable multifunction devices 100, 300 or 500, and the associated processes.
[0175] Figures 6A to 6Y Exemplary user interfaces for managing translations according to some implementation schemes are illustrated. The user interfaces in these figures are used to illustrate the processes described below, including... Figure 7 and Figure 8 The process in.
[0176] Figure 6A Scenario 680 is illustrated, including user 680A of computer system 600. Computer system 600 includes telephone 600A and headset 600B. Scenario 680 shows two people, user 680A and person 680B, sitting at the same table. User 680A is a user of computer system 600 (e.g., logged into computer system 600 using a user account, owns computer system 600, and / or is wearing components of computer system 600). Person 680B is not logged into computer system 600, does not own computer system 600, and / or is not wearing components of computer system 600. User 680A speaks and understands a first language (e.g., English) and has a limited ability to (e.g., not) speak and understand a second language (e.g., Spanish). Person 680B speaks and understands a second language (e.g., Spanish) and has a limited ability to (e.g., not) speak and understand a first language (e.g., English). Figures 6A to 6YAs illustrated, although user 680A and person 680B do not share a common language, computer system 600 facilitates conversation between user 680A and person 680B. Scenario 680 is not part of the displayed user interface but is illustrated to aid in understanding the described techniques. User 680A is wearing headset 600B, which wirelessly communicates with user 680A's phone 600A. In some embodiments, phone 600A has been configured by user 680A to operate a virtual assistant using a first language (e.g., English), such that the virtual assistant outputs responses to commands in the first language. In some embodiments, phone 600A has been configured (e.g., based on input from user 680A) to perform translation between a first language and a second language, such as by downloading language information corresponding to the first language and / or the second language from an online source. Phone 600A is in a locked state, as optionally indicated by displaying a lock indicator 604A and a lock screen user interface 604 via display 602. In some implementations, the 600A phone remains locked throughout the translation process, such as Figures 6A to 6Y As illustrated. In some implementations, the phone 600A is unlocked during part or all of the translation process.
[0177] As described in more detail below, in some implementations, when computer system 600 is not in translation mode, computer system 600 does not output a translation of speech (e.g., via audio speakers and via a display). For example, if the microphone of computer system 600 is not enabled to allow computer system 600 to detect speech, computer system 600 does not translate the detected speech, and / or computer system 600 does not output translated speech. Once computer system 600 is in translation mode (e.g., in response to detecting a corresponding user request), computer system 600 automatically detects (e.g., both user 680A and other persons such as person 680B) speech and outputs a corresponding translation via one or more of the speakers of headset 600B, the speakers of telephone 600A, and / or the display of telephone 600A (e.g., if no additional user input is required or received). Thus, computer system 600 detects a request to operate in translation mode, and user 680A and person 680B can participate in multilingual conversations without providing external user input at computer system 600. In some implementations, voice is detected using one or more microphones of the headset 600B and / or one or more microphones of the telephone 600A.
[0178] exist Figure 6AAt this point, telephone 600A is not operating in translation mode (e.g., translation mode is disabled and / or not yet enabled). Person 680B speaks (e.g., 660A), but computer system 600 does not detect the audio, does not transcribe the audio, and / or does not output the transcription / translation of the speech because computer system 600 is not operating in translation mode. Therefore, in Figure 6B At this point, telephone 600A continues to display the lock screen user interface 604 via display 602. Similarly, because computer system 600 is not operating in translation mode, computer system 600 does not translate and / or output the translation of user 680A's speech.
[0179] exist Figure 6B At this point, computer system 600 detects a request to enable translation mode, such as input 650A pointing to headset 600B (e.g., touch input and / or air gestures) and / or input 650B pointing to telephone 600A (e.g., button press and / or touch input).
[0180] exist Figure 6C At this point, in response to detecting a request to enable translation mode (e.g., input 650A and / or 650B), computer system 600 begins operating in translation mode (and throughout...). Figures 6C to 6Y (The system remains in translation mode). In some embodiments, in response to detecting a request to enable translation mode (e.g., inputs 650A and / or 650B), computer system 600 outputs audio 670A via headset 600B (e.g., one or more speakers of the headset) (e.g., not via one or more speakers of telephone 600A), which instructs computer system 600 to translate speech from a remote person, such as person 680B, into a language (e.g., a second language, such as Spanish). For example, computer system 600 outputs audio indicating that computer system 600 is listening to Spanish speech for translation into a first language. In some embodiments, such as... Figure 6C As shown, in response to a detected request to enable translation mode (e.g., input 650A and / or 650B), computer system 600 displays session transcript user interface 610 via display 602 (e.g., by replacing the display of lock screen user interface 604, and optionally continuing to display lock indicator 604A). For example, session transcript user interface 610 includes an indication 610A for a second language, from which computer system 600 translates speech when speech in that language is detected from a remote person, such as person 680B.
[0181] exist Figure 6DWhen computer system 600 is operating in translation mode, person 680B speaks speech 660B, as shown in scenario 680. For example, person 680B says “¿Hola, cómo estás?” (in Spanish). Computer system 600 uses one or more microphones of headset 600B (and optionally, not the microphone of telephone 600A) to detect speech 660B. Computer system 600 optionally determines that the detected speech 660B comes from a remote speaker (e.g., not user 680A) and / or that the detected speech 660B is in a second language. In response to the detection of speech 660B, computer system 600 translates speech 660B from the second language (e.g., Spanish) to the first language (e.g., English). In some embodiments, computer system 600 translates from that language based on a previously received user selection for the second language used for translation, as described above. In some embodiments, computer system 600 translates from the second language based on the determination that speech 660B was spoken in the second language. For example, computer system 600 detects speech, determines the language of the speech, and translates the speech into a first language using the determined language. In some embodiments, translating speech 660B from a second language to a first language includes performing speech recognition to transcribe the speech audio (e.g., 660B) into text, such as transcribing speech audio in a second language into text in the second language. In some embodiments, the text in the second language is then translated into text in the first language, which is then output by computer system 600.
[0182] exist Figure 6EIn response to the computer system 600 detecting speech 660B, the computer system 600 outputs a translation of speech 660B. In some embodiments, the computer system 600 outputs an audio translation 670B of speech 660B (in a first language) via one or more speakers of the headset 600B, and outputs a visual translation 612A as part of the conversational transcript user interface 610. In some embodiments, the computer system 600 displays text 614A of speech 660B in a second language (e.g., untranslated) as part of the conversational transcript user interface 610 (e.g., concurrently with the visual translation 612A). For example, the visual translation 612A and text 614A are displayed in the same speech bubble, positioned to one side (e.g., the left side) of the conversational transcript user interface 610 to indicate that person 680B is the speaker. In some examples, the font size of text 614A is smaller than the font size of visual translation 612A. In some embodiments, the telephone 600A does not automatically output audio (translated or untranslated) corresponding to speech 660B in response to the detection of speech 660B. Therefore, user 680A hears the translation of voice 660B, and user 680A and / or user 680B see the translation of voice 660B.
[0183] exist Figures 6F to 6H At this point, while computer system 600 continues to operate in translation mode, user 680A speaks (e.g., voice 662A), such as... Figures 6F to 6GScenario 680 is shown. For example, user 680A says, “I'm trying to find a place to have dinner before it gets dark” (in English). Computer system 600 uses one or more microphones of headset 600B (and optionally, not the microphone of telephone 600A) to detect speech 662A. Computer system 600 optionally determines that the detected speech 662A comes from a local speaker (e.g., user 680A) and / or that the detected speech 662A is in a first language. Computer system 600 translates speech 662A from the first language (e.g., English) into a second language (e.g., Spanish) (e.g., without further user input). In some embodiments, computer system 600 translates from the first language based on a previously received selection of the first language for translation, as described above. In some embodiments, computer system 600 translates from the first language based on the determination that speech 662A was spoken in the first language. In some embodiments, translating speech 662A from a first language to a second language includes performing speech recognition to transcribe the speech audio into text, such as transcribing speech audio in a first language into text in the first language. The text in the first language is then translated into text in the second language, which is then output by computer system 600 (e.g., 616A and / or 672A). In some embodiments, computer system 600 translates into the second language based on a previously received selection of that language, as described above. In some embodiments, computer system 600 translates into the second language based on previously detected (e.g., from a remote person) speech in the second language (e.g., speech 660B).
[0184] In some implementations, to reduce the delay in translating the output speech 662A, the computer system 600 begins (e.g., via display 602 and / or via one or more speakers) to output the translation of the detected speech 662A without waiting for pauses in speech 662A and without waiting for the end of sentences in speech 662A, as per [the relevant documentation / reference]. Figure 6J A more detailed description.
[0185] exist Figures 6G to 6IWhen (e.g., in response to) computer system 600 detecting a corresponding portion of speech 662A, computer system 600 translates and outputs a corresponding translation of the corresponding portion of speech 662A, such as by using a marker, as described in detail below. In some embodiments, computer system 600 outputs an audio translation 672A (in a second language) of speech 662A via one or more speakers of telephone 600A, and outputs a visual translation 616A as part of conversational transcript user interface 610. In some embodiments, computer system 600 displays text 618A of speech 660B in a first language (e.g., untranslated) as part of conversational transcript user interface 610. For example, visual translation 616A and text 618A are displayed in the same speech bubble, positioned on the opposite side (e.g., the right side) of conversational transcript user interface 610 to indicate that user 680A is the speaker. In this example, the font size of text 618A is smaller than the font size of visual translation 616A. In some implementations, telephone 600A does not output audio corresponding to voice 662A (e.g., 672A, translated or untranslated). Therefore, user 680A and / or person 680B hear (with reduced latency) and / or see (with reduced latency) the translation of voice 662A.
[0186] Figure 6J An example procedure for reducing latency when translating the detected speech is illustrated. At 682A, computer system 600 has detected... Figure 6F The computer system 600 receives speech 662A and performs speech recognition to transcribe the speech audio into transcribed text 682A1, such as transcribing speech audio in a first language into text in a first language when speech is detected. In some embodiments, the computer system 600 inserts a marker 684A (e.g., a comma) into the transcribed (untranslated) text 682A1 of speech 682A. Once the marker (e.g., 684A) is inserted into the transcribed text 682A1, the computer system 600 translates up to the portion of the transcribed text 682A1 containing the inserted marker ("I'm trying to find") and begins output (e.g., as shown in the image). Figure 6G Translate the text (as shown) without waiting for pauses in the speech (e.g., speech 662A) and without waiting for the end of the sentence in the speech (e.g., speech 662A). The remainder of the transcribed text 682A1 (e.g., “a pla…”) is preserved and becomes part of the next speech segment to be analyzed. For example, in Figure 6J At position 682B, computer system 600 is already... Figure 6GAdditional speech 662A is detected, and speech recognition has been performed to transcribe the speech audio into transcribed text 682B1, which includes the remainder of the transcribed text 682A1 after marker 684A. In some embodiments, computer system 600 inserts marker 684B (e.g., a comma) into the transcribed (untranslated) text 682B1 of speech 682A. Once the marker (e.g., 684B) is inserted into the transcribed text 682B1, computer system 600 translates up to the portion of the transcribed text 682B1 up to the inserted marker (“aplace to have dinner”), and computer system 600 queues the translations for output (e.g., as shown in the image). Figure 6H As shown), without waiting for pauses in the speech (e.g., speech 662A) and without waiting for the end of the sentence in the speech (e.g., speech 662A). The remainder of the transcribed text 682B1 (e.g., “before…”) is preserved and becomes part of the next speech segment being analyzed. For example, in Figure 6J At point 682C, computer system 600 is already... Figure 6H Additional speech 662A is detected, and speech recognition has been performed to transcribe the speech audio into transcribed text 682C1, which includes the remainder of transcribed text 682B1 after marker 684B (e.g., "before"). Computer system 600 detects the end of the sentence (e.g., period 684C) and therefore does not insert additional markers. Computer system 600 translates up to the end of the sentence in transcribed text 682C1 ("before itgets dark"), and computer system 600 queues the translations for output (e.g., as shown in the image). Figure 6I (As shown). Therefore, computer system 600 outputs the translation of the detected speech with a significantly reduced delay (e.g., compared to waiting for a pause in the speech or the end of a sentence). Although commas (e.g., 684A, 684B) are used as markers in this example, other markers, such as other punctuation marks or text symbols, may also be used. In some embodiments, markers (e.g., commas) are not inserted based on grammatical correctness, but rather based on identifying the position in the speech where translation / output can begin. In some embodiments, when the translation is output (e.g., via a display and / or via speech from one or more speakers using the computer system), the markers are not used for pronunciation and / or intonation purposes.
[0187] Similarly, computer system 600 can insert tags into translated text (e.g., 682A2, 682B2) instead of inserting tags (e.g., 684A) into transcribed (untranslated) text (e.g., 682A1, 682B1), as... Figure 6J As shown. Therefore, when speech is detected and transcribed, the transcribed text is translated. The computer system inserts markers (e.g., 686A and / or 686B) into the translated text to indicate that the portion preceding the marker can be output (e.g., via a display and / or via spoken audio using one or more speakers) and / or queued for output. Figure 6J As shown, at 682A, computer system 600 inserts marker 686A and begins outputting the underlined translated text (e.g., up to the marked portion) in 682A. Similarly, at 682B, computer system 600 inserts marker 686B and begins outputting (or queuing to output) the underlined translated text (e.g., up to the marked portion) in 682B. At 682C, computer system 600 detects the end of the sentence (e.g., based on period 686C) and outputs (or queues to output) the underlined translated text in 686C (e.g., the remainder of the sentence). Thus, computer system 600 outputs the translation of the detected speech with significantly reduced latency (e.g., compared to waiting for a pause in the speech or the end of a sentence). Although commas (e.g., 686A, 686B) are used as markers in this example, other markers, such as other punctuation marks or text symbols, can also be used. In some implementations, markers (e.g., commas) are not inserted based on grammatical correctness, but rather based on identifying where the output in the speech can begin. In some implementations, when translating the output (e.g., via a display and / or via one or more speakers using a computer system), markers are not used for pronunciation and / or intonation purposes.
[0188] exist Figures 6K to 6MWhen (e.g., in response to) the computer system 600 detecting a corresponding portion of speech 660C, the computer system 600 translates and outputs a corresponding translation of the corresponding portion of speech 660C, such as by inserting a marker, as described in more detail below. In some embodiments, the computer system 600 outputs an audio translation 670C of speech 660C (in a first language) via one or more speakers of headphones 600B, and outputs a visual translation 612B as part of the conversational transcript user interface 610. In some embodiments, the computer system 600 displays text 614B of speech 660C in a second language (e.g., untranslated) as part of the conversational transcript user interface 610. For example, the visual translation 612B and text 614B are displayed in the same speech bubble, positioned to one side (e.g., the left side) of the conversational transcript user interface 610 to indicate that person 680B is the speaker. In this example, the font size of text 614B is smaller than the font size of visual translation 612B. In some implementations, telephone 600A does not output audio corresponding to voice 660C (e.g., 672A, translated or untranslated). Therefore, user 680A hears the translation of voice 660C (with reduced latency), and user 680A and / or person 680B see the translation of voice 660C (with reduced latency).
[0189] Figure 6N An example procedure for reducing latency when translating the detected speech is illustrated. At 682D, computer system 600 has detected... Figure 6K The computer system 600 receives speech 660C and performs speech recognition to transcribe the speech audio into transcribed text 682D1, such as transing speech audio in a second language into text in a second language when speech is detected. In some embodiments, the computer system 600 inserts a marker 684D (e.g., a comma) into the transcribed (untranslated) text 682D1 of speech 660C. Once the marker (e.g., 684D) is inserted into the transcribed text 682D1, the computer system 600 translates up to the portion of the transcribed text 682D1 containing the inserted marker (“Hay un buen”) and begins output (e.g., as shown in the image). Figure 6L Translate the text (as shown) without waiting for pauses in the speech (e.g., speech 660C) and without waiting for the end of the sentence in the speech (e.g., speech 660C). The remainder of the transcribed text 682D1 (e.g., “restaurante…”) is preserved and becomes part of the next speech segment to be analyzed. For example, in Figure 6N At location 682E, computer system 600 is already... Figure 6LAdditional speech 660C is detected, and speech recognition has been performed to transcribe the speech audio into transcribed text 682E1, which includes the remainder of transcribed text 682D1 after marker 684D (e.g., “restaurante”). Computer system 600 detects the end of the sentence (e.g., period 684E) and therefore does not insert additional markers. Computer system 600 translates up to the end of the sentence (“restaurante de sándwiches cerca”) of transcribed text 682E1, and computer system 600 queues the translation (“sandwichrestaurant nearby”) for output (e.g., as shown in the image). Figure 6M (As shown). Therefore, computer system 600 outputs the translation of the detected speech with a significantly reduced delay (e.g., compared to waiting for a pause in the speech or the end of a sentence). Although a comma (e.g., 684D) is used as a marker in this example, other markers, such as other punctuation marks or text symbols, may also be used. In some embodiments, the marker (e.g., comma) is not inserted based on grammatical correctness, but rather based on identifying the position in the speech where the translation / output can begin. In some embodiments, the marker is not used for pronunciation and / or intonation purposes when the translation is output (e.g., via a display and / or via speech from one or more speakers using the computer system).
[0190] Similarly, computer system 600 can insert tags into translated text (e.g., 682A2, 682B2) instead of inserting tags (e.g., 684A) into transcribed (untranslated) text (e.g., 682A1, 682B1), as... Figure 6J As shown. Therefore, when speech is detected and transcribed, the transcribed text is translated. The computer system inserts markers (e.g., 686A and / or 686B) into the translated text to indicate that the portion preceding the marker can be output (e.g., via a display and / or via spoken audio using one or more speakers) and / or queued for output. Figure 6JAs shown, at 682A, computer system 600 inserts marker 686A and begins outputting the underlined translated text (e.g., up to the marked portion) in 682A. Similarly, at 682B, computer system 600 inserts marker 686B and begins outputting (or queuing to output) the underlined translated text (e.g., up to the marked portion) in 682B. At 682C, computer system 600 detects the end of the sentence (e.g., based on period 686C) and outputs (or queues to output) the underlined translated text in 686C (e.g., the remainder of the sentence). Thus, computer system 600 outputs the translation of the detected speech with significantly reduced latency (e.g., compared to waiting for a pause in the speech or the end of a sentence). Although commas (e.g., 686A, 686B) are used as markers in this example, other markers, such as other punctuation marks or text symbols, can also be used. In some implementations, markers (e.g., commas) are not inserted based on grammatical correctness, but rather on identifying where the output in the speech can begin. In some implementations, when the translation is output as speech (e.g., via a computer system's display and / or one or more speakers), markers are not used for pronunciation and / or intonation purposes.
[0191] exist Figures 60 to 6P When computer system 600 is operating in translation mode, user 680A speaks (e.g., voice 662B), such as... Figure 6O The scenario is illustrated in scenario 680. For example, user 680A says “Excellent!” (in English). Computer system 600 uses one or more microphones of headset 600B (and optionally, not the microphone of telephone 600A) to detect speech 662B. Computer system 600 optionally determines that the detected speech 662B comes from a local speaker (e.g., user 680A) and / or that the detected speech 662A is in a first language. Computer system 600 translates speech 662B from the first language (e.g., English) into a second language (e.g., Spanish). In some embodiments, computer system 600 translates from the first language based on a previously received selection of the first language for translation, as described above. In some embodiments, computer system 600 translates from the first language based on the determination that speech 662B was spoken in the first language. In some embodiments, translating speech 662B from the first language to the second language includes performing speech recognition to transcribe the speech audio into text, such as transcribing speech audio in the first language into text in the first language. The text in the first language is then translated into text in the second language, which is then output by the computer system 600.
[0192] exist Figure 6P In response to the computer system 600 detecting speech 662B, the computer system 600 outputs a translation of speech 662B. In some embodiments, the computer system 600 outputs an audio translation 672B of speech 662B (in a second language) via one or more speakers of the telephone 600A, and outputs a visual translation 616B as part of the conversational transcript user interface 610. In some embodiments, the computer system 600 displays text 618B of speech 662B in a first language (e.g., untranslated) as part of the conversational transcript user interface 610. For example, the visual translation 616B and text 618B are displayed in the same speech bubble, positioned to one side (e.g., the right side) of the conversational transcript user interface 610 to indicate that user 680A is the speaker. In this example, the font size of text 618B is smaller than the font size of visual translation 616B. Therefore, user 680A and person 680B hear and / or see the translation of speech 662B. In some implementations, telephone 600A does not output audio (translated or untranslated) corresponding to voice 662B.
[0193] exist Figure 6P At this point, computer system 600 detects user input (e.g., 650C). In response to detecting user input and based on a determination that the user input (e.g., 650C) points to a corresponding conversational transcript (e.g., visual translation 612B, text 614B, and / or corresponding speech bubble) corresponding to the speech of person 680B (e.g., the user input is a tap or swipe on the corresponding conversational transcript), computer system 600 abandons the output of audio corresponding to the corresponding speech (e.g., does not output audio for visual translation 612B and text 614B). For example, computer system 600 does not output audio for 612B and / or 614B in response to detecting input 650C. In response to detecting user input and based on the determination that the user input (e.g., 650D and / or 650E) points to a corresponding session transcription (e.g., visual translation 616B, text 618B, and / or corresponding speech bubble) corresponding to the user 680A's speech and / or to a playback option 610E (e.g., a play button or other graphic) corresponding to the user 680A's speech (e.g., a tap or swipe on the corresponding session transcription and / or the playback option), computer system 600 outputs (e.g., re-outputs) the audio corresponding to the corresponding speech (e.g., 672C) (e.g., outputting the audio corresponding to visual translation 616B), such as... Figure 6Q As shown, and / or change the display orientation of the displayed content (e.g., change from portrait orientation to landscape orientation), such as Figure 6QAs shown. In some embodiments, computer system 600 uses one or more speakers of telephone 600A without using headset 600B to output (e.g., re-output) the audio translation of the corresponding speech of user 680A. Therefore, computer system 600 enables user 680A to cause computer system 600 to (e.g., again) generate the audio output of the translation of user 680A's speech. In some embodiments, based on a determination that user input (e.g., 650D and / or 650E) points to a corresponding session transcript corresponding to user 680A's speech (e.g., the user input is a tap or swipe on that corresponding session transcript), telephone 600A additionally transforms into displaying an enlarged version of the corresponding transcript via user interface 630 for displaying an enlarged translation, such as... Figure 6Q As shown. In some implementations, in response to the detection of individual user input, such as detecting that the phone 600A is rotated to landscape orientation, the phone 600A switches to displaying an enlarged version of the corresponding transcription via a user interface 630 for displaying the enlarged translation, such as... Figure 6Q As shown.
[0194] exist Figure 6Q At this point, computer system 600 displays an enlarged version of visual translation 616B as part of user interface 630. The enlarged version of visual translation 616B displayed as part of user interface 630 (e.g., as shown in...) Figures 6Q to 6U (in) greater than the visual translation 616B displayed as part of the session transcript user interface 610 (e.g., in Figure 6P as well as Figures 6W to 6Y (For example, the magnified version is displayed in a font size that is larger, higher, wider, and / or bolder than that). The magnified translation, displayed as part of the user interface 630, makes it easier for people such as person 680B to see the translation of user 680A's speech.
[0195] In some implementation schemes, and as per [the relevant information] Figures 6Q to 6V In more detail, when the computer system 600 is displaying a user interface 630 for displaying magnified translations, the computer system 600 displays the translation of the speech of user 680A, but not the translation of the speech of other people such as person 680B.
[0196] exist Figure 6RIn this scenario, when computer system 600 is operating in translation mode and displaying user interface 630 for magnified translation, person 680B speaks speech 660D, as shown in scenario 680. For example, person 680B says “¿Cuando vasa ir?” (in Spanish). Computer system 600 uses one or more microphones of headset 600B (and optionally, not the microphone of telephone 600A) to detect speech 660D. In response to detecting speech 660D, computer system 600 translates speech 660D from a second language (e.g., Spanish) to a first language (e.g., English).
[0197] exist Figure 6S In response to the computer system 600 detecting speech 660D, the computer system 600 outputs a translation of speech 660D. In some embodiments, the computer system 600 outputs an audio translation 670D of speech 660D (in the first language) via one or more speakers of the headset 600B, instead of outputting a corresponding visual translation on the display of the telephone 600A, such as... Figure 6S As shown. In some implementations, telephone 600A does not automatically output audio (translated or untranslated) corresponding to voice 660D in response to the detection of voice 660D. Therefore, user 680A hears the translation of voice 660D, while user 680A and / or person 680B can continue to see the translation of voice 662B (of user 680A).
[0198] exist Figure 6T At this point, computer system 600 detects ambient noise exceeding a threshold level and, in response, outputs an indication that the ambient noise is too loud (e.g., audio 670E). In some examples, computer system 600 uses one or more microphones of headset 600B to detect the ambient noise level. In some examples, computer system 600 uses one or more microphones of telephone 600A and / or headset 600B to detect the ambient noise level. For example, computer system 600 (e.g., using one or more speakers of headset 600B instead of telephone 600A's speakers) outputs audio 670E that instructs user 680A to move telephone 600A closer to person 680B (e.g., so that the microphone of telephone 600A can better detect the audio of person 680B).
[0199] exist Figure 6U At this point, user 680A has moved phone 600A closer to person 680B, thereby enabling one or more microphones on phone 600A to better detect audio (e.g., voice) from person 680B. Figure 6UWhen computer system 600 is operating in translation mode and displaying user interface 630 for magnified translation, user 680A speaks voice 662C, as shown in scenario 680. For example, user 680A says “Now.” (in English). Computer system 600 uses one or more microphones of headset 600B to detect voice 662C. In response to the detection of voice 662C, computer system 600 translates voice 662C from a first language (e.g., English) to a second language (e.g., Spanish). In some implementations (e.g., when ambient noise is above a threshold level, and optionally after user 680A has moved phone 600A closer to person 680B), computer system 600 uses one or more microphones of phone 600A (e.g., without using the microphone of headset 600B) to detect the voice of person 680B, the voice of a person not wearing headset 600B, the voice of a person closer to phone 600A than headset 600B, and / or the voice of a person speaking a second language. In some implementations (e.g., when ambient noise is above a threshold level, and optionally after user 680A has moved phone 600A closer to person 680B), computer system 600 uses one or more microphones of headset 600B (e.g., without using the microphone of phone 600A) to detect the speech of user 680A, the speech of the person wearing headset 600B, the speech of the person who is closer to headset 600B than phone 600A, and / or the speech of the person who speaks their first language.
[0200] exist Figure 6V In response to the computer system 600 detecting speech 662C, the computer system 600 outputs a translation of speech 662C. In some embodiments, the computer system 600 outputs an audio translation 672C (in a second language) of speech 662C via one or more speakers of the telephone 600A, instead of outputting the corresponding audio translation via the speakers of the headset 600B, such as... Figure 6V As shown. In some embodiments, telephone 600A does not automatically output audio (translated or untranslated) corresponding to voice 662C in response to the detection of voice 662C. In some embodiments, in response to the detection of voice 662C, computer system 600 outputs a corresponding visual translation 616C on the display of telephone 600A as part of user interface 630. For example, visual translation 616C replaces the display of visual translation 616B. Thus, user 680A and / or person 680B hears and sees the translation of voice 662C.
[0201] In some implementation schemes, such as Figures 6Q to 6VAs shown, user interface 630 includes a pause button 630A and / or a speaker button 630B. In some embodiments, selecting the pause button 630A pauses (and / or resumes) the translation (e.g., audio and visual translation of detected speech). In some embodiments, selecting the speaker button 630B enables phone 600A to output audio of the translated text (e.g., 616B) currently displayed as part of user interface 630. In some embodiments, selecting the speaker button 630B causes phone 600A to display controls (e.g., a volume slider and / or volume buttons) for controlling the volume of the audio output of phone 600A. In some embodiments, speaker button 630B mutes (and / or unmutes) the audio output of future translated speech. In some embodiments, user interface 630 includes the translation of user 680A's speech (e.g., 616B and / or 616C) but not the transcription of speech in the original language (e.g., 618B and / or 618C). In some implementations, the user interface 630 is updated to include a translation of the voice of user 680A, but not to include a translation of the voice of person 680B.
[0202] exist Figure 6V At this point, telephone 600A detects input pointing to button 630C (e.g., tapping input and / or pressing and holding input 650F). In response to detecting input pointing to button 630C, computer system 600 stops displaying user interface 630 and returns to displaying session transcript user interface 610, as... Figure 6W (e.g., landscape orientation) and / or Figure 6X (For example, in portrait orientation) As shown. Figures 6W to 6X As shown, the conversation transcript user interface 610 includes translations of speech detected while the phone 600A is displaying the user interface 630, including translations displayed as part of the user interface 630 (e.g., translation to a second language, such as visual translation 616C) and translations not displayed as part of the user interface 630 (e.g., translation to a first language, such as visual translation 612C). In some embodiments, when ambient noise is above a threshold level (and optionally, after user 680A has moved the phone 600A closer to person 680B), the computer system 600 uses one or more microphones of the phone 600A to detect the speech of person 680B. In some embodiments, when ambient noise is above a threshold level (and optionally, even after user 680A has moved the phone 600A closer to person 680B), the computer system 600 uses one or more microphones of the headset 600B to detect the speech of user 680A.
[0203] exist Figure 6XAt this point, computer system 600 detects input pointing to object 610B (e.g., tap input and / or touch and hold input 650G). In response to detecting input pointing to object 610B, computer system 600 updates session transcript user interface 610 to exclude transcriptions of the speech of person 680B in the original language (e.g., 614A to 614C), while continuing to include transcriptions of the speech of user 680A in the original language (e.g., 618A to 618C) and translations of the speech of both users (e.g., 612A to 612C and / or 616A to 616C), as... Figure 6Y As shown. In some implementations, in response to detecting input pointing to object 610B, computer system 600 replaces object 610B with object 610C, as... Figure 6Y As shown, object 610B is visually updated.
[0204] exist Figure 6Y At a time when telephone 600A is displaying a session transcript user interface 610 that does not include a transcription of the speech of person 680B in the original language (e.g., 614A to 614C), telephone 600A detects input (e.g., 650G and / or 650H). In response to the detected input and based on a determination that the input (e.g., 650G) points to object 610C, computer system 600 updates the session transcript user interface 610 to include a transcription of the speech of person 680B in the original language (e.g., 614A to 614C), while continuing to include a transcription of the speech of user 680A in the original language (e.g., 618A to 618C) and translations of the speech of both users (e.g., 612A to 612C and / or 616A to 616C), as... Figure 6Y As shown. In response to the detection of input and based on the determination that the input (e.g., 650H) points to an end object 610G, the computer system 600 exits the translation mode and thus stops automatically detecting speech, translating speech, and / or (e.g., via audio and / or visually) outputting translated speech. In some embodiments, in response to the detection of input and based on the determination that the input (e.g., 650H) points to an end object 610G, the computer system 600 returns to displaying the lock screen user interface 604, as shown. Figure 6A As shown. In some implementations, in response to the detection of input and based on a determination that the input (e.g., 650H) points to an end object 610G, the computer system 600 deletes session content (e.g., transcribed and / or translated speech) processed while the computer system 600 is in translation mode, thereby providing increased security and privacy.
[0205] Figure 7This is a flowchart illustrating a method for managing translation according to some implementation schemes. Method 700 is performed at a computer system (e.g., 100, 300, 500, and / or 600) (e.g., a smartphone, smartwatch, tablet, laptop, desktop computer, wearable device, and / or head-mounted device) that communicates with: one or more audio devices (e.g., one or more microphones (e.g., one or more microphones of 600A), one or more speakers, one or more headphones with a built-in microphone (e.g., 600B), one or more headsets, and / or one or more head-mounted headphones); one or more input devices (e.g., touch-sensitive surfaces (…) For example, touch-sensitive displays, such as 602; mice; keyboards; visual input devices (e.g., one or more cameras (e.g., infrared cameras, depth cameras, visible light cameras, and / or gaze-tracking cameras)); motion sensors (e.g., accelerometers and / or gyroscopes); and / or one or more mechanical input devices (e.g., pressable input mechanisms; buttons; rotatable input mechanisms; crowns; and / or dials)); and one or more display generating components (e.g., one or more displays 602, one or more touchscreen displays, one or more monitors, holographic display systems, transparent displays, and / or head-mounted display systems). Some operations in method 700 may be optionally combined, some operations may be optionally changed in order, and some operations may be optionally omitted.
[0206] As described below, Method 700 provides an intuitive way to manage translations. This method reduces the cognitive burden on users managing translations, thereby creating a more efficient human-computer interface. For battery-powered computing devices, it enables users to manage translations and / or participate in sessions faster and more efficiently, saving power and increasing the time between battery recharges.
[0207] The devices, methods, and / or computer-readable storage media described below enhance device operability and (e.g., by assisting users in providing appropriate input when operating / interacting with the device and reducing user errors) make the user-device interface more efficient. This, in turn, reduces power consumption and / or improves device battery life by enabling users to use the device more quickly and efficiently. Performing operations when a set of conditions are met without requiring further user input (such as by automatically translating speech and outputting speech translations (e.g., audio and / or visual)) enhances device operability by reducing unnecessary input and / or steps to navigate between different user interfaces or control groups, thereby reducing device energy consumption.
[0208] When not operating in translation mode that automatically detects spoken audio via one or more audio devices and (e.g., via one or more audio devices and / or via one or more display generation components) outputs a translation of the detected spoken audio (e.g., in... Figure 6B When the computer system (e.g., 600) is not operating in translation mode, the computer system detects (702) one or more sets of inputs (e.g., 650A and / or 650B, user input, touch input, and / or air gestures) via one or more input devices to enable translation mode. In some embodiments, when the computer system is not operating in translation mode, the computer system does not automatically translate the detected spoken audio and / or does not automatically output a translation of the detected spoken audio, such as... Figures 6A to 6B As shown.
[0209] In response to the detection of one or more sets of inputs (e.g., 650A and / or 650B) for enabling translation mode, the computer system (e.g., 600) operates in translation mode (704), the operation including, without requiring (and / or detecting) additional user input (e.g., via one or more input devices, without requiring any command, and / or before detecting the speech of a session participant): detecting (706) the speech of a session of two or more session participants (e.g., 680A and 680B) (e.g., 662A to 662C and / or 660B to 660D) via one or more audio devices, the speech including a first speech in a first language (e.g., 662A to 662C) and a second speech in a second language different from the first language (660B to 660D); outputting (708) an audio translation of the first speech (e.g., Audio translation of a second language and / or a third language different from the first and second languages (e.g., 670B to 670D) and / or a second speech (e.g., 672A to 672C, in the first language and / or in a third language different from the first and second languages); and concurrent display (710) of a conversation transcript (e.g., 610) via one or more display generation components (e.g., 602) and with audio translation of an output conversation (e.g., at least a portion of the conversation) (e.g., 670B to 670D and / or 672A to 672C), the conversation transcript comprising transcripts of the first and second speech of a conversation of two or more conversation participants (e.g., 612A to 612C, 614A to 614C, 616A to 616C and / or 618A to 618C) (e.g., textual, in the first language, in the second language, in the third language and / or a combination thereof).
[0210] In some implementations, in response to detecting one or more sets of inputs (e.g., 650A and / or 650B) for enabling a translation mode (e.g., as part of initiating and / or operating within the translation mode), a computer system (e.g., 600) outputs an instruction (e.g., 670A) to the computer system (e.g., 600) for the language (e.g., Spanish, Chinese, Japanese, Korean, English, and / or German) to which it will provide translation, via one or more audio devices (e.g., headphones worn by the user of the computer system and / or before outputting the audio translation). In some implementations, the computer system instructs to translate Spanish speech by outputting audio saying "Listen to Spanish".
[0211] In some embodiments, audio translation of the first speech output via one or more audio devices includes audio translation of the first speech into a second language (e.g., 672A). In some embodiments, audio translation of the second speech output via one or more audio devices includes audio translation of the second speech into a first language (e.g., 670B). In some embodiments, the computer system translates the first speech from a first language to a second language and translates the second speech from a second language back to a first language.
[0212] In some implementations, a set of one or more inputs (e.g., 650A and / or 650B) for enabling the translation mode includes a first touch input (e.g., 650A, tap and / or press) at a first earpiece (e.g., 600B, worn by a user of the computer system on the first ear) and (e.g., concurrently and / or in combination) a second touch input (e.g., tap and / or press) at a second earpiece (e.g., worn by a user of the computer system on the second ear), which is different from the first earpiece.
[0213] In some implementations, the computer system is in a locked mode (e.g., as by...). Figure 6B When a set of one or more inputs (e.g., 650A and / or 650B) for enabling translation mode is detected (as indicated by 604A at the location), this is done when the computer system is in a locked mode (e.g., as indicated by 604A at the location). Figures 6C to 6Y The session transcript is displayed when (as indicated by 604A in the document) (e.g., as part of 610). In some embodiments, when a set of one or more inputs is received, the computer system is displaying a lock screen user interface, and in response to detecting the set of one or more inputs, the computer system replaces the lock screen user interface with a session transcript user interface (e.g., which is updated as the session progresses to include the session transcript).
[0214] In some implementations, when operating in translation mode, the computer system (e.g., 600) detects the corresponding speech via one or more audio devices. In response to the detection of the corresponding speech: based on a determination that the corresponding speech satisfies one or more of a first set of conditions (e.g., the speech is far-field speech, the speech originates from a user not wearing headphones as part of the computer system, the speech is identified as the speech of a person who is not a user of the computer system, the speech originates from a person more than a threshold distance from the computer system, and / or the speech is in a second language), the computer system (e.g., 600) outputs (e.g., audio output via one or more audio devices (e.g., via headphones worn by the user of the computer system) and / or visual output via one or more display generation components) a translation of the corresponding speech into the corresponding language (e.g., a first language). In some implementations, the computer system translates the speech from the second language to the corresponding language (e.g., the first language) before outputting the translation of the speech.
[0215] In some implementations, the appropriate language is selected based on the language of the virtual assistant (e.g., a digital voice assistant, an AI natural language assistant) operating at the computer system (e.g., 600) (e.g., the appropriate language is selected as the same as the given language). In some implementations, the computer system receives user input to select a preferred language for the virtual assistant operating at the computer system. Subsequently, when a session translation is requested, the computer system automatically selects the preferred language for translating speech that is not in the preferred language.
[0216] In some embodiments, when operating in translation mode and in response to detecting a corresponding speech: based on a determination that the corresponding speech meets one or more conditions of a second set (e.g., not meeting one or more conditions of the first set, the speech is near-field speech, the speech originates from a user wearing headphones as part of the computer system, the speech is identified as the speech of a person who is a user of the computer system, the speech originates from a person less than a threshold distance from the computer system, and / or the speech is in a first language), the computer system (e.g., 600) outputs (e.g., audio output via one or more audio devices (e.g., via the speakerphone of a telephone of the computer system) and / or visual output via one or more display generating components) a translation of the corresponding speech into a second language different from the corresponding language (e.g., 672A). In some embodiments, the computer system translates the corresponding speech from the first language to the second language before outputting the translation of the corresponding speech. In some embodiments, outputting the translation of the corresponding speech into the second language includes visually (e.g., via one or more display generating components) outputting the translation of the corresponding speech into the second language without outputting the audio of the translation of the corresponding speech into the second language (and / or in the first language).
[0217] In some embodiments, outputting a translation of the corresponding speech in the corresponding language includes outputting the translation of the corresponding speech (e.g., 670B) via a first audio device (e.g., 600B, headphones, and / or headset) from one or more audio devices, without outputting the audio translation of the corresponding speech in the corresponding language via a second audio device (e.g., 600A) from one or more audio devices. In some embodiments, outputting a translation of the corresponding speech in a second language includes outputting the translation of the corresponding speech (e.g., 672A) via a second audio device (e.g., 600A and / or the speakerphone) from one or more audio devices, without outputting the audio translation of the corresponding speech in a second language via a first audio device (e.g., 600B) from one or more audio devices. In some embodiments, different language translations for different participants in the conversation are output to different speakers.
[0218] In some implementations, prior to the audio translation of the first speech (e.g., in a second language and / or in a third language different from the first and second languages) and the audio translation of the second speech via one or more audio devices, user input for selecting a second language (e.g., for translating the first speech into the second language and / or for translating the second speech from the second language) is received via one or more input devices (e.g., when not already in translation mode).
[0219] In some implementations, before outputting audio translation of the first speech (e.g., in a second language and / or in a third language different from the first and second languages) and audio translation of the second speech via one or more audio devices, the second language is automatically selected (e.g., based on detected speech, such as the second speech, and / or when operating in translation mode) without receiving user input for selecting the second language.
[0220] According to some embodiments, detecting the speech of a conversation of two or more participants, including a first speech in a first language and a second speech in a second language different from the first language, via one or more audio devices includes: detecting the first speech in the first language via a first microphone (e.g., 600A) and a second microphone (e.g., 600B), and detecting the second speech in the second language via the first microphone (e.g., 600A) and the second microphone (e.g., 600B). In some embodiments, the same set of one or more microphones is used to detect the audio of the participants (e.g., two microphones are used to detect the audio of the first participant, and the same two microphones are used to detect the audio of the second participant).
[0221] In some implementations, when operating in translation mode, the computer system (e.g., 600) detects the ambient noise level (or, optionally, determines that the participant's audio volume is below a threshold volume). Based on (e.g., in response to) the determination that the ambient noise level is above the threshold noise level, the computer system (e.g., 600) (e.g., via one or more audio devices and / or via one or more display generation components) outputs an indication (e.g., 670E) that at least a portion of the computer system (e.g., 600A, microphone and / or telephone) should be moved closer to one of the two or more session participants (e.g., 680B). In some implementations, based on (e.g., in response to) the determination that the ambient noise level is not above the threshold noise level, the computer system abandons the output of the indication that at least a portion of the computer system (e.g., microphone and / or telephone) should be moved closer to one of the two or more session participants.
[0222] In some implementations, displaying a conversation transcript of a conversation of two or more conversation participants (e.g., as part of 610) includes displaying a transcript of a first speech in a first language (e.g., 618A), a transcript of a first speech in a second language (e.g., 616A), a transcript of a second speech in a first language (e.g., 612A), and a transcript of a second speech in a second language (e.g., 614A) via one or more display generation components (e.g., 602).
[0223] In some implementations, when operating in translation mode (e.g., as in...), Figure 6P In a computer system (e.g., 600), a user input (e.g., 650D, 650E, touch gesture, and / or air gesture) is detected pointing to a corresponding transcript of a corresponding speech in a conversation transcript, wherein the corresponding transcript of the corresponding speech is the speech of a corresponding conversation participant among two or more conversation participants. In response to detecting user input (e.g., 650D and / or 650E) pointing to a corresponding transcript of a corresponding speech in a conversation transcript: the computer system (e.g., 600) stops displaying conversation transcripts of two or more conversation participants (e.g., ...). Figure 6Q (As shown). In response to detecting user input pointing to a corresponding transcript of the corresponding speech in the session transcript (e.g., 650D and / or 650E): the computer system (e.g., 600) displays an amplified transcript user interface (e.g., 630, replacing the session transcript with an amplified transcript user interface) via one or more display generation components (e.g., 602), which includes an amplified version of the corresponding transcript of the corresponding speech of the corresponding session participant (e.g., Figure 6Q(616B at the location). In some embodiments, the session transcript is replaced with an amplified version of the corresponding transcript of the corresponding speech of the corresponding session participant. In some embodiments, the amplified version of the corresponding transcript is larger than the corresponding transcript pointed to by the user input and displayed as part of the session transcript. When an amplified transcript user interface (e.g., 630) including an amplified version of the corresponding transcript of the corresponding speech is displayed, the computer system (e.g., 600) detects the corresponding audio via one or more audio devices (e.g., 660D and / or 662C). In response to the detection of the corresponding audio: based on a determination that the corresponding audio includes speech that satisfies one or more of a first set of conditions (e.g., speech spoken by a user of the computer system, speech in a third language different from the first and second languages, and / or speech in the first language), the computer system (e.g., 600) (e.g., by replacing the amplified version of the corresponding transcript of the corresponding speech with a transcript of speech that satisfies one or more of the first set of conditions) updates the amplified transcript user interface (e.g., 630) to include the transcript of speech that satisfies one or more of the first set of conditions (e.g., amplified transcript), such as in Figure 6V Based on the determination that the corresponding audio includes speech that meets one or more conditions of a second group different from the first group (e.g., speech spoken by someone who is not a participant in the conversation, speech spoken by someone who is not a user of the computer system, speech spoken in a third language different from the first and second languages, and / or speech spoken in the second language), the computer system (e.g., 600) abandons updating the amplified transcript user interface to include transcripts (e.g., amplified transcripts) that meet the second group of conditions (e.g., and optionally, continues to display amplified versions of the corresponding transcripts of the corresponding speech as part of the amplified transcript user interface). Figure 6S As shown.
[0224] In some implementations, in response to user input (e.g., 650D and / or 650E) detecting a corresponding transcription of the corresponding speech in a session transcript, a computer system (e.g., 600) outputs the audio of the corresponding transcription via one or more audio devices (e.g., Figure 6Q (Ref. 672C). In some implementations, the speakerphone of the computer system's telephone is used instead of a headset and / or headphones worn by the user of the computer system to output the transcribed audio.
[0225] In some embodiments, when operating in translation mode and displaying an amplified version of the corresponding transcript of the corresponding speech of the corresponding session participant via one or more display generation components (e.g., as part of 630), the computer system (e.g., 600) detects the corresponding audio via one or more audio devices. In response to detecting the corresponding audio and based on a determination that the corresponding audio includes speech satisfying one or more of a first set of conditions (e.g., speech spoken by a user of the computer system, speech in a third language different from the first and second languages, and / or speech in the first language), the computer system (e.g., 600) outputs an audio translation (e.g., 672C, translation from the first language to the second language) of the speech satisfying the first set of conditions (e.g., 662C). In some embodiments, outputting the audio translation of the speech satisfying the first set of conditions (e.g., in the second language) via one or more audio devices includes outputting the audio translation via one or more speakers integrated into a device (e.g., one or more speakers in a telephone integrated into the computer system), which is displaying an amplified version of the corresponding transcript via one or more display generation components (e.g., 602), as in... Figure 6V In some embodiments, the audio translation (e.g., in a second language) of speech that meets one or more of the first set of conditions output via one or more audio devices does not include the audio translation output via a head-mounted speaker.
[0226] In some implementations, when operating in translation mode, an amplified version of the corresponding transcript of the corresponding speech of the corresponding session participant is displayed via one or more display generation components (e.g., 602). Figure 6R When 616B and / or as part of 630 is detected, the computer system (e.g., 600) detects the corresponding audio (e.g., 660D) via one or more audio devices. In response to the detection of the corresponding audio (e.g., 660D) and based on the determination that the corresponding audio includes speech satisfying one or more of a second set of conditions (e.g., it is a far-field speaker, speech spoken by someone who is not a participant in the conversation, speech spoken by a user of the computer system, speech spoken by someone who is not a user of the computer system, speech in a third language different from the first and second languages, and / or speech in the first language), an audio translation of the speech satisfying the second set of conditions is output (e.g., ...). Figure 6S(670D at the location, translating from a second language to a first language). In some embodiments, audio translation (e.g., in the first language) of speech (e.g., 660D) that meets a second set of one or more conditions is output via one or more audio devices (e.g., at a headset and / or headphones in a computer system) including outputting audio translation via one or more headphones (e.g., 600B, headset and / or headphones) of one or more audio devices (e.g., not outputting audio translation via one or more speakers in a telephone integrated into a computer system).
[0227] In some implementations, when an amplified transcript user interface (e.g., 630) displays an amplified version of the corresponding transcript including the corresponding speech, a computer system (e.g., 600) detects a request to display the session transcript (e.g., 610) via one or more input devices. Figure 6V (650F at the location). In response to detecting a request to display the session transcript, the computer system (e.g., 600) displays the session transcript (e.g., as a...) via one or more display generation components (e.g., 602). Figure 6W and / or Figure 6Y (part of 610), including transcripts (e.g., 660D) of speech that meet one or more of the second set of conditions (e.g., 612C, 614C, standard-size transcripts and / or non-amplified transcripts) (e.g., a conversation transcript includes transcripts of speech from two or more participants, while an amplified transcript user interface excludes transcripts of speech from at least one of the two or more participants).
[0228] In some implementations, when operating in translation mode, the computer system (e.g., 600) detects user input (e.g., 650C, touch gestures, and / or air gestures) pointing to a second corresponding transcription of a second corresponding speech in the session transcript, wherein the second corresponding transcription of the second corresponding speech is the speech of a second corresponding session participant (e.g., 680B) who is different from the corresponding session participant (e.g., 680A) among two or more session participants. In response to detecting user input (e.g., 650C) pointing to a second corresponding transcription of a second corresponding speech in the session transcript, the computer system (e.g., 600) abandons displaying the magnified transcript user interface (e.g., continues displaying...). Figure 6P The user interface for the session transcript is 610.
[0229] In some implementations, when operating in translation mode and displaying a session transcript (e.g., 610), the computer system (e.g., 600) detects corresponding audio via one or more audio devices (e.g., 662B). In response to detecting the corresponding audio and based on a determination that the corresponding audio includes speech satisfying one or more of a first set of conditions (e.g., speech spoken by someone who is not a participant in the session, speech spoken by a user of the computer system (e.g., 680A), speech in a third language different from the first and second languages, and / or speech in the first language), the computer system (e.g., 600) abandons outputting an audio translation of the speech satisfying one or more of the first set of conditions (e.g., optionally not outputting audio translation 672B, as indicated by the dashed line of audio translation 672B).
[0230] In some implementations, when operating in translation mode and displaying a session transcript (e.g., 610), the computer system (e.g., 600) detects corresponding audio via one or more audio devices (e.g., 660C). In response to the detection of the corresponding audio and based on a determination that the corresponding audio includes speech satisfying one or more of a second set of conditions (e.g., speech spoken by a far-field speaker, speech in a third language different from the first and second languages, and / or speech in the second language), the computer system (e.g., 600) outputs an audio translation of the speech satisfying the second set of conditions via one or more audio devices (e.g., at the computer system's headphones 600B and / or headsets) (e.g., ...). Figures 6L to 6M At 670C, in the first language).
[0231] In some implementations, the audio translation (e.g., in a first language) of speech that meets one or more of the second set of conditions is output via one or more audio devices (e.g., at a headset and / or headphones in a computer system). This includes outputting the audio translation via one or more speakers of one or more of the set of one or more audio devices (e.g., 600B, headset and / or headphones) (e.g., without outputting the audio translation via one or more speakers in a telephone integrated into the computer system).
[0232] In some embodiments, the session transcript is displayed (e.g., 610) when the computer system is locked (e.g., as indicated by 604A). In some embodiments, displaying the session transcript includes replacing the display of the lock screen user interface with the display of the session transcript.
[0233] In some implementations, when operating in translation mode (e.g., in...) Figure 6PIn response to the detection of user input (e.g., 650D, 650E, touch gestures, and / or air gestures) corresponding to a corresponding transcription of the corresponding speech in the conversation transcript, a computer system (e.g., 600) detects user input (e.g., 650D, 650E, touch gestures, and / or air gestures) corresponding to a corresponding transcription of the corresponding speech in the conversation transcript. An audio translation of the corresponding speech is output via one or more audio devices (e.g., a speaker in a telephone integrated into the computer system). Figure 6Q At point 672C, in a second language (e.g., audio translation of the corresponding speech). In some embodiments, the speakerphone of the computer system is used instead of a headset and / or headphones worn by the user of the computer system to output the audio translation of the corresponding speech.
[0234] In some embodiments, audio translation of the corresponding speech (e.g., in a second language) output via one or more audio devices includes outputting the audio translation via one or more speakers integrated into a device (e.g., 600A) (e.g., one or more speakers in a telephone integrated into a computer system), which is displaying the corresponding transcription (e.g., 616B and / or 618B) via one or more display generation components (e.g., 602). In some embodiments, audio translation of the corresponding speech (e.g., in a second language) output via one or more audio devices does not include outputting the audio translation via a headset speaker (e.g., 600B).
[0235] It should be noted that the above text regarding method 700 (for example, Figure 7 The details of the process described herein also apply in a similar manner to the methods described below. For example, method 800 may optionally include one or more features of the various methods described above with reference to method 700. For example, the computer system is the same computer system. Also, the audio output process of method 800 is used during the translation process of method 700. For the sake of brevity, these details will not be repeated below.
[0236] Figure 8 This is a flowchart illustrating a method for outputting audio according to some embodiments. Method 800 is performed at a computer system (e.g., 100, 300, 500, and / or 600) (e.g., smartphones, smartwatches, tablets, laptops, desktop computers, wearable devices, and / or headsets) communicating with one or more audio devices (e.g., one or more microphones, one or more speakers, one or more headphones with built-in microphones, one or more headsets, and / or wearable devices). Some operations in method 800 may be optionally combined, some operations may be optionally changed in order, and some operations may be optionally omitted.
[0237] The devices, methods, and / or computer-readable storage media described below enhance device operability and (e.g., by assisting users in providing appropriate input when operating / interacting with the device and reducing user errors) make the user-device interface more efficient. This, in turn, reduces power consumption and / or improves device battery life by enabling users to use the device more quickly and efficiently. Performing an operation when a set of conditions are met without requiring further user input (such as by starting audio output) enhances device operability by reducing unnecessary input and / or steps to navigate between different user interfaces or control groups, thereby reducing device energy consumption. Furthermore, starting audio output before detecting the end of a sentence (e.g., translation of detected speech) reduces output latency.
[0238] As described below, Method 800 provides an intuitive way to output audio. This method reduces the cognitive burden on users while listening to audio, such as during translation, thereby creating a more efficient human-computer interface. For battery-powered computing devices, it allows users to start listening to audio faster and more efficiently, saving power and increasing the time between battery charging sessions.
[0239] A computer system (e.g., 600) (e.g., via one or more audio devices, such as one or more microphones, one or more headphones with built-in microphones, one or more headsets, and / or one or more headphones) receives (802) audio including voice (e.g., Figures 6F to 6H 662A and / or Figures 6K to 6L The computer system (e.g., 600) receives (804) an input audio stream including speech, and upon receiving (804) the input audio stream, performs (806) speech processing on the input audio stream, including performing speech recognition to generate text (e.g., a transcription and / or translation of the speech from the input audio stream) based on the speech from the input audio stream. Figure 6J 682A to 682C and / or Figure 6N (682D to 682E). Upon receiving (804) an input audio stream including speech, the computer system (e.g., 600) determines (808) a position for inserting punctuation (e.g., 684A, 686A, 684B, 686B, 684D and / or 686D) within the text, wherein the position is not the end of the sentence. Upon receiving (804) an input audio stream including speech, the computer system (e.g., 600) inserts (810) punctuation (e.g., as shown in the image) at the determined position within the text. Figure 6J and / or Figure 6N(As shown). Upon receiving (804) an input audio stream including speech, before detecting the end of a sentence, and based on the inserted punctuation at a determined position within the text (e.g., in response to determining the position for inserting punctuation within the text and / or in response to inserting punctuation within the text), the computer system (e.g., 600) begins to output (812) the first portion of the text before the inserted punctuation (e.g., ) via one or more audio devices (e.g., 600A and / or 600B). Figure 6J The portion of the speech preceding 684A and / or 686A; Figure 6N Oral audio of the portion of speech preceding 684D and / or 686D (in the text) Figure 6G 672A and / or Figure 6L (e.g., the second part of the spoken audio after the inserted punctuation, without outputting the text).
[0240] In some implementations, performing speech processing on the input audio stream includes converting the speech from a first language (e.g., Figure 6J 682A1 and / or Figure 6N Translate 682D1 into a second language different from the first language (e.g., Figure 6J 682A2 and / or Figure 6N (682D2 in the original text). In some implementations, the speech of the audio input stream is in a first language, and the resulting text (which is based on the speech of the audio input stream) is in a second language.
[0241] In some embodiments, the location for inserting punctuation is determined based on a transcription (e.g., text) of the speech from the audio input stream in a first language. In some embodiments, the speech from the audio input stream is in a first language, and the location for inserting punctuation (e.g., inserting punctuation into the first language transcription) is determined using a transcription of the speech in the first language (e.g., not using a transcription in a second language). In some embodiments, the inserted punctuation is then carried over into the translated transcription in the second language.
[0242] In some implementations, the location for inserting punctuation is determined based on a second-language transcription (e.g., text) of the speech from the audio input stream. In some implementations, the speech from the audio input stream is in a first language, and the first-language transcription of the speech is used to translate the text into a second language. The second-language transcription is then used (e.g., without using the first-language transcription) to determine the location for inserting punctuation (e.g., inserting punctuation into the second-language transcription).
[0243] In some implementations, the punctuation inserted at the designated location within the text is artificial punctuation (e.g., inserted as a marker of location, but not for grammatical correctness). In some implementations, the punctuation inserted at the designated location within the text is grammatically incorrect and / or unnecessary, but helps to identify the designated location in the text.
[0244] In some implementations, the punctuation inserted at the determined location within the text is (e.g., grammatically correct or incorrect) a comma (e.g., 684A, 686A, 684B, 686B, 684D and / or 686D).
[0245] In some implementations, determining the location for inserting punctuation within the text includes determining the location based on the content of the speech (e.g., based on words, word order, and / or number of words).
[0246] In some implementations, the location for inserting punctuation within the text is determined independently of the duration of pauses (e.g., missing speech) between words (e.g., in detected speech). In some implementations, the location for inserting punctuation is independent of the rhythm of the user's speech (e.g., pacing, pitch, and / or volume variations) and / or the duration of pauses between spoken words.
[0247] It should be noted that the above text regarding method 800 (for example, Figure 8 The details of the process described also apply in a similar manner to the methods described above. For example, method 800 may optionally include one or more features of the various methods described above with reference to method 700. For example, the computer system is the same computer system. Also, the audio output process of method 800 is used during the translation process of method 700. For the sake of brevity, these details will not be repeated above.
[0248] For purposes of explanation, the foregoing description has been given by reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible based on the teachings above. These embodiments were chosen and described in order to best explain the principles of these techniques and their practical application. Others skilled in the art will thus be able to best utilize these techniques and the various embodiments with various modifications suitable for the particular intended use.
[0249] While this disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. It should be understood that such changes and modifications are considered to be included within the scope of this disclosure and examples as defined by the claims.
[0250] As described above, one aspect of this technology involves collecting and using data from various sources to improve the delivery of translated speech that may be of interest to the user. This disclosure anticipates that, in some instances, such collected data may include personal information that uniquely identifies or can be used to contact or locate specific individuals. Such personal information may include demographic data, location-based data, telephone numbers, email addresses, social network IDs, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.
[0251] This disclosure recognizes that the use of such personal information data in this technology can benefit users. For example, personal information data can be translated into another language.
[0252] This disclosure anticipates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will comply with robust privacy policies and / or privacy measures. For example, such entities should consider deleting stored content of conversations after translation is completed. Specifically, such entities should implement and adhere to privacy policies and measures recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable entity purposes and should not be shared or sold outside of these legitimate purposes. Furthermore, such collection / sharing should be conducted only after receiving informed consent from users. Additionally, such entities should consider taking any necessary steps to protect and safeguard the right to access such personal information data and ensure that other entities with access to personal information data comply with the privacy policies and procedures of other entities. Furthermore, such entities may subject themselves to third-party assessments to demonstrate compliance with widely accepted privacy policies and measures. Moreover, policies and measures should be adapted to the specific types of personal information data collected and / or accessed, and to applicable laws and standards, including considerations of specific jurisdictions. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while in other countries, health data may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy measures should be advocated for different types of personal data in each country.
[0253] Regardless of the foregoing, this disclosure also anticipates implementation schemes for users to selectively block the use or access to personal information data. That is, this disclosure anticipates providing hardware and / or software components to prevent or block access to such personal information data. In addition to providing "opt-in" and "opt-out" options, this disclosure also anticipates providing notifications related to access to or use of personal information. For example, users may be notified when downloading an application that their personal information data will be accessed, and then reminded again just before the application accesses the personal information data. As another example, users and / or session participants may be notified that a session is being translated.
[0254] Furthermore, the intent of this disclosure is that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. This risk can be minimized by limiting data collection and deleting data when it is no longer needed (such as at the end of a translated conversation). Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods.
[0255] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it is also contemplated that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present technology will not become inoperable due to the absence of all or part of such personal information data.
Claims
1. A method, the method comprising: At a computer system that communicates with one or more audio devices, one or more input devices, and one or more display generation components: When not operating in translation mode, which automatically detects spoken audio via the one or more audio devices and outputs a translation of the detected spoken audio, a set of one or more inputs for enabling the translation mode are detected via the one or more input devices; as well as In response to detecting one or more sets of inputs for enabling the translation mode, operation is performed in the translation mode, which includes the option to enable the translation mode without requiring additional user input: Detecting the speech of a conversation of two or more participants via the one or more audio devices, the speech including a first speech in a first language and a second speech in a second language different from the first language; The audio translation of the first speech and / or the audio translation of the second speech are output via the one or more audio devices; as well as The session transcript is displayed via one or more display generation components and simultaneously with the output of the audio translation of the session, the session transcript including transcriptions of the first and second speech of the session of the two or more session participants.
2. The method according to claim 1, further comprising: In response to detecting one or more inputs for enabling the translation mode, an instruction is output via one or more audio devices to the computer system for the language to which it will provide translation.
3. The method according to any one of claims 1 to 2, wherein: The audio translation of the first speech output via the one or more audio devices includes the audio translation of the first speech in the second language; and The audio translation of the second speech output via the one or more audio devices includes the audio translation of the second speech in the first language.
4. The method according to any one of claims 1 to 3, wherein the set of one or more inputs for enabling the translation mode includes a first touch input at a first earpiece and a second touch input at a second earpiece other than the first earpiece.
5. The method according to any one of claims 1 to 4, wherein: When the computer system is in locked mode, the set of one or more inputs for enabling the translation mode are detected; and The session transcript is displayed when the computer system is in the locked mode.
6. The method according to any one of claims 1 to 5, further comprising: When operating in the translation mode: Detect the corresponding speech via the one or more audio devices; as well as In response to the detection of the corresponding voice: Based on the determination that the corresponding speech satisfies one or more of the first group of conditions, the translation of the corresponding speech into the corresponding language is output.
7. The method of claim 6, wherein the corresponding language is selected based on the language of the virtual assistant operating at the computer system.
8. The method according to any one of claims 6 to 7, further comprising: When operating in the translation mode and in response to detecting the corresponding speech: Based on the determination that the corresponding speech satisfies one or more of the second set of conditions, output a translation of the corresponding speech in a second language that is different from the corresponding language.
9. The method according to claim 8, wherein: The output of the translation of the corresponding speech in the corresponding language includes outputting the translation of the corresponding speech via a first audio device in the group of one or more audio devices, without outputting the audio translation of the corresponding speech in the corresponding language via a second audio device in the group of one or more audio devices; and The translation of the corresponding speech into the second language includes outputting the translation of the corresponding speech via the second audio device in the group of one or more audio devices, without outputting the audio translation of the corresponding speech into the second language via the first audio device in the group of one or more audio devices.
10. The method according to any one of claims 1 to 9, further comprising: Before outputting the audio translation of the first speech and the audio translation of the second speech via the one or more audio devices, user input selecting the second language is received via the one or more input devices.
11. The method according to any one of claims 1 to 9, further comprising: Before outputting the audio translation of the first speech and the audio translation of the second speech via the one or more audio devices, the second language is automatically selected without receiving user input to select the second language.
12. The method of any one of claims 1 to 11, wherein detecting speech comprising a first speech in the first language and a second speech in a second language different from the first language of two or more session participants via the one or more audio devices comprises: The first speech in the first language is detected via the first microphone and the second microphone; as well as The second speech in the second language is detected via the first microphone and the second microphone.
13. The method according to any one of claims 1 to 12, further comprising: When operating in the translation mode, the ambient noise level is detected; as well as Based on the determination that the ambient noise level is higher than a threshold noise level, an indication is output regarding that at least a portion of the computer system should be moved closer to one of the two or more session participants.
14. The method according to any one of claims 1 to 13, wherein displaying the session transcript of the session of the two or more session participants comprises: The transcription of the first speech in the first language, the transcription of the first speech in the second language, the transcription of the second speech in the first language, and the transcription of the second speech in the second language are displayed via the one or more display generation components.
15. The method according to any one of claims 1 to 14, the method further comprising: When operating in the translation mode, user input is detected pointing to a corresponding transcription of the corresponding speech in the session transcript, wherein the corresponding transcription of the corresponding speech is the speech of the corresponding session participant among the two or more session participants; and In response to detecting user input pointing to the corresponding transcription of the corresponding speech in the session transcript: Stop displaying the session transcripts of the two or more session participants; as well as An amplified transcript user interface is displayed via one or more display generation components, the amplified transcript user interface including an amplified version of the corresponding transcript of the corresponding speech of the corresponding session participant; When the user interface of the amplified transcript, including the corresponding speech, is displayed, the corresponding audio is detected via the one or more audio devices; and In response to the detection of the corresponding audio: Based on the determination that the corresponding audio includes speech that satisfies one or more of the first set of conditions, the amplified transcript user interface is updated to include the transcript of the speech that satisfies one or more of the first set of conditions; as well as If the corresponding audio is determined to include speech that satisfies one or more conditions of a second group different from the first group of one or more conditions, then updating the amplified transcript user interface to include transcripts of speech that satisfy one or more conditions of the second group is abandoned.
16. The method according to claim 15, further comprising: In response to detecting user input pointing to the corresponding transcription of the corresponding speech in the session transcript: The corresponding transcribed audio is output via the one or more audio devices.
17. The method according to any one of claims 15 to 16, further comprising: When operating in the translation mode and displaying the amplified version of the corresponding transcript of the corresponding speech of the corresponding session participant via the one or more display generation components, the corresponding audio is detected via the one or more audio devices; as well as In response to detecting the corresponding audio and determining that the corresponding audio includes speech that satisfies one or more of the first set of conditions, an audio translation of the speech that satisfies one or more of the first set of conditions is output.
18. The method of claim 17, wherein the audio translation of the speech satisfying one or more of the first set of conditions is output via the one or more audio devices, the audio translation being output via one or more speakers integrated into the devices, the devices being displayed via the one or more display generation components as an amplified version of the corresponding transcription.
19. The method according to any one of claims 15 to 18, the method further comprising: When operating in the translation mode and displaying the amplified version of the corresponding transcript of the corresponding speech of the corresponding session participant via the one or more display generation components, the corresponding audio is detected via the one or more audio devices; as well as In response to detecting the corresponding audio and determining that the corresponding audio includes speech that satisfies one or more of the second set of conditions, an audio translation of the speech that satisfies one or more of the second set of conditions is output.
20. The method of claim 19, wherein the audio translation of the speech satisfying the second set of one or more conditions is output via the one or more audio devices comprises outputting the audio translation via one or more headphone speakers of the one or more sets of audio devices.
21. The method according to any one of claims 15 to 20, further comprising: When the user interface of the amplified transcript, which includes the amplified version of the corresponding transcript containing the corresponding speech, is displayed, a request to display the session transcript is detected via the one or more input devices; as well as In response to the detection of the request to display the session transcript, the session transcript is displayed via the one or more display generation components, including displaying transcripts of the speech that satisfy one or more of the second set of conditions.
22. The method according to any one of claims 15 to 21, further comprising: When operating in the translation mode, user input pointing to a second corresponding transcription of a second corresponding speech in the session transcript is detected, wherein the second corresponding transcription of the second corresponding speech is the speech of a second corresponding session participant among the two or more session participants who is different from the corresponding session participant; and In response to detecting user input pointing to the second corresponding transcript of the second corresponding speech in the session transcript, the display of the magnified transcript user interface is abandoned.
23. The method according to any one of claims 1 to 22, further comprising: When operating in the translation mode and displaying the session transcript, the corresponding audio is detected via the one or more audio devices; as well as In response to detecting the corresponding audio and determining that the corresponding audio includes speech that satisfies one or more of the first set of conditions, the audio translation of the speech that satisfies one or more of the first set of conditions is abandoned.
24. The method according to any one of claims 1 to 23, further comprising: When operating in the translation mode and displaying the session transcript, the corresponding audio is detected via the one or more audio devices; as well as In response to detecting the corresponding audio and determining that the corresponding audio includes speech that satisfies one or more of the second set of conditions, an audio translation of the speech that satisfies one or more of the second set of conditions is output via the one or more audio devices.
25. The method of claim 24, wherein the audio translation of the speech satisfying the second set of one or more conditions is output via the one or more audio devices comprises outputting the audio translation via one or more headphone speakers of the one or more sets of audio devices.
26. The method according to any one of claims 1 to 25, wherein the session transcript is displayed when the computer system is locked.
27. The method according to any one of claims 1 to 26, further comprising: When operating in the translation mode, user input corresponding to the corresponding transcription of the corresponding speech in the conversation transcript is detected; as well as In response to detecting user input corresponding to the corresponding transcript of the corresponding speech in the conversation transcript, an audio translation of the corresponding speech is output via the one or more audio devices.
28. The method of claim 27, wherein the audio translation of the corresponding speech output via the one or more audio devices comprises outputting the audio translation via one or more speakers integrated into the devices, the devices displaying the corresponding transcription via the one or more display generation components.
29. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more audio devices, one or more input devices, and one or more display generation components, the one or more programs including instructions for performing the method according to any one of claims 1 to 28.
30. A computer system configured to communicate with one or more audio devices, one or more input devices, and one or more display generation components, the computer system comprising: One or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the method according to any one of claims 1 to 28.
31. A computer system configured to communicate with one or more audio devices, one or more input devices, and one or more display generation components, the computer system comprising: Components for performing the method according to any one of claims 1 to 28.
32. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system communicating with one or more audio devices, one or more input devices, and one or more display generation components, the one or more programs comprising instructions for performing the method according to any one of claims 1 to 28.
33. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more audio devices, one or more input devices, and one or more display generation components, the one or more programs including instructions for: when not operating in a translation mode in which oral audio is automatically detected via the one or more audio devices and a translation of the detected oral audio is output, detecting, via the one or more input devices, a set of one or more inputs to enable the translation mode; as well as In response to detecting one or more sets of inputs for enabling the translation mode, operation is performed in the translation mode, which includes the option to enable the translation mode without requiring additional user input: Detecting the speech of a conversation of two or more participants via the one or more audio devices, the speech including a first speech in a first language and a second speech in a second language different from the first language; The audio translation of the first speech and / or the audio translation of the second speech are output via the one or more audio devices; as well as The session transcript is displayed via one or more display generation components and simultaneously with the output of the audio translation of the session, the session transcript including transcriptions of the first and second speech of the session of the two or more session participants.
34. A computer system configured to communicate with one or more audio devices, one or more input devices, and one or more display generation components, the computer system comprising: One or more processors; and The memory stores one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: When not operating in translation mode, which automatically detects spoken audio via the one or more audio devices and outputs a translation of the detected spoken audio, a set of one or more inputs for enabling the translation mode are detected via the one or more input devices; as well as In response to detecting one or more sets of inputs for enabling the translation mode, operation is performed in the translation mode, which includes the option to enable the translation mode without requiring additional user input: Detecting the speech of a conversation of two or more participants via the one or more audio devices, the speech including a first speech in a first language and a second speech in a second language different from the first language; The audio translation of the first speech and / or the audio translation of the second speech are output via the one or more audio devices; as well as The session transcript is displayed via one or more display generation components and simultaneously with the output of the audio translation of the session, the session transcript including transcriptions of the first and second speech of the session of the two or more session participants.
35. A computer system configured to communicate with one or more audio devices, one or more input devices, and one or more display generation components, the computer system comprising: A component for detecting a set of one or more inputs for enabling the translation mode via the one or more input devices when not operating in a translation mode that automatically detects spoken audio via the one or more audio devices and outputs a translation of the detected spoken audio; and Components for operating in the translation mode in response to detecting one or more sets of inputs for enabling the translation mode, the translation mode including the option to perform the translation without requiring additional user input: A component for detecting the speech of a conversation of two or more participants via the one or more audio devices, the speech including a first speech in a first language and a second speech in a second language different from the first language; Components for outputting audio translation of the first speech and / or audio translation of the second speech via the one or more audio devices; and A component for displaying a session transcript via the one or more display generation components and simultaneously outputting an audio translation of the session, the session transcript comprising transcripts of the first and second speech of the session of the two or more session participants.
36. A computer program product comprising one or more programs configured to execute by one or more processors of a computer system communicating with one or more audio devices, one or more input devices, and one or more display generation components, the one or more programs comprising instructions for performing the following operations: When not operating in translation mode, which automatically detects spoken audio via the one or more audio devices and outputs a translation of the detected spoken audio, a set of one or more inputs for enabling the translation mode are detected via the one or more input devices; as well as In response to detecting one or more sets of inputs for enabling the translation mode, operation is performed in the translation mode, which includes the option to enable the translation mode without requiring additional user input: Detecting the speech of a conversation of two or more participants via the one or more audio devices, the speech including a first speech in a first language and a second speech in a second language different from the first language; The audio translation of the first speech and / or the audio translation of the second speech are output via the one or more audio devices; as well as The session transcript is displayed via one or more display generation components and simultaneously with the output of the audio translation of the session, the session transcript including transcriptions of the first and second speech of the session of the two or more session participants.
37. A method, the method comprising: At a computer system that communicates with one or more audio devices: Receive input audio streams, including voice. as well as Upon receiving the input audio stream, which includes speech: Perform speech processing on the input audio stream, including performing speech recognition to generate text based on the speech of the input audio stream; Determine the position for inserting punctuation within the text, wherein the position is not the end of the sentence; Insert punctuation at the determined location within the text; as well as Before detecting the end of a sentence and based on the inserted punctuation at a determined position within the text, the spoken audio of the first portion of the text before the inserted punctuation is output via the one or more audio devices.
38. The method of claim 37, wherein performing speech processing on the input audio stream comprises translating the speech from a first language into a second language different from the first language.
39. The method according to any one of claims 37 to 38, wherein the position for inserting the punctuation is determined based on the transcription of the speech in the first language using the audio input stream.
40. The method according to any one of claims 37 to 39, wherein the position for inserting the punctuation is determined based on the transcription of the speech in the second language using the audio input stream.
41. The method according to any one of claims 37 to 40, wherein the punctuation inserted at the determined position within the text is artificial punctuation.
42. The method according to any one of claims 37 to 41, wherein the punctuation inserted at the determined position within the text is a comma.
43. The method according to any one of claims 37 to 42, wherein determining the position for inserting punctuation within the text comprises determining the position for inserting punctuation within the text based on the content of the speech.
44. The method according to any one of claims 37 to 43, wherein the position for inserting punctuation within the text is determined independently of the duration of the pause between words.
45. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more audio devices, the one or more programs including instructions for performing the method according to any one of claims 37 to 44.
46. A computer system configured to communicate with one or more audio devices, the computer system comprising: One or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the method according to any one of claims 37 to 44.
47. A computer system configured to communicate with one or more audio devices, the computer system comprising: Components for performing the method according to any one of claims 37 to 44.
48. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system communicating with one or more audio devices, the one or more programs comprising instructions for performing the method according to any one of claims 37 to 44.
49. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system communicating with one or more audio devices, the one or more programs including instructions for: Receive input audio streams, including voice. as well as Upon receiving the input audio stream, which includes speech: Perform speech processing on the input audio stream, including performing speech recognition to generate text based on the speech of the input audio stream; Determine the position for inserting punctuation within the text, wherein the position is not the end of the sentence; Insert punctuation at the determined location within the text; as well as Before detecting the end of a sentence and based on the inserted punctuation at a determined position within the text, the spoken audio of the first portion of the text before the inserted punctuation is output via the one or more audio devices.
50. A computer system configured to communicate with one or more audio devices, the computer system comprising: One or more processors; and The memory stores one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: Receive input audio streams, including voice. as well as Upon receiving the input audio stream, which includes speech: Perform speech processing on the input audio stream, including performing speech recognition to generate text based on the speech of the input audio stream; Determine the position for inserting punctuation within the text, wherein the position is not the end of the sentence; Insert punctuation at the determined location within the text; as well as Before detecting the end of a sentence and based on the inserted punctuation at a determined position within the text, the spoken audio of the first portion of the text before the inserted punctuation is output via the one or more audio devices.
51. A computer system configured to communicate with one or more audio devices, the computer system comprising: A component for receiving input audio streams, including speech; and Components for performing the following operations upon receiving the input audio stream, including speech: Perform speech processing on the input audio stream, including performing speech recognition to generate text based on the speech of the input audio stream; Determine the position for inserting punctuation within the text, wherein the position is not the end of the sentence; Insert punctuation at the determined location within the text; as well as Before detecting the end of a sentence and based on the inserted punctuation at a determined position within the text, the spoken audio of the first portion of the text before the inserted punctuation is output via the one or more audio devices.
52. A computer program product comprising one or more programs configured to execute by one or more processors of a computer system communicating with one or more audio devices, the one or more programs comprising instructions for: Receive input audio streams, including voice. as well as Upon receiving the input audio stream, which includes speech: Perform speech processing on the input audio stream, including performing speech recognition to generate text based on the speech of the input audio stream; Determine the position for inserting punctuation within the text, wherein the position is not the end of the sentence; Insert punctuation at the determined location within the text; as well as Before detecting the end of a sentence and based on the inserted punctuation at a determined position within the text, the spoken audio of the first portion of the text before the inserted punctuation is output via the one or more audio devices.
Citation Information
Patent Citations
Method and apparatus for integrating manual input
US20020015024A1
Gestures for touch sensitive input devices
US20060026521A1
Gestures for touch sensitive input devices
US20060026536A1
Virtual input device placement on a touch screen user interface
US20060033724A1
Multipoint touchscreen
US20060097991A1
