Receiver initiated mirroring session

Abstract

This disclosure relates in general to communication between computer systems, and more specifically to technologies for communicating user interface content.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Non - Provisional Application No. 18 / 379,128, filed on October 11, 2023, entitled "RECEIVER INITIATED MIRRORING SESSION", and U.S. Provisional Application No. 63 / 496,931, filed on April 18, 2023, entitled "RECEIVER INITIATED MIRRORING SESSION", which are hereby incorporated by reference in their entirety.

[0002] The present disclosure generally relates to communication between computer systems, and more specifically, to techniques for communicating user interface content.

Background Art

[0003] Peer - to - peer communication typically uses a distributed network architecture where computer systems on a network can function as both clients and servers, enabling them to directly exchange information and resources with each other.

Summary of the Invention

[0004] Current techniques for communicating data between computer systems are generally ineffective and / or inefficient. For example, some techniques require a user to open an application on a computer system and set up a connection with a different computer system through communication between the application and the different computer system. This disclosure provides a more effective and / or efficient technique for communicating data between computer systems, using the example of a smartphone application connecting to a wearable device. It should be noted that other types of computer systems can be used with the techniques described herein. For example, a smartphone can connect to a laptop using the techniques described herein. Furthermore, the techniques may optionally complement or replace other techniques for communicating data between computer systems.

[0005] Several techniques for communicating data between computer systems are described herein.

[0006] In some embodiments, a method performed on a first computer system is described. In some embodiments, the method includes receiving connection data for a second communication channel different from the first communication channel from the second computer system while the first computer system is connected to the second computer system via a first communication channel; using the connection data to connect to the second computer system via the second communication channel; and transmitting media output data to the second computer system via the second communication channel.

[0007] In some embodiments, a non-temporary computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of a first computer system. In some embodiments, one or more programs include instructions to receive connection data for a second communication channel different from the first communication channel from the second computer system while the first computer system is connected to the second computer system via a first communication channel, to use the connection data to connect to the second computer system via the second communication channel, and to transmit media output data to the second computer system via the second communication channel.

[0008] In some embodiments, a temporary computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of a first computer system. In some embodiments, one or more programs include instructions to receive connection data for a second communication channel, different from the first communication channel, from a second computer system while the first computer system is connected to the second computer system via a first communication channel, to use the connection data to connect to the second computer system via the second communication channel, and to transmit media output data to the second computer system via the second communication channel.

[0009] In some embodiments, a first computer system is described. In some embodiments, the first computer system comprises one or more processors and a memory that stores one or more programs configured to be executed by one or more processors. In some embodiments, one or more programs include instructions to receive connection data for a second communication channel different from the first communication channel from the second computer system while the first computer system is connected to the second computer system via a first communication channel, to use the connection data to connect to the second computer system via the second communication channel, and to transmit media output data to the second computer system via the second communication channel.

[0010] In some embodiments, a first computer system is described. In some embodiments, the first computer system performs the following steps: receiving connection data for a second communication channel different from the first communication channel from the second computer system while the first computer system is connected to the second computer system via a first communication channel; using the connection data to connect to the second computer system via the second communication channel; and transmitting media output data to the second computer system via the second communication channel. It is equipped with means to perform each of these actions.

[0011] In some embodiments, a computer program product is described. In some embodiments, a computer program product is described that includes one or more programs configured to run on one or more processors of a first computer system. In some embodiments, one or more programs include instructions to receive connection data from a second computer system for a second communication channel different from the first communication channel while the first computer system is connected to the second computer system via a first communication channel, to use the connection data to connect to the second computer system via the second communication channel, and to transmit media output data to the second computer system via the second communication channel.

[0012] In some embodiments, a method performed on a first computer system is described. In some embodiments, the method includes: sending a first request to a second computer system for the second computer system to join a communication channel, wherein the first request includes first connection data; sending a second request to the second computer system for the second computer system to join a communication channel, wherein the second request includes second connection data different from the first connection data, in accordance with the determination that a valid response has been received from the second computer system within a threshold period after sending the first request; and connecting to the second computer system via the communication channel in accordance with the determination that a valid response has been received from the second computer system within a threshold period.

[0013] In some embodiments, a non-temporary computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of a first computer system. In some embodiments, the one or more programs include instructions that connect to the second computer system via a communication channel, by sending a first request to the second computer system, the first request comprising first connection data, and after the first request has been sent, a valid response has not been received from the second computer system within a threshold period, and by sending a second request to the second computer system, the second request comprising second connection data different from the first connection data, and a valid response has been received from the second computer system within a threshold period.

[0014] In some embodiments, a temporary computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of a first computer system. In some embodiments, the one or more programs include instructions that connect to the second computer system via a communication channel, by sending a first request to the second computer system, the first request comprising first connection data, and after the first request has been sent, a valid response has not been received from the second computer system within a threshold period, and by sending a second request to the second computer system, the second request comprising second connection data different from the first connection data, and a valid response has been received from the second computer system within a threshold period.

[0015] In some embodiments, a first computer system is described. In some embodiments, the first computer system comprises one or more processors and a memory that stores one or more programs configured to be executed by one or more processors. In some embodiments, one or more programs include instructions to send to a second computer system a first request for the second computer system to join a communication channel, the first request comprising first connection data, and, in accordance with the determination that no valid response is received from the second computer system within a threshold period after the first request has been sent, to send to the second computer system a second request for the second computer system to join a communication channel, the second request comprising second connection data different from the first connection data, and, in accordance with the determination that a valid response is received from the second computer system within a threshold period, to connect with the second computer system via the communication channel.

[0016] In some embodiments, a first computer system is described. In some embodiments, the first computer system includes means for performing each of the following steps: sending a first request to a second computer system, the first request being a first request for the second computer system to join a communication channel, wherein the first request includes first connection data; sending a second request to the second computer system, the second request being a second request for the second computer system to join a communication channel, wherein the second request includes second connection data different from the first connection data, based on the determination that a valid response has been received from the second computer system within a threshold period after sending the first request; and connecting with the second computer system via the communication channel based on the determination that a valid response has been received from the second computer system within a threshold period.

[0017] In some embodiments, a computer program product is described. In some embodiments, a computer program product is described which includes one or more programs configured to be executed by one or more processors of a first computer system. In some embodiments, one or more programs include instructions to send to a second computer system a first request for the second computer system to join a communication channel, the first request including first connection data, and, according to a determination that no valid response is received from the second computer system within a threshold period after the first request has been sent, to send to the second computer system a second request for the second computer system to join a communication channel, the second request including second connection data different from the first connection data, and, according to a determination that a valid response is received from the second computer system within a threshold period, to connect with the second computer system via the communication channel.

[0018] In some embodiments, a method performed on a first computer system is described. In some embodiments, the method includes: sending to a second computer system one or more user interface elements of the first computer system and a mapping that includes one or more action identifiers corresponding to one or more UI elements of the first computer system; receiving from the second computer system an action identifier from among the one or more action identifiers, wherein the action identifier is associated with a UI element from among the one or more UI elements; and performing an action associated with the action identifier in response to receiving an action identifier associated with a UI element.

[0019] In some embodiments, a non-temporary computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of a first computer system. In some embodiments, one or more programs include instructions that send to a second computer system one or more user interface elements of the first computer system and a mapping that includes one or more action identifiers corresponding to one or more UI elements of the first computer system, and that receive from the second computer system an action identifier among the one or more action identifiers, wherein the action identifier is associated with one or more UI elements, and that perform the action associated with the action identifier in response to receiving the action identifier associated with the UI element.

[0020] In some embodiments, a temporary computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of a first computer system. In some embodiments, one or more programs include instructions that send to a second computer system one or more user interface elements of the first computer system and a mapping that includes one or more action identifiers corresponding to one or more UI elements of the first computer system, and that receive from the second computer system an action identifier among the one or more action identifiers, wherein the action identifier is associated with one or more UI elements, and that perform the action associated with the action identifier in response to receiving the action identifier associated with the UI element.

[0021] In some embodiments, a first computer system is described. In some embodiments, the first computer system comprises one or more processors and memory for storing one or more programs configured to be executed by the one or more processors. In some embodiments, one or more programs include instructions for sending to a second computer system one or more user interface elements of the first computer system and a mapping including one or more action identifiers corresponding to one or more UI elements of the first computer system, receiving from the second computer system an action identifier among the one or more action identifiers, wherein the action identifier is associated with one or more UI elements, and executing the action associated with the action identifier in response to receiving the action identifier associated with the UI element.

[0022] In some embodiments, a first computer system is described. In some embodiments, the first computer system comprises means for performing each of the following steps: sending to a second computer system one or more user interface elements of the first computer system and a mapping that includes one or more action identifiers corresponding to one or more UI elements of the first computer system; receiving from the second computer system an action identifier from among one or more action identifiers, wherein the action identifier is associated with one or more UI elements; and performing an action associated with the action identifier in response to receiving an action identifier associated with a UI element.

[0023] In some embodiments, a computer program product is described. In some embodiments, a computer program product is described that includes one or more programs configured to run on one or more processors of a first computer system. In some embodiments, one or more programs include instructions to send to a second computer system one or more user interface elements of the first computer system and a mapping including one or more action identifiers corresponding to one or more UI elements of the first computer system; to receive from the second computer system an action identifier among the one or more action identifiers, wherein the action identifier is associated with one or more UI elements; and to perform the action associated with the action identifier in response to receiving the action identifier associated with the UI element.

[0024] In some embodiments, a method is described that is performed on a first computer system. In some embodiments, the method includes receiving from a second computer system one or more user interface elements of the second computer system and a mapping that includes an action identifier mapped to a UI element among one or more UI elements of the second computer system, wherein the action identifier is associated with a first type of input in the second computer system; detecting an input associated with a UI element among one or more UI elements, wherein the input is a second type of input, different from a first type of input; and using the mapping, transmitting to the second computer system an action identifier associated with a first type of input, according to the determination that the input being a second type of input corresponds to a first type of input in the second computer system.

[0025] In some embodiments, a non-temporary computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of a first computer system. In some embodiments, one or more programs include instructions that receive from a second computer system one or more user interface elements of the second computer system and a mapping that includes an action identifier mapped to a UI element among one or more UI elements of the second computer system, wherein the action identifier is associated with a first type of input in the second computer system; that detect an input associated with a UI element among one or more UI elements, wherein the input is a second type of input, different from a first type of input; and that, using the mapping, transmit to the second computer system an action identifier associated with a first type of input, according to a determination that the input being a second type of input corresponds to a first type of input in the second computer system.

[0026] In some embodiments, a temporary computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of a first computer system. In some embodiments, one or more programs include instructions that receive from a second computer system one or more user interface elements of the second computer system and a mapping that includes an action identifier mapped to a UI element among one or more UI elements of the second computer system, wherein the action identifier is associated with a first type of input in the second computer system; that detect an input associated with a UI element among one or more UI elements, wherein the input is a second type of input, different from a first type of input; and that, using the mapping, transmit to the second computer system an action identifier associated with a first type of input, according to a determination that the input being a second type of input corresponds to a first type of input in the second computer system.

[0027] In some embodiments, a first computer system is described. In some embodiments, the first computer system comprises one or more processors and a memory storing one or more programs configured to be executed by the one or more processors. In some embodiments, the one or more programs receive, from a second computer system, a mapping including one or more user interface elements of the second computer system and action identifiers mapped to the UI elements among the one or more UI elements of the second computer system, the action identifiers being associated with a first type of input in the second computer system, detect an input associated with a UI element among the one or more UI elements, the input being a second type of input different from the first type of input, and transmit, to the second computer system, an action identifier associated with the first type of input according to a determination that the input, which is the second type of input, corresponds to the first type of input in the second computer system, the transmission being based on the mapping.

[0028] In some embodiments, a first computer system is described. In some embodiments, the first computer system performs the following steps: receiving, from a second computer system, a mapping including an action identifier mapped to a UI element among one or more UI elements of the second computer system, the action identifier being associated with a first type of input in the second computer system; detecting an input associated with the UI element among the one or more UI elements, the input being a second type of input different from the first type of input; and using the mapping, sending, to the second computer system, an action identifier associated with the first type of input according to a determination that the input that is the second type of input corresponds to the first type of input in the second computer system.

[0029] In some embodiments, a computer program product is described. In some embodiments, a computer program product is described that includes one or more programs configured to be executed by one or more processors of a first computer system. In some embodiments, the one or more programs receive, from a second computer system, a mapping including one or more user interface elements of the second computer system and an action identifier mapped to a UI element among the one or more UI elements of the second computer system, the action identifier being associated with a first type of input in the second computer system, detect an input associated with the UI element among the one or more UI elements, the input being a second type of input different from the first type of input, and using the mapping, send, to the second computer system, an action identifier associated with the first type of input according to a determination that the input that is the second type of input corresponds to the first type of input in the second computer system.

[0030] The executable instructions that perform these functions are optionally contained within a non-temporary computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0031] For a better understanding of the various embodiments described, please refer to the following “Modes for Carrying Out the Invention” in conjunction with the drawings. Similar reference numbers refer to corresponding parts throughout those drawings. [Brief explanation of the drawing]

[0032] [Figure 1] This is a diagram of a computing system according to several embodiments.

[0033] [Figure 2] This block diagram shows a device having interconnected subsystems according to several embodiments.

[0034] [Figure 3] An exemplary network diagram illustrating the relationship between two computer systems and a server, according to several embodiments, is shown.

[0035] [Figure 4] This document illustrates exemplary displays of user interface elements for two computer systems, according to several embodiments.

[0036] [Figure 5] A table is shown illustrating exemplary mappings in several embodiments.

[0037] [Figure 6] The following are illustrative communication diagrams of two computer systems according to several embodiments.

[0038] [Figure 7] The following are illustrative communication diagrams of two computer systems according to several embodiments.

[0039] [Figure 8] This is a flowchart illustrating a method for communicating data between devices, according to several embodiments. [Figure 9] This is a flowchart illustrating a method for communicating data between devices, according to several embodiments. [Figure 10] This is a flowchart illustrating a method for communicating data between devices, according to several embodiments. [Figure 11] This is a flowchart illustrating a method for communicating data between devices, according to several embodiments. [Modes for carrying out the invention]

[0040] The following description includes exemplary methods, parameters, etc. However, it should be noted that such descriptions are not intended to limit the scope of this disclosure, but rather are provided to describe exemplary embodiments.

[0041] Methods described herein may include one or more steps that are conditional on one or more conditions being met. It should be understood that a method is performed over multiple iterations of the same process, and different steps of the method may be met in different iterations. For example, if a method requires performing a first step in response to a determination that one or more criteria are met, and a second step in response to a determination that one or more criteria are not met, a person skilled in the art will understand that the steps of the method are repeated until both conditions are met, not in a specific order. Thus, a method described using conditional steps can be rewritten as a method repeated until each of the conditions described in the method is met. However, this is not required of a claim in a system or computer-readable medium if the claim includes instructions for performing one or more steps that are conditional on one or more conditions being met. Since instructions for a claim in a system or computer-readable medium are stored in one or more processors and / or one or more memory locations, a claim in a system or computer-readable medium includes logic that can determine whether one or more conditions have been met without explicitly repeating the steps of the method until all of the conditions on which the steps in the method are conditional are met. Those skilled in the art will also understand that, as with a method having conditional steps, a system or computer-readable storage medium may repeat the steps of the method as many times as necessary to ensure that all of the conditional steps have been performed.

[0042] In the following description, terms such as “first,” “second,” etc., are used to describe various elements, but these elements should not be limited by those terms. In some embodiments, these terms are used to distinguish one element from another. For example, without departing from the scope of the various described embodiments, a first subsystem may be called a second subsystem, and similarly, a subsystem device may be called a subsystem device. In some embodiments, the first subsystem and the second subsystem are two distinct references to the same subsystem. In some embodiments, the first subsystem and the second subsystem are both subsystems, but they are not identical subsystems or subsystems of the same type.

[0043] The terminology used in the descriptions of the various embodiments described herein is intended solely to describe specific embodiments and is not intended to be limiting. In the descriptions of the various embodiments and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless otherwise specified in the context. Furthermore, it should be understood that, as used herein, the terms "and / or" refer to and include any and all possible combinations of one or more of the enumerated items relating to the invention. It will be further understood that, as used herein, the terms "includes," "including," "comprises," and / or "comprising" specify the presence of the described functions, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other functions, integers, steps, actions, elements, components, and / or groups thereof.

[0044] The phrase "if" can be interpreted, at will, depending on the context, as meaning "when," "upon," "in response to determining," "in response to detecting," or "in accordance with a determination that." Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" can be interpreted, at will, depending on the context, as meaning "upon determining," "in response to determining," "upon detecting [the stated condition or event]," "in response to detecting [the stated condition or event]," or "in accordance with the determination that [a stated condition or event] is."

[0045] Referring to Figure 1, a block diagram of the computing system 100 is shown. Computing system 100 is a non-limiting example of a computing system that can be used to perform the functions described herein. It should be noted that other computer architectures of computing systems can be used to perform the functions described herein.

[0046] In the illustrated example, the computing system 100 includes a processor subsystem 110 that communicates (e.g., wired or wirelessly) with memory 120 (e.g., system memory) and an I / O interface 130 via an interconnect 150 (e.g., a system bus, one or more memory locations, or other communication channels for connecting multiple components of the computing system 100). In addition, the I / O interface 130 communicates (e.g., wired or wirelessly) with an I / O device 140. In some embodiments, the I / O interface 130 is included together with the I / O device 140 so that the two constitute a single component. It should be noted that there may be one or more I / O interfaces, and each I / O interface communicates with one or more I / O devices. In some embodiments, multiple instances of the processor subsystem 110 can communicate via the interconnect 150.

[0047] The computing system 100 may be any of various types of devices, including but not limited to a system-on-a-chip, a server system, a personal computer system (e.g., a smartphone, smartwatch, wearable device, tablet, laptop computer, and / or desktop computer), and a sensor. In some embodiments, the computing system 100 is included in or communicates with a physical component for the purpose of modifying the physical component in response to an instruction. In some embodiments, the computing system 100 receives an instruction to modify a physical component and causes the physical component to be modified in response to the instruction. In some embodiments, the physical component is modified via an actuator, an electrical signal, and / or an algorithm. Examples of such physical components include acceleration control, brakes, gearboxes, hinges, motors, pumps, refrigeration systems, springs, suspension systems, steering control, vacuum systems, and / or valves. In some embodiments, a sensor includes one or more hardware components that detect information about the physical environment adjacent to (e.g., surrounding) the sensor. In some embodiments, the hardware components of a sensor include sensing components (e.g., image sensors or temperature sensors), transmitting components (e.g., laser or wireless transmitters), receiving components (e.g., laser or wireless receivers), or any combination thereof. Examples of sensors include angle sensors, chemical sensors, brake pressure sensors, contact sensors, non-contact sensors, electrical sensors, flow sensors, force sensors, gas sensors, humidity sensors, image sensors (e.g., camera sensors, radar sensors, and / or LiDAR sensors), inertial measurement units, leak sensors, level sensors, light detection and ranging systems, metal sensors, motion sensors, particle sensors, photoelectric sensors, position sensors (e.g., global positioning systems), precipitation sensors, pressure sensors, proximity sensors, radio wave detection and ranging systems, radiation sensors, speed sensors (e.g., measuring the velocity of an object), temperature sensors, time-of-flight sensors, torque sensors, and ultrasonic sensors. In some embodiments, the sensor includes a combination of multiple sensors.In some embodiments, sensor data is captured by fusing data from one sensor with data from one or more other sensors. Although a single computing system is shown in Figure 1, computing system 100 can also be implemented as two or more computing systems working together.

[0048] In some embodiments, the processor subsystem 110 includes one or more processors or processing units configured to execute program instructions for performing the functions described herein. For example, the processor subsystem 110 can run an operating system, a middleware system, one or more applications, or any combination thereof.

[0049] In some embodiments, the operating system manages the resources of the computing system 100. Examples of operating system types covered herein include batch operating systems (e.g., Multiple Virtual Storage (MVS)), time-sharing operating systems (e.g., Unix), distributed operating systems (e.g., Advanced Interactive eXecutive (AIX)), network operating systems (e.g., Microsoft Windows Server), and real-time operating systems (e.g., QNX). In some embodiments, the operating system may include procedures, sets of instructions, software components, and / or drivers for controlling and managing common system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware and software components. In some embodiments, the operating system uses a priority-based scheduler that assigns priorities to different tasks that the processor subsystem 110 can perform. In such examples, the priority assigned to a task is used to identify the next task to be performed. In some embodiments, the priority-based scheduler identifies the next task to be performed when the previous task has finished execution. In some embodiments, the highest-priority task is performed until completion unless another higher-priority task is ready.

[0050] In some embodiments, the middleware system provides one or more services or capabilities, or both, to an external application (e.g., one or more applications running on the processor subsystem 110) that is not provided by the operating system (e.g., data management, application services, messaging, authentication, API management, etc.). In some embodiments, the middleware system is designed so that a heterogeneous computer cluster provides hardware abstraction, low-level device control, implementation of commonly used functions, inter-process message passing, package management, or any combination thereof. Examples of middleware systems include Lightweight Communications and Marshalling (LCM), PX4, Robot Operating System (ROS), and ZeroMQ. In some embodiments, the middleware system uses a graph architecture to represent processes and / or operations, with processing taking place in nodes that can receive, post, and multiplex sensor data messages, control messages, status messages, planning messages, actuator messages, and other messages. In such examples, the graph architecture can define an application (e.g., an application running on the processor subsystem 110 as described above) such that different operations of the application are contained in different nodes within the graph architecture.

[0051] In some embodiments, messages sent from a first node in a graph architecture to a second node in the graph architecture are performed using a publish-subscribe model, where the first node publishes data on a channel that the second node can subscribe to. In such examples, the first node can store the data in memory (e.g., memory 120 or some local memory of the processor subsystem 110) and notify the second node that the data has been stored in memory. In some embodiments, the first node notifies the second node that the data has been stored in memory by sending a pointer (e.g., a memory pointer such as identifying a memory location) to the second node so that the second node can access the data from the location where the first node stored the data. In some embodiments, the first node sends the data directly to the second node so that the second node does not need to access memory based on the data received from the first node.

[0052] The memory 120 may include computer-readable media (e.g., non-temporary or temporary computer-readable media) available for storing program instructions executable by the processor subsystem 110 in order to cause the computing system 100 to perform the various operations described herein. For example, the memory 120 may store program instructions for performing functions related to methods 800, 900, 1000, and 11000 described below.

[0053] Memory 120 can be implemented using a variety of physical, non-temporary storage media, such as hard disk storage devices, floppy disk storage devices, removable disk storage devices, flash memory, random access memory (RAM-SRAM, EDO RAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), and read-only memory (PROM, EEPROM, etc.). The memory of the computing system 100 is not limited to primary storage devices such as memory 120. The computing system 100 may also include other forms of storage devices, such as cache memory in the processor subsystem 110 and secondary storage devices on I / O devices 140 (e.g., hard drives, storage arrays, etc.). In some embodiments, these other forms of storage devices may also store program instructions executable by the processor subsystem 110 and perform the operations described herein. In some embodiments, the processor subsystem 110 (or each processor in the processor subsystem 110) includes a cache or other form of onboard memory.

[0054] The I / O interface 130 can be any type of interface configured to communicate with other devices. In some embodiments, the I / O interface 130 includes a bridge chip (e.g., a southbridge) from a front bus to one or more rear buses. The I / O interface 130 can communicate with one or more I / O devices (e.g., I / O device 140) via one or more corresponding buses or other interfaces. Examples of I / O devices include storage devices (hard drives, optical drives, removable flash drives, storage arrays, SANs, or their associated controllers), network interface devices (e.g., to local or wide area networks), sensor devices (e.g., cameras, radar, LiDAR, ultrasonic sensors, GPS, inertial measurement devices, etc.), and auditory or visual output devices (e.g., speakers, lights, display generating components, screens, projectors, etc.). In some embodiments, the computing system 100 communicates with a network via a network interface device (e.g., configured to communicate via Wi-Fi®, Bluetooth®, Ethernet, etc.). In some embodiments, the computing system 100 is directly wired to the network.

[0055] Other I / O devices (e.g., I / O device 140) optionally include one or more physical input controls (e.g., physical user interface elements) such as physical buttons (e.g., push buttons, rocker buttons, etc.), dials, rotatable input mechanisms (e.g., pressable or non-pressable), slider switches, joysticks, and click wheels. In some embodiments, the input controller(s) may optionally be coupled to (or not coupled to) one or more of the following: a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more buttons optionally include up / down buttons for controlling the volume of the speaker and / or microphone. One or more buttons optionally include push buttons (e.g., physical buttons 402 and 404 in Figure 4). One or more buttons may create and / or provide input as one or more different input patterns. Buttons may receive press inputs (e.g., button activation) and release inputs (e.g., button deactivation (e.g., release)). The button can alternatively receive press and hold inputs (e.g., a threshold period elapses before a release input is received) (also referred to herein as “press and hold input”). In some embodiments, the computing system communicates with one or more input devices (e.g., via wireless communication over wired communication). In some embodiments, one or more input devices include a touch-sensitive surface (e.g., a trackpad as part of a touch-sensitive display). In some embodiments, one or more input devices include one or more camera sensors (e.g., one or more optical sensors and / or one or more depth camera sensors), for example, to track user gestures as input (e.g., hand gestures and / or air gestures). In some embodiments, one or more input devices are integrated with the computing system. In some embodiments, one or more input devices are separate from the computing system 100.In some embodiments, an air gesture is a gesture detected without the user touching (or independently of) an input element that is part of the device, and is based on detected movement of a part of the user's body in the air, including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), movement of the user's body relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of the user's other hand relative to one hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture involving movement of the hand in a predetermined pose by a predetermined amount and / or speed, or a shake gesture involving rotation of a part of the user's body by a predetermined speed or amount).

[0056] In some embodiments, the I / O device 140 optionally includes a contact / motion module for detecting gesture input by the user. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different motion, timing, and / or intensity of the detected contact). Therefore, gestures are optionally detected by detecting a specific contact pattern. For example, detecting a finger tap gesture input (also called a tap input) includes detecting a finger down event, followed by a finger up (lift-off) event at the same position (or substantially the same position) as the finger down event (e.g., the icon position). For example, detecting a finger tap-and-hold gesture input (also called a tap-and-hold input) includes detecting a finger down event that continues at the same position (or substantially the same position) as the finger down event (e.g., at the icon position) for at least a threshold period. For example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger down event, followed by one or more finger drag events, and then a finger up (lift-off) event. In some embodiments, the computing system 100 detects one or more gesture inputs directed towards and / or interacting with the user interface of the computing system 100 (for example, to select a displayed virtual input control, such as an icon, slider, or toggle).

[0057] Figure 2 shows a block diagram of device 200 having interconnected subsystems. In the illustrated example, device 200 includes three distinct subsystems (i.e., a first subsystem 210, a second subsystem 220, and a third subsystem 230) that communicate with each other (e.g., wired or wirelessly) to form a network (e.g., a personal area network, a local area network, a wireless local area network, a metropolitan area network, a wide area network, a storage area network, a virtual private network, an enterprise internal private network, a campus area network, a system area network, and / or a controller area network). An example of a possible computer architecture of subsystems like those included in Figure 2 is shown in Figure 1 (i.e., computing system 100). Although three subsystems are shown in Figure 2, device 200 may include more or fewer subsystems.

[0058] In some embodiments, some subsystems are not connected to other subsystems (for example, the first subsystem 210 can be connected to the second subsystem 220 and the third subsystem 230, but the second subsystem 220 cannot be connected to the third subsystem 230). In some embodiments, some subsystems are connected via one or more wires, while others are connected wirelessly. In some embodiments, messages are set up between the first subsystem 210, the second subsystem 220, and the third subsystem 230 so that when an individual subsystem sends a message, the other subsystems receive the message (for example, via wires and / or buses). In some embodiments, one or more subsystems are wirelessly connected to one or more computing systems outside of device 200, such as a server system. In such examples, subsystems may be configured to communicate wirelessly with one or more computing systems outside of device 200.

[0059] In some embodiments, device 200 includes a housing that completely or partially encloses subsystems 210-230. Examples of device 200 include home appliances (e.g., refrigerators or air conditioning systems), robots (e.g., robotic arms or robotic vacuum cleaners), and vehicles. In some embodiments, device 200 is configured to navigate within a physical environment (with or without user input).

[0060] In some embodiments, one or more subsystems of device 200 are used to control, manage, and / or receive data from one or more other subsystems of device 200 and / or from one or more remote computing systems from device 200. For example, a first subsystem 210 and a second subsystem 220 may each be cameras that capture images, and a third subsystem 230 may use the captured images for decision-making. In some embodiments, at least a portion of device 200 functions as a distributed computing system. For example, a task may be divided into different parts, with a first part performed by the first subsystem 210 and a second part performed by the second subsystem 220.

[0061] Currently, attention is focused on technologies for communicating data between computer systems. Such technologies are described in the context of smartphone applications connecting to wearable devices. It should be recognized that other types of electronic devices may be used in conjunction with the technologies described herein. For example, controllers and / or accessories may be connected to other accessories using the technologies described herein. Furthermore, the technologies may optionally complement or replace other technologies for connecting computer systems.

[0062] Figure 3 shows an exemplary network diagram 300 illustrating the relationship between two computer systems (e.g., portable multifunction devices) (e.g., a smartwatch 310 and a smartphone 340) and a synchronization server 350, according to several embodiments. As shown in Figure 3, the smartwatch 310 communicates with the smartphone 340. The smartwatch 310 and / or smartphone 340 may include one or more of the features described with respect to the computing system 100 in Figure 1 and / or the device 200 in Figure 2. In some embodiments, the smartwatch 310 is a wearable device (e.g., a wristwatch) and the smartphone 340 is a smartphone, and both are associated with the same user account (e.g., logged into the same user account and / or trusted by it).

[0063] As shown in Figure 3, the smartwatch 310 and the smartphone 340 are connected via a first communication channel 320 and a second communication channel 330. For example, the smartwatch 310 and the smartphone 340 communicate via both the first communication channel 320 and the second communication channel 330 (e.g., exchanging data and / or messages) (e.g., sending and receiving messages simultaneously via both channels, alternately via both channels, and / or sporadically on each of the two channels as needed). In some embodiments, the smartwatch 310 and the smartphone 340 communicate via a single communication channel (e.g., the first communication channel 320 or the second communication channel 330). For example, the smartwatch 310 and the smartphone 340 may be connected only via the first communication channel 320, and communication between the smartwatch 310 and the smartphone 340 is exchanged via that channel. In some embodiments, the data communicated between the smartwatch 310 and the smartphone 340 includes media output data representing media output by the smartwatch 310 (e.g., display data representing what is displayed by the smartwatch 310). In some embodiments, the data communicated between the smartwatch 310 and the smartphone 340 includes display data displayed on the smartwatch 310 (e.g., data associated with and / or containing displayable content (e.g., a user interface), such as display mirroring data, also referred to herein as “screen mirroring data”). In some embodiments, the display data communicated between the smartwatch 310 and the smartphone 340 includes user interface element data for the smartwatch 310 (e.g., virtual and / or physical input controls).

[0064] In Figure 3, the smartwatch 310 and the smartphone 340 communicate via one or more peer-to-peer ("P2P") networks. In some embodiments, the first communication channel 320 is a P2P network. In some embodiments, the second communication channel 330 is a P2P network. For example, a P2P network is a network in which individual nodes within the network directly share and exchange resources with each other without relying on a central server or agency. For example, the smartwatch 310 sends and receives display data to and from the smartphone 340 via a P2P network, and similarly, the smartphone 340 sends and receives display data to and from the smartwatch 310 via a P2P network.

[0065] As shown in Figure 3, one or more of the smartwatch 310 and smartphone 340 communicate with the synchronization server 350. In some embodiments, the synchronization server 350 is a network node (e.g., a Wi-Fi router and / or connection point), a server and / or a third computer system (e.g., different from the smartwatch 310 and smartphone 340). The synchronization server 350 may include one or more of the features described with respect to the computing system 100 in Figure 1 and / or the device 200 in Figure 2.

[0066] In some embodiments, the synchronization server 350 assists and / or authorizes the pairing operation between the smartwatch 310 and the smartphone 340. In some embodiments, the synchronization server 350 authenticates that the user account associated with the smartwatch 310 is associated with and / or authorized to be associated with the smartphone 340 (for example, thereby enabling access to data and establishing communication).

[0067] In some embodiments, the synchronization server 350 acts as an intermediary for data communication between the smartwatch 310 and the smartphone 340. In some embodiments, the synchronization server 350 transmits and / or receives data to and from the smartwatch 310 and / or the smartphone 340. For example, the smartwatch 310 requests data from the smartphone 340 by sending a data request to the synchronization server 350, and the synchronization server 350 sends the request to the smartphone 340. In some embodiments, the synchronization server 350 receives data from the smartphone 340 and transmits data to the smartwatch 310. In some embodiments, the synchronization server 350 receives requests to and from the smartwatch 310 and / or the smartphone 340.

[0068] In some embodiments, the synchronization server 350 does not act as an intermediary for data communication between the smartwatch 310 and the smartphone 340. In some embodiments, a request received from the smartphone 340 causes the smartwatch 310 to send data (e.g., media output data (e.g., display data and / or audio data) and / or information associated with one or more UI elements within the media output data, as described in more detail herein) directly to the smartphone 340 via the first communication channel 320 and / or the second communication channel 330. In some embodiments, the synchronization server 350 acts as an intermediary for requests between the smartwatch 310 and the smartphone 340, but does not act as an intermediary for data exchange. For example, the smartwatch 310 requests data from the smartphone 340 by sending a data request to the synchronization server 350, and the synchronization server 350 sends the request to the smartphone 340. The smartphone 340 then communicates the requested data to the smartwatch 310 via the first communication channel 320 and / or the second communication channel 330.

[0069] In some embodiments, the synchronization server 350 provides authentication credentials (e.g., password and / or network information) to the smartphone 340 for use when establishing a connection to the smartwatch 310. For example, the smartwatch 310 sends a request to the synchronization server 350 to establish communication with the smartphone 340. The smartwatch 310 sends authentication credentials to the synchronization server 350 to join a second communication channel 330. The synchronization server 350 sends the authentication credentials to the smartphone 340, and the smartphone 340 joins the second communication channel 330 using the authentication credentials (e.g., by providing the credentials to the smartwatch 310).

[0070] In some embodiments, the synchronization server 350 is optional. For example, the smartwatch 310 and / or smartphone 340 can communicate directly (for example, to exchange requests and / or data) without being connected to the synchronization server 350 (for example, at a given time) and / or while connected to the synchronization server 350 (for example, without using the synchronization server 350 to exchange requests and / or data).

[0071] Figure 4 shows an exemplary representation of user interface elements of two computer systems (e.g., portable multifunction devices such as a smartwatch 410 and a smartphone 440) according to several embodiments. Figure 4 shows a smartwatch 410 communicating with a smartphone 440. The smartwatch 410 and / or smartphone 440 may each include one or more of the features described with respect to the computing system 100 in Figure 1, the device 200 in Figure 2, the smartwatch 310 in Figure 3, and / or the smartphone 340 in Figure 3. For example, the smartwatch 410 may correspond to the smartwatch 310, and the smartphone 440 may correspond to the smartphone 340, and both may be associated with the same user account (e.g., logged into and / or trusted by the same user account).

[0072] As shown in Figure 4, the smartwatch 410 and the smartphone 440 are connected via a communication channel 430. In some embodiments, the communication channel 430 represents a single communication channel. In some embodiments, the communication channel 430 represents multiple communication channels (e.g., a first communication channel 320 and a second communication channel 330). In some embodiments, the smartwatch 410 is paired with the smartphone 440 (e.g., via Bluetooth®) to exchange data (e.g., notifications, messages, images, calendar data, and / or other data).

[0073] As shown in Figure 4, the smartwatch 410 includes physical buttons 402, 404, and a rotatable input mechanism 406. In some embodiments, the smartwatch 410 includes more, fewer, and / or different input mechanisms. As shown in Figure 4, the smartwatch 410 displays a first user interface on the display 408, which includes user interface elements 412, 414, 416, and 418. In Figure 4, user interface elements 412, 414, 416, and 418 correspond to selectable representations of applications (which may also be called icons, affordances, controls, and / or complications). User interface element 412 corresponds to a representation of a calendar application. User interface element 414 corresponds to a representation of a weather application. User interface element 416 corresponds to a representation of an activity application. User interface element 418 corresponds to a representation of a music application. In some embodiments, the smartwatch 410 displays additional and / or alternative user interface elements.

[0074] In Figure 4, the smartwatch 410 communicates display data and / or one or more input control data to the smartphone 440 via the communication channel 430.

[0075] In some embodiments, input controls are virtual and / or physical elements (also called user interface elements) associated with a user interface. For example, user interface elements 412, 414, 416, and 418, physical buttons 402 and 404, and the rotatable input mechanism 406 are examples of input controls. In some embodiments, inputs associated with an input control (e.g., detected at its location) via a smartwatch 410 cause the smartwatch 410 to perform a corresponding action or function (e.g., one or more processes and / or operations).

[0076] In some embodiments, the display data includes screen mirroring data for the smartwatch 410. In some embodiments, the input control data includes input controls, including physical elements of the smartwatch 410 (e.g., physical buttons 402 and 404, and a rotatable input mechanism 406) and virtual UI elements (e.g., user interface elements 412, 414, 416, and 418). In some embodiments, the input control data for the smartwatch 410 includes mapping of input controls. In some embodiments, the mapping describes one or more details of the relationship between one or more input controls and one or more locations in the user interface displayed on the smartwatch 410 (e.g., where the input controls are located) and one or more available actions for the input controls. In some embodiments, the display data includes both mapping data and screen mirroring data for the smartwatch 410.

[0077] As shown in Figure 4, the smartphone 440 displays an accessibility interface via the display 438. For example, the smartphone 440 is used to display a representation of the smartwatch 410 on the display 438. It should be noted that the smartphone 440 can display other user interfaces using the techniques described herein. As shown in Figure 4, the representation of the smartwatch 410 includes a representation of the input controls of the smartwatch 410. In some embodiments, the accessibility interface provides one or more alternative and / or additional functions for the user interface of the smartwatch 410. In Figure 4, the smartphone 440 receives display data and / or input control data from the smartwatch 410 via the communication channel 430. The accessibility interface of the smartwatch 410 displayed on the smartphone 440 includes representations of physical buttons 432 and 434 (corresponding to physical buttons 402 and 404 of the smartwatch 410, respectively), a rotatable input mechanism 436 (corresponding to the rotatable input mechanism 406 of the smartwatch 410), and user interface elements 442, 444, 446, and 448 (corresponding to user interface elements 412, 414, 416, and 418 displayed on the smartwatch 410, respectively).

[0078] In some embodiments, the smartphone 440 displays display data via the display 438. For example, as shown in Figure 4, the accessibility interface displayed by the smartphone 440 includes display data for the user interface displayed by the smartwatch 410 (e.g., mirrored, same size or different size (e.g., larger or smaller)). In some embodiments, the smartphone 440 displays display data simultaneously with input control data. In some embodiments, the display data includes data associated with displayed content that is not input control (e.g., visual information relating to user interface element 412 and the background digits on the watch face in Figure 4). In some embodiments, the display data does not include data associated with displayed content that is not input control (e.g., visual information relating to user interface element 412, but not information relating to the background digits on the watch face in Figure 4).

[0079] The accessibility interfaces and / or accessibility features described herein can provide additional and / or alternative capabilities for interacting with the user interface (e.g., of the smartwatch 410). In some embodiments, the smartphone 440 acts as a proxy for receiving user input to the smartwatch 410, and as a result, the input received by the smartphone 440 produces one or more results on the smartwatch 410 as if the input had been received via the smartwatch 410. For example, input directed to the selection of user interface element 442 is received on the smartphone 440, and accordingly, the smartphone 440 communicates with the smartwatch 410 to perform a corresponding action on the smartwatch 410 as if the input had been directed to the selection of user interface element 412.

[0080] The ability to use a computer system (e.g., a smartphone 440) as a proxy can enable the use of one or more capabilities of one computer system in conjunction with the user interface of another computer system. In some embodiments, the smartphone 440 is used as a proxy for interacting with features and / or applications available to (e.g., installed on) (e.g., not available on the smartphone 440) on the smartwatch 410. For example, the smartphone 440 can receive input representing a selection of user interface elements 446 representing an activity application that is not installed on the smartphone 440. The smartphone 440 communicates with the smartwatch 410 to perform a corresponding action as if the smartwatch 410 had received input on a user interface element of the smartwatch 410, and accordingly the smartwatch 410 can display the corresponding activity application (e.g., which may be mirrored on the smartphone 440 according to the techniques described herein). The activity application can be interacted with via input on the smartphone 440 even if the application is not installed on the smartphone 440.

[0081] In some embodiments, the smartphone 440 is used as a proxy for interacting with features and / or applications that are not available on the smartwatch 410 (e.g., installed) (e.g., available on the smartphone 440). In some embodiments, the smartphone 440 provides additional input and / or output options other than those available on the smartwatch 410. For example, the smartphone 440 may include a screen reader function (e.g., an application) for reading aloud user interface data (e.g., displayed UI elements), which is not available on the smartwatch 410. In another example, the smartphone 440 may include a voice input function (e.g., an application) that accepts voice input for interacting with (e.g., selecting) user interface data (e.g., displayed UI elements), which is not supported by the smartwatch 410.

[0082] The ability to use a computer system as a proxy for another computer system in a way that extends its functionality may require the exchange of certain data (e.g., corresponding to a displayed user interface). In some embodiments, certain data includes mapping data (also referred to herein as “mapping”). The mapping data can provide sufficient information to the receiving (e.g., proxy) computer system (e.g., smartphone 440 in Figure 4) so ​​that it can use extended features (e.g., text for use by text-to-speech screen reading functionality, location information, and / or actions associated with UI elements (e.g., virtual buttons) so that they can interact with the UI elements). For example, smartphone 440 receives a mapping of user interface element 412 from smartwatch 410 via communication channel 430. User interface element 412 is displayed at a first location and has tap input and tap-and-hold input available actions. As shown in Figure 4, smartphone 440 displays user interface element 442 representing user interface element 412 (of smartwatch 410). The smartphone 440 detects a voice input corresponding to a selection of the representation of user interface element 442 (e.g., "select the music application icon"). Using mapping, the smartphone 440 determines that the input corresponds to a press action and transmits an indication of the press input and / or an indication of the action resulting from the press input to the smartwatch 410 via the communication channel 430. The result of this example is that the user can use voice commands to select an application. As used herein, “tap input” refers to an input associated with a specific virtual UI element (e.g., displayed on a touch-sensitive surface), and “press” refers to an input associated with a physical UI element (e.g., a physical pressable button) (e.g., not associated with a specific virtual UI element).

[0083] Figure 5 shows Table 500, which represents exemplary mappings according to several embodiments. While the mappings in Table 500 are shown as including specific data and / or specific arrangements of data, the data included in a mapping is not limited to the examples shown in Table 500. In some embodiments, a mapping may include more, less, and / or different data than that shown in Figure 5. In some embodiments, a mapping is transmitted from a first computer system (e.g., smartwatch 310 and / or smartwatch 410) to a second computer system (e.g., smartphone 340 and / or smartphone 440). The mappings in Table 500 are presented in table format simply as an exemplary tool. In some embodiments, the mappings are stored and / or exchanged in different formats suitable for grouping and / or illustrating relationships between data (e.g., available actions and locations for a given UI element).

[0084] As shown in Figure 5, the mapping is presented as each containing three types of information in a column. Each row contains an input control, an action available for the indicated input control, and a location associated with the input control. In some embodiments, the mapping location in a row may be empty, or otherwise indicate that all specific locations apply to individual input controls and / or available actions, or that there are no specific locations that apply to any input control and / or available action, because the available action (e.g., rotate) is not associated with a specific location in the corresponding user interface. As shown in Figure 5, the input controls are user interface elements 1-7. In some embodiments, each of user interface elements 1-7 corresponds to each of the input controls in Figure 4, including user interface elements 412, 414, 416, and 418, as well as the rotatable input mechanism 406, and the physical buttons 402 and 404 (respectively).

[0085] As shown in Figure 5, the available actions are Action 1, Action 2, Action 3, Action 4, Action 5, and Action 6. As shown in Figure 5, for some input controls, some of the available actions are the same, and some of the available actions are different. For example, UI element 7 can use the same actions as UI element 6, while UI element 1 can use different actions than UI element 6. In some embodiments, the available actions are inputs, actions, and / or inputs with actions. For example, an action can correspond to a type of input, where Action 1 is a tap input, Action 2 is a tap and hold input, Action 3 is a rotate down input, Action 4 is a rotate up input, Action 5 is a press input, and / or Action 6 is a press and hold input. For example, an action can correspond to an action resulting from an input associated with a UI element. Action 1 represents launching the corresponding application of the user interface element, Action 2 represents launching the settings menu of the corresponding user interface element, Action 3 represents scrolling down the displayed content, Action 4 represents scrolling up the displayed content, Action 5 represents displaying a third user interface, and Action 6 represents displaying a fourth user interface. In some embodiments, additional and / or alternative actions are available for each input control.

[0086] As shown in Figure 5, locations associated with each input control are enumerated. In some embodiments, a location is the position of the input control relative to the user interface (e.g., what is displayed using it) and / or the display component (e.g., what it is displayed on). For example, a location is a set of coordinates of one or more user interface elements (e.g., user interface elements 412, 414, 416, and 418) and / or the position of the physical inputs of the individual computer system (e.g., physical buttons 432 and 434, and / or rotatable input mechanism 436). In some embodiments, the coordinates define the boundaries of the user interface elements (e.g., the size and / or dimensions of the user interface elements). In some embodiments, the location defines the shape of the user interface elements. In some embodiments, the location includes the orientation of the respective computer system.

[0087] As shown in Figure 5, row 510 contains UI element 1 at location 1 and includes actions 1 and 2 as available actions. Row 520 contains UI element 2 at location 2 and includes actions 1 and 2 as available actions. Row 530 contains UI element 3 at location 3 and includes actions 1 and 2 as available actions. Row 540 contains UI element 4 at location 4 and includes actions 1 and 2 as available actions. Row 550 contains UI element 5 at location 5 and includes actions 3, 4, 5, and 6 as available actions. Row 560 contains UI element 6 at location 6 and includes actions 5 and 6 as available actions. Row 570 contains UI element 7 at location 7 and includes actions 5 and 6 as available actions.

[0088] In some embodiments, each of the UI elements 1 to 7 in Figure 5 corresponds to the input control shown in Figure 4. For example, UI element 1 is user interface element 412 in Figure 4, which corresponds to a representation of a calendar application. UI element 1 is associated with action 1 and action 2, where action 1 is a tap input and action 2 is a tap-and-hold input. In some embodiments, UI element 1 is associated with action 1 and action 2, where action 1 is an action to launch the associated application (e.g., the calendar application) and action 2 is an action to launch the settings menu of the associated application. The location of UI element 1 is location 1, which corresponds to the UI element at the top center of the first user interface displayed via the display on the smartwatch 410 (e.g., display 408). As another example, UI element 2 is user interface element 414 in Figure 4, which corresponds to a representation of a weather application. UI element 2 is associated with action 1 and action 2, where action 1 is a tap input and action 2 is a tap-and-hold input. In some embodiments, UI element 2 is associated with actions 1 and 2, where action 1 is the operation to launch the associated application (e.g., a music application), and action 2 is the operation to launch the settings menu of the associated application. The location of UI element 2 is location 2, which corresponds to the UI element in the right center of the first user interface displayed via the display 408 on the smartwatch 410. For example, UI element 3 is user interface element 416 in Figure 4, which corresponds to the representation of the activity application. UI element 3 is associated with actions 1 and 2, where action 1 is a tap input and action 2 is a tap-and-hold input. In some embodiments, UI element 1 is associated with actions 1 and 2, where action 1 is the operation to launch the associated application (e.g., an activity application), and action 2 is the operation to launch the settings menu of the associated application.The location of UI element 3 is location 3, which corresponds to the lower-center UI element of the first user interface displayed via the display on the smartwatch 410 (e.g., display 408). For example, UI element 4 is user interface element 418 in Figure 4, which corresponds to the representation of a music application. UI element 4 is associated with actions 1 and 2, where action 1 is a tap input and action 2 is a tap-and-hold input. In some embodiments, UI element 1 is associated with actions 1 and 2, where action 1 is the action of launching the associated application (e.g., a music application) and action 2 is the action of launching the settings menu of the associated application. The location of UI element 4 is location 4, which corresponds to the left-center UI element of the first user interface displayed via the display on the smartwatch 410 (e.g., display 408). In yet another example, UI element 5 is the rotatable input mechanism 406 in Figure 4. UI element 5 is associated with actions 3, 4, 5, and 6, where action 3 is a rotation down input and action 4 is a rotation up input. Action 5 is a press input, and Action 6 is a press-and-hold input. In some embodiments, UI element 5 is associated with Actions 3, 4, 5, and 6, where Action 3 is the action of scrolling down the user interface to display additional content, Action 4 is the action of scrolling up the user interface to display additional content, Action 5 is the action of displaying a third user interface, and Action 6 is the action of displaying a fourth user interface. The location of UI element 6 is in the upper right of the smartwatch 410. For example, UI element 6 is the physical button 402 in Figure 4. UI element 6 is associated with Actions 5 and 6, where Action 5 is a press input and Action 6 is a press-and-hold input.In some embodiments, UI element 6 is associated with actions 5 and 6, where action 5 is an operation to display a third user interface and action 6 is an operation to display a fourth user interface. The location of UI element 6 is location 6, which corresponds to the leftmost physical button of the smartwatch 410. For example, UI element 7 is the physical button 404 in Figure 4. UI element 6 is associated with actions 5 and 6, where action 5 is a press input and action 6 is a press and hold input. In some embodiments, UI element 7 is associated with actions 5 and 6, where action 5 is an operation to display a third user interface and action 6 is an operation to display a fourth user interface. The location of UI element 7 is location 7, which corresponds to the rightmost physical button of the smartwatch 410.

[0089] Figure 6 shows an exemplary communication diagram of two computer systems (e.g., a portable multifunction device such as a smartwatch and a smartphone) according to several embodiments. As shown in Figure 6, Figure 600 includes a first computer system 602 communicating with a second computer system 604. The first computer system 602 and / or the second computer system 604 may include one or more of the features described with respect to the computing system 100 in Figure 1, the device 200 in Figure 2, the smartwatch 310 in Figure 3, the smartwatch 410 in Figure 4, the smartphone 340 in Figure 3, and / or the smartphone 440 in Figure 4. For example, the first computer system 602 may be a wearable device (e.g., a wristwatch), and the second computer system 604 may be a smartphone, both associated with the same user account (e.g., logged into the same user account and / or trusted by it).

[0090] In 606, the first computer system 602 and the second computer system 604 are connected via a first communication channel (e.g., a first communication channel 320 and / or a second communication channel 430). For example, the first computer system 602 and the second computer system 604 are connected via the first communication channel in response to user input received by one or more of the respective computer systems (e.g., as part of a pairing operation). In some embodiments, the first communication channel is established by pairing between the first computer system 602 and the second computer system 604. For example, both the first computer system 602 and the second computer system 604 may be paired by associating each with the same user account (e.g., logged into and / or trusted by the same user account). In some embodiments, while the first computer system 602 and the second computer system 604 are connected via the first communication channel, the second computer system 604 receives a request to connect with first connection data.

[0091] In 608, the second computer system 604 sends a request to the first computer system 602 via the first communication channel to connect to (e.g., include) the first connection data. For example, upon receiving input corresponding to the selection of a user interface object, the second computer system 604 sends a request in 608 to the first computer system 602 to connect to the first connection data. In some embodiments, the first connection data includes network information and / or credentials that can be used to connect to the second communication channel. In some embodiments, the connection request includes a request to connect via the second communication channel. For example, the connection request is a request to join the second communication channel using the first connection data. In some embodiments, the connection request includes a request to send display data from the first computer system 602 to the second computer system 604 via the second communication channel (e.g., block 620 described below). For example, a connection request includes an instruction to transmit display data from the first computer system 602 via the second communication channel in response to connecting with the second computer system 604 via the second communication channel. In some embodiments, the second computer system 604 optionally continues to send additional requests, such as 610, and the second computer system 604 sends a request to connect with second connection data via the first communication channel. In some embodiments, the connection data differs between the first and second connection data. In some embodiments, additional requests (such as those described in 608 and / or 610 and / or similarly) are automatically sent (e.g., via a communication channel such as the first communication channel) if a valid response is not received by 614 within a threshold time.

[0092] In some embodiments, the first computer system 602 attempts to establish a connection with the second computer system 604 via a second communication channel by receiving a connection request at 608, receiving any additional requests (e.g., at 610), and, in response to receiving a request, sending a response at 614. In some embodiments, the first computer system 602 attempts to establish a connection after each request (e.g., in conjunction with and / or in response to) and / or automatically based on each request. At 614, the first computer system 602 sends a response to the second computer system 604 via the second communication channel. In some embodiments, the response includes the first connection data received at 608. In some embodiments, if multiple instances of the connection data have been sent to the first computer system 602 by the second computer system 604 (e.g., in requests), the first computer system 602 sends the most recent connection data in the response (e.g., if subsequent credentials have been issued, credentials issued earlier in time become invalid). For example, if the second computer system 604 sends a connection request 608 (using the first connection data) and a connection request 610 (using the second connection data) to the first computer system 602, the first computer system 602 sends the second connection data in response at 614.

[0093] At 616, the second computer system 604 optionally checks at 636 that the received credentials are valid (e.g., up-to-date). In some embodiments, the second computer system 604 checks at 614 that the received credentials are valid in response to the response received. In some embodiments, if the second computer system 604 determines at 636 that the credentials are invalid (e.g., not up-to-date), the second computer system 604 returns to 608 and transmits connection data. In some embodiments, when the second computer system 604 returns to 608, the connection data is the same as the first connection data previously transmitted to the first computer system 602. In some embodiments, when the second computer system 604 returns to 608 to transmit connection data, the connection data is different from when the second computer system 604 first transmitted the first connection data at 608. In some embodiments, the process by which the second computer system 604 sends an additional connection request (e.g., in 610) includes connection data that is different from the connection data previously sent to the first computer system 602 (e.g., including third connection data that is different from the first and second connection data). In some embodiments, the second computer system 604 continues to send connection data and check that the credentials are up-to-date until, in 616, the second computer system 604 determines that the credentials are up-to-date and / or that the termination conditions have been met (e.g., the time or number of attempts has elapsed). As shown by 640, each of 606, 608, 610, 614, 616, and 636 occurs while the first communication channel is maintained.

[0094] In 618, in response to the first computer system 602 receiving valid (e.g., up-to-date) credentials, the second computer system 604 and the first computer system 602 connect via a second communication channel. In some embodiments, the second computer system 604 and the first computer system 602 connect via the second communication channel in 618, while the first communication channel is maintained. In some embodiments, the response transmitted in 614 includes one or more communications (e.g., messages, packets, data, and / or frames) associated with connecting via the second communication channel in 618 (e.g., forming a connection, establishing a connection, and / or verifying a connection). For example, connection data received in 608 and / or additional requests from the second computer system 604 cause the first computer system 602 to attempt to connect to the second computer system 604 and / or succeed in connecting, in response to receiving the requests. In some embodiments, after the first computer system 602 connects with the second computer system 604 via a second communication channel, the first computer system 602 transmits display data 620 and / or user interface element data in 622.

[0095] In 620, the first computer system 602 transmits display data 620 to the second computer system 604 via the second communication channel. In 622, the first computer system 602 transmits user interface element data (as described herein) to the second computer system 604 via the first communication channel. In some embodiments, 620 and 622 are performed simultaneously and / or in the reverse order described herein. In some embodiments, transmitting display data in 620 includes transmitting screen mirroring data. In some embodiments, transmitting user interface element data in 622 includes mapping input control data (e.g., or a portion thereof) of Figure 4 via the first communication channel and / or the exemplary mapping of Figure 5. For example, the first computer system 602 transmits input control such as the representation of user interface element 412 and rotatable input mechanism 406, as described above with respect to Figure 4. In another example, the first computer system 602 transmits an exemplary mapping between UI element 1 shown in line 510 and UI element 6 shown in line 560. In some embodiments, in response to and / or after receiving display data and / or user interface element data, the second computer system 604 displays the display data and / or user interface element data received from the first computer system 602 in 620 and 622. In some embodiments, the second computer system 604 receives input directed to the displayed user interface element data in 624.

[0096] In some embodiments, transmitting display data in 620 includes transmitting audio data from the first computer system 602 via a second communication channel. In some embodiments, the audio data is a digital representation of an audio signal generated or processed by the first computer system 602. For example, the audio data transmitted from the first computer system 602 corresponds to the display data transmitted in block 620. As another example, the audio data transmitted from the first computer system 602 represents an audio signal output by the first computer system 602. In some embodiments, the first computer system 602 transmits the audio data while the first computer system 602 transmits the display data (e.g., together, together, and / or simultaneously). In some embodiments, the first computer system 602 transmits the audio data before and / or after the first computer system 602 transmits the display data.

[0097] In 624, the second computer system 604 receives inputs corresponding to user interface elements for which corresponding data was received in 622. For example, the second computer system 604 receives inputs associated with a user interface element (e.g., at its location) (e.g., selection of display data displayed on a touchscreen interface). In some embodiments, the input received by the computer system (e.g., the second computer system 604) (e.g., in 624) is an input detected via a component that communicates with the computer system. For example, the received input could be a touch input (e.g., tap, gesture, and / or selection) received via an input device that is a touch-sensitive surface (e.g., part of and / or connected to) that communicates with the computer system (e.g., the second computer system 604). In some embodiments, the input received by the computer system (e.g., the second computer system 604) (e.g., in 624) is an input received from another computer system (e.g., a synchronization server 350) that is different from the computer system (e.g., different from the second computer system 604). For example, a third computer system (e.g., different from the first and / or second computer systems) can detect an input and communicate the input and / or its representation to the second computer system 604, and receiving such communication corresponds to receiving an input.

[0098] In some embodiments, the input received at 624 corresponds to an action available for a user input element, as shown in the mapping received at 622. In one example, the input at 624 is the input for UI element 1 in row 510 of Figure 5. In such an example, UI element 1 corresponds to user interface element 442. The available actions are tap input and / or tap-and-hold input. In some embodiments, the received input is a touch input. In some embodiments, the received input is of a different type from the available actions. For example, the received input is a voice input representing a request to tap the input on UI element 1, and the second computer system 604 determines that the voice input corresponds to an available action for a tap input on UI element 1. In another example, the input at 624 corresponds to the input for a selection of UI element 5 represented by row 560 of Figure 5, and the available actions are rotate-up input, rotate-down input, press input, and press-and-hold input. In such an example, UI element 5 corresponds to a representation of a rotatable input mechanism 436. The received input is a rotation-up input on the representation of the rotatable input mechanism 406, which corresponds to the rotation of the (e.g., physical) rotatable input mechanism 436 (e.g., towards the top of the UI displayed when viewing the display). In some embodiments, in response to the input received at 624, the second computer system 604 determines an action identifier at 626.

[0099] In 626, the second computer system 604 determines an action identifier corresponding to the input received in 624. In some embodiments, the second computer system 604 determines the action identifier using a mapping received by the first computer system 602.

[0100] In some embodiments, the action identifier is an identifier for an action associated with a user interface element (e.g., the UI element or user interface) (e.g., available and / or corresponding). In some embodiments, the action identifier is any data appropriate to represent the action. For example, the action identifier can be a string, a value, a number, a character, and / or a variable. For example, when a second computer system 604 transmits an action identifier to the first computer system 602 in 628, the action identifier transmitted by the second computer system 604 is a value and / or a string. In some embodiments, the action identifier identifies the type of input (e.g., an identifier indicating which UI element the input applies to). For example, a tap input has an identifier that is represented as "tap_input" in communication between computer systems. In some embodiments, the action identifier identifies the type of input for a particular UI element (e.g., so the identifier of the UI to which the tap corresponds does not need to be transmitted). For example, a tap input has the identifier "tap_input_21", and there are at least 21 different tap inputs available for a given UI, and each action identifier for a tap input maps to a different UI element.

[0101] For example, referring back to Figure 4, the representational tap input of user interface element 442 corresponds to the tap input on user interface element 412 on the smartwatch 410. In this example, the action identifier indicates the tap input on user interface element 412. For example, referring back to Figure 4, the representational rotation down input of rotatable input mechanism 436 corresponds to the rotation down input of rotatable input mechanism 406. In this example, the action identifier indicates the rotation down input.

[0102] In some embodiments, the action identifier is an identifier for an input and an action performed on the first computer system 602. For example, referring back to Figure 4, a tap input on the representation of user interface element 442 corresponds to an action to launch a calendar application on the smartwatch 410. In this example, the action identifier identifies the tap input on user interface element 442 and the action to launch the calendar application. For example, referring back to Figure 4, an input to rotate the representation of rotatable input mechanism 436 has a corresponding action to scroll content down on the first user interface. In this example, the action identifier indicates the action to scroll content down. In some embodiments, in response to the determination of the action identifier in 626, the second computer system 604 transmits the action identifier.

[0103] In 628, the second computer system 604 transmits an action identifier to the first computer system 602 via the first communication channel. In some embodiments, upon receiving the action identifier in 628, the first computer system 602 performs an action in 630. In 630, the first computer system 602 performs the action (e.g., input and / or operation) indicated by the received action identifier. In some embodiments, the action corresponds to an action identifier communicated from the second computer system 604. For example, the action identifier received by the first computer system 602 identifies a tap input on user interface element 412 (represented by the representation of user interface element 442 in Figure 4) that corresponds to an operation to launch the calendar application, as if the input had been received directly on user interface element 412. For example, the action identifier received by the first computer system 602 may identify an operation to launch the calendar application. In 630, the corresponding action is to launch the calendar application. In another example, the action identifier received by the first computer system 602 is a rotation down input of the rotatable input mechanism 406 (for example, since the rotatable input mechanism 436 is a virtual representation of the rotatable input mechanism 406, a physical rotation down of the rotatable input mechanism 406 can correspond to a rotation down input on 436, a physical rotation up of the rotatable input mechanism 406 can correspond to a rotation up input on 436, and a press (for example, if the rotatable input mechanism 406 is press-down) can be a tap input on 436). The corresponding action is to scroll down the content as if the rotatable input mechanism 406 had been directly rotated (for example, a rotation input is registered without the rotatable input mechanism being physically rotated, and rather the action identifier functions as a substitute for physical rotation to produce the same or similar action). For example, the action identifier received by the first computer system 602 identifies the action of scrolling down the content on the first user interface.The corresponding action is to scroll down the content on the first user interface.

[0104] In some embodiments, performing an action updates the display of the first computer system 602. For example, launching the calendar application associated with the user interface element 412 in Figure 4 (e.g., due to an action received, which identifies an action and / or receives input associated with the calendar UI element) replaces the first user interface with a second user interface representing the calendar application. In another example, if the user interface is scrollable, scrolling the content on the user interface displays additional information by scrolling the content on the user interface, such that the rotatable input mechanism 406 scrolls the content on the display as if it had been rotated. For example, if the second user interface is a displayed representation of the calendar application, a rotate-down input associated with the rotatable input mechanism 406 (e.g., registered as an input corresponding to a rotate-down on the UI element 436) causes the first computer system 602 to display additional calendar events than before the scroll-down. In some embodiments, after an action is performed in 630 and / or after the display is updated in the first computer system 602, the first computer system 602 transmits the updated display data in 632 and / or the updated user interface element data in 634.

[0105] In 632, the first computer system 602 transmits the updated display data to the second computer system 604 via a second communication channel. For example, referring to Figure 4, transmitting the updated display data in 632 includes transmitting updated screen mirroring data from the second user interface when the calendar application is launched. For example, after the calendar application is launched, display data is transmitted that includes a visual representation of the user interface and UI elements of the calendar application's landing page. Such UI elements may include boxes representing calendar events, dates, days of the week, months, and / or years. In some embodiments, the updated screen mirroring data includes transmitting only the display data that has been changed and / or updated from the display data previously transmitted in 620. For example, if the input results in a rotation-down operation where some previously displayed UI elements are still visible, but a new UI element is now displayed, only the data for the new UI element may be transmitted. In some embodiments, transmitting the updated display data includes transmitting all the display data of the user interface (even if some of the display data has been changed and / or updated). In some embodiments, upon receiving updated display data from the second computer system 604 at 632, the second computer system 604 displays the updated display data on the second computer system 604. In some embodiments, the second computer system 604 stops displaying the display data received at 620 and instead displays the display data received at 632.

[0106] In some embodiments, transmitting updated display data in 632 includes transmitting updated audio data from the first computer system 602 via a second communication channel. In some embodiments, the updated audio data includes the same, subsequent, additional, and / or alternative audio signals as the audio data described above in 620.

[0107] In 634, the first computer system 602 transmits updated user interface element data via the first communication channel. For example, referring to Figure 4, transmitting updated user interface element data includes transmitting a new set of user interface elements (e.g., and associated data and / or mappings) from the second user interface when the calendar application is launched. In some embodiments, transmitting updated user interface element data includes transmitting only user interface element data that has been changed and / or updated from user interface element data previously transmitted in 622. In some embodiments, transmitting updated user interface element data includes transmitting all user interface element data of the user interface (even if, for example, only a portion of the user interface element data has been changed and / or updated). In some embodiments, the updated user interface element data is displayed by the first computer system 602. In some embodiments, the updated user interface element data is the result of an action performed in 630. In some embodiments, upon receiving updated user interface element data at 634 from the second computer system 604, the second computer system 604 displays the updated user interface element data on the second computer system 604. In some embodiments, the second computer system 604 stops displaying the user interface element data received at 622 and instead displays the updated user interface element data received at 632.

[0108] As indicated by 642, the first and second communication channels are maintained in each of 618, 620, 622, 624, 626, 628, 630, 632, and 634. For example, as shown in Figure 6, both the first and second communication channels are used to exchange data between the first and second computer systems. In some embodiments, some data is exchanged via the first communication channel, and different data is exchanged via the second communication channel. In some embodiments, it should be noted that the first and / or second communication channels are not maintained in each of 618, 620, 622, 624, 626, 628, 630, 632, and 634, such as when the communication channels are not in use.

[0109] As described herein, using different communication channels for different data provides a responsive interaction to the user. For example, by transmitting data through the first and second communication channels in Figure 6, it becomes possible to transmit data through both channels at least partially simultaneously and / or alternately. This allows data to be transferred more efficiently between devices, since neither channel transmits all the data. For example, the first communication channel transmits user interface element data, and the second communication channel transmits display data. In some embodiments, the communication channels are optimized for exchanging data having specific characteristics (e.g., display data). For example, the second communication channel can use a connection and / or protocol with extremely low latency, which is more suitable for exchanging real-time display data (which may require a large amount of bandwidth). As a result, a more responsive user interface experience is obtained for the user. For example, received user input 624 can be quickly transmitted to the first computer system 602, and as a result, updated display data and updated user interface element data can be quickly transferred to the second computer system 604.

[0110] Figure 7 shows an exemplary communication diagram between two computer systems (e.g., portable multifunction devices such as a smartwatch and a smartphone) according to several embodiments. As shown in Figure 7, Figure 700 includes a first computer system 702 communicating with a second computer system 704. The first computer system 702 and / or the second computer system 704 may include one or more of the features described with respect to the computing system 100 in Figure 1, the device 200 in Figure 2, the smartwatch 310 in Figure 3, the smartwatch 410 in Figure 4, the first computer system 602 in Figure 6, the smartphone 340 in Figure 3, the smartphone 440 in Figure 4, and / or the second computer system 604 in Figure 6. For example, the first computer system 702 may be a wearable device (e.g., a wristwatch), and the second computer system 704 may be a smartphone, both associated with the same user account (e.g., logged into the same user account and / or trusted by it).

[0111] In 706, the first computer system 702 transmits a first user interface element to the second computer system 704. In some embodiments, transmitting a first user interface element includes transmitting input control as shown in Figures 4, 5, and 6. For example, transmitting a first user interface element may include transmitting data representing the user interface element 414 in Figure 4 and / or data representing the physical button 404. In some embodiments, the first computer system 702 receives a request to transmit a user interface element in 706 and transmits a first user data element accordingly. In some embodiments, the first computer system 702 receives and / or detects a request from the second computer system 704. For example, the first computer system 702 receives a request via a connection request from the second computer system 604 (similar to the request to connect the first computer system 602 and the second computer system 704 in 608 and / or 610 in Figure 6).

[0112] In some embodiments, upon receiving the first user interface element at 706, the second computer system 704 displays the first user interface element.

[0113] In 708, the first computer system 702 transmits a first mapping of actions. In some embodiments, transmitting a first mapping of actions includes transmitting a mapping as shown in Table 500 of Figure 5. For example, transmitting a first mapping of actions may include transmitting the mapping of UI element 2 in row 520 and / or UI element 7 in row 570 of Figure 5. In some embodiments, the first computer system 702 receives a request in 708 to transmit a first mapping of actions and transmits the first mapping of actions accordingly. In some embodiments, the first computer system 702 receives a request in 706 to transmit user interface elements (e.g., one or more requests) and / or receives a first mapping of actions in 708 and transmits the first user interface elements and / or the first mapping of actions accordingly. In some embodiments, the first computer system 702 receives and / or detects a request from the second computer system 704. For example, the first computer system 702 receives a request via a connection request from the second computer system 704 (e.g., a connection request between the first computer system 602 and the second computer system 604 in 608 and / or 610). In some embodiments, after receiving a first mapping of actions in 708, the second computer system displays a first user interface element that it can interact with (e.g., via the received input).

[0114] At 710, the second computer system 704 receives input. In some embodiments, the input is a touch input (e.g., tap, gesture, and / or select). In some embodiments, the input received at 710 is an input received by the second computer system 704. For example, the received input is a touch input (e.g., tap, gesture, and / or select) received on the touch-sensitive surface of the second computer system 704. In some embodiments, the input received at 710 is an input received by an auxiliary computer system. In some embodiments, the second computer system 704 receives input at a location associated with a user interface element (e.g., displayed on the second computer system 704) (identified in data received from the first computer system 702 at 706). In some embodiments, the input received at 710 corresponds to an action available from the exemplary mapping in Figure 5. For example, the input received at 710 could be an input at UI element 2 in row 520 of Figure 5. In such examples, the available actions are tap input and / or tap-and-hold input to UI element 2. In some embodiments, the received input is a tap input.

[0115] In some embodiments, the input received by the second computer system 704 is of a different type from the available actions. For example, the received input may be a voice input that provides a tap input on the UI element 2 (e.g., representing a request to register). In such an example, the second computer system 604 determines (e.g., in 712 described later) that the voice input corresponds to a tap input on the UI element 2, and as a result determines that the voice input corresponds to the available actions of the tap input on the UI element 2. In some embodiments, in response to receiving the input in 710, the second computer system 704 determines the action identifier in 712.

[0116] In 712, the second computer system 704 determines an action identifier. In some embodiments, the second computer system 704 determines an action identifier by associating the received input with an action identifier received from the mapping in 708. In some embodiments, the action identifier corresponds to the input received in 710 (e.g., a tap). In some embodiments, the action identifier is an action performed on the first computer system 702 (e.g., launching an application). For example, if the input received in 710 corresponds to a tap input in UI element 2 in row 520 of Figure 5, the action identifier is the input and / or action. As another example, if the action identifier is an identifier of an input, the action identifier identifies the input that provides a tap in UI element 2. With respect to Figure 4, if UI element 2 is user interface element 444 on smartphone 440 and user interface element 414 on smartwatch 410, the action identifier provides a tap in user interface element 414 on smartwatch 410. As a third example, if the action identifier is an identifier of an action, the action identifier identifies the action that corresponds to the tap input 2. As a result of the input received by user interface element 444 corresponding to a tap on UI element 2, the action is the action corresponding to a tap on UI element 2 in Figure 5. With respect to Figure 4, if UI element 2 is user interface element 444 on smartphone 440 and user interface element 414 on smartwatch 410, the action identifier of the tap input is the action of launching the weather application. In some embodiments, depending on the determination of the action identifier in 712, the second computer system 704 transmits the action identifier in 714 to the first computer system 702.

[0117] In 714, the second computer system 704 transmits an action identifier to the first computer system 702. In some embodiments, transmitting an action identifier includes the second computer system 704 transmitting a user interface element corresponding to the action identifier. For example, in 712, the second computer system 704 determines that the input received in 710 corresponds to a press of UI element 5 in table 500. In this example, the second computer system 704 transmits the action identifier of the press and at least the indication of UI element 5 to the first computer system 702. In some embodiments, in response to receiving the action identifier in 714, the first computer system 702 performs the action in 716.

[0118] In 716, the first computer system 702 performs an action (for example, based on and / or in response to receiving an action identifier in 714). In some embodiments, the action corresponds to an action identifier communicated by the second computer system 704. For example, if the action identifier is an input identifier, the first computer system 702 performs an action corresponding to the action identifier of a tap on the user interface element 414 on the smartwatch 410. In another example, if the action identifier is an operation identifier, the first computer system 702 performs an action corresponding to the action identifier of an operation that launches a weather application.

[0119] In some embodiments, performing an action updates the display of the first computer system 702. For example, by launching and / or selecting the weather application associated with the user interface element 414 in Figure 4, the first user interface is changed and / or updated to a third user interface representing the weather application. In some embodiments, in response to displaying the updated and / or changed user interface element in the first computer system 702, the first computer system 702 transmits a second user interface element in 718 and / or transmits a second mapping 720 of the action.

[0120] In 718, the first computer system 702 transmits the updated second user interface elements to the second computer system 704. For example, referring to Figure 4, transmitting the updated user interface elements in 718 includes transmitting the updated user interface element data for the second user interface when the weather application is launched. In some embodiments, transmitting the updated user interface element data includes transmitting only the user interface element data that has been changed and / or updated from the user interface element data previously transmitted from 706. In some embodiments, transmitting the updated user interface element data includes transmitting all of the user interface element data (even if only a portion of the user interface element data has been changed and / or updated). In some embodiments, the updated user interface element data is displayed on the first computer system 702.

[0121] In 720, the first computer system 702 transmits a second mapping of actions. In some embodiments, transmitting a second mapping of actions includes transmitting all mappings of actions. In some embodiments, transmitting a second mapping of actions includes transmitting updated mapping data for a second user interface compared to the first mapping of data transmitted in 708. For example, launching a weather application generates a new UI element (e.g., by the first computer system 702) with weather information and / or graphics of weather conditions that were not displayed before launch. The new UI element may include new actions available in the new location and / or the same actions in the new location. In this example, the first computer system 702 transmits a second mapping in 720 that includes updated actions, updated UI elements, and / or updated locations in the second mapping of actions in 720.

[0122] Figure 8 is a flowchart illustrating a method for communication between computer systems (e.g., Method 800) according to several embodiments. Some operations of Method 800 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0123] As described below, Method 800 provides a technique for communication between computer systems. Method 800 can provide a new and / or more effective method for communication between computer systems, thereby creating a more efficient interface. In the case of battery-powered computing devices, enabling faster and more efficient communication between computer systems saves power and extends the time between battery charges.

[0124] In some embodiments, Method 800 is performed on a first computer system (e.g., a phone, computer, tablet, and / or wearable) (e.g., smartwatch 310, smartwatch 410, first computer system 602, first computer system 702, computing system 100, and / or device 200). In some embodiments, the first computer system is a phone, wristwatch, tablet, fitness tracking device, wearable device, television, multimedia device, accessory, speaker, head-mounted display (HMD), and / or personal computing device. In some embodiments, the first computer system communicates with input / output devices such as one or more cameras, speakers, microphones, sensors, and / or display components (e.g., display 408).

[0125] In 802, while the first computer system is connected to a second computer system (e.g., a telephone, computer, tablet, and / or wearable device) (e.g., a smartphone 340, a smartphone 440, a second computer system 604, a second computer system 704, a computing system 100, and / or device 200) via a first communication channel (e.g., a channel for transmitting and / or receiving data) (e.g., the first communication channel 320, communication channel 430, and / or the first communication channel in Figure 6) (e.g., connected via the first communication channel 606), the first computer system receives a second communication channel (e.g., a channel for transmitting and / or receiving data) different from the first communication channel (e.g., the second communication channel in Figure 6) from the second computer system. The receiving device receives connection data (e.g., data that enables the receiving device to connect to the second computer system, such as address and / or credential data) (e.g., first connection data 608) on channel 330, communication channel 430, and / or a second communication channel (e.g., depending on different networks and / or communication protocols, depending on different communication technologies (e.g., Wi-Fi vs. Bluetooth, and / or Bluetooth vs. Near Field Communication (NFC)), using different network resources (e.g., frequency, radio channel, antenna, hardware, software, scheduling, modulation, and / or demodulation techniques), and / or using different types of connections (e.g., ad-hoc Wi-Fi network connections (e.g., peer-to-peer), and / or infrastructure network connections)). In some embodiments, the second computer system is a telephone, a wristwatch, a tablet, a fitness tracking device, a wearable device, a television, a multimedia device, an accessory, a speaker, a head-mounted display (HMD), and / or a personal computing device. In some embodiments, the second computer system communicates with one or more input / output devices such as cameras, speakers, microphones, sensors, and / or display components.In some embodiments, a communication channel refers to one or more of the following: a physical connection, a logical connection, and / or communication using a specific communication protocol.

[0126] In 804, while the first computer system is connected to the second computer system via the first communication channel, the first computer system connects to the second computer system via the second communication channel (for example, by performing one or more actions to communicate) (for example, by connecting via the second communication channel) using connection data (e.g., submitting credentials and / or connecting to a network address (e.g., responding in 614)).

[0127] In 806, while the first computer system is connected to the second computer system via a first communication channel, the first computer system transmits media output data (e.g., screen mirroring data (e.g., display data in 620), UI element data (e.g., physical and / or virtual controls) (e.g., multiple user interface elements 412, 414, 416, and 418, and / or physical buttons 432 and 434, and / or rotatable input mechanism 436) (e.g., user interface element data in 622, first user interface element in 706), display data, and / or audio data) to the second computer system via the second communication channel. In some embodiments, the first communication channel is established by pairing between the first computer system and the second computer system (e.g., the first computer system and the second computer system form a Bluetooth pair and connect via Bluetooth based on the existing pairing).

[0128] In some embodiments, receiving connection data for a second communication channel includes receiving connection data for the second communication channel (e.g., first connection data in 608) via the first communication channel. In some embodiments, the first computer system receives connection data for the second communication channel via a third communication channel different from the first and second communication channels.

[0129] In some embodiments, using connection data to connect to a second computer system includes the first computer system using connection data to connect to the second computer system in response to receiving connection data via the first communication channel (for example, the connection data includes requests, commands, and / or instructions to the first computer system to connect using the connection data). In some embodiments, the first computer system uses the connection data to connect to the second computer system automatically (for example, without the intervention of user input to associate content with another computer system).

[0130] In some embodiments, connection data includes one or more selected from the group of network information (e.g., network name, network address, port information, and / or network channel, such as information for identifying a network and / or establishing a connection with another computer system) and authentication information (e.g., credentials, password, passcode, certificate, signature, and / or information usable to authenticate a computer system).

[0131] In some embodiments, after transmitting media output data via a second communication channel, the first computer system maintains (e.g., continues) the display of the media output data. In some embodiments, the media output data is displayed simultaneously with the first computer system transmitting the media output data to the second computer system via the second communication channel.

[0132] In some embodiments, the first computer system detects updated media output data (e.g., updated screen mirroring data, and / or updated UI element data (e.g., representing physical and / or virtual controls)). In some embodiments, the updated media output data is an update to the media output data (e.g., the next frame displayed via a display generation component, and / or a change to the media output data). In some embodiments, the updated media output data is generated by the first computer system in response to an input (e.g., one or more inputs representing gestures, taps, and / or selections) corresponding to a selection of user interface elements (e.g., physical and / or virtual controls) contained in the media output data. In some embodiments, upon detection of the updated media output data, the first computer system transmits (e.g., updated display data in 632) the updated media output data to the second computer system via a second communication channel (e.g., transmission and / or communication). In some embodiments, after transmitting the updated media output data, the second computer system displays the updated media output data (e.g., via a display component communicating with the second computer system).

[0133] In some embodiments, in conjunction with (e.g., after, during, accordingly, and / or before) transmitting media output data over a second communication channel, the first computer system transmits (e.g., transmits and / or communicates) over the first communication channel the identification of one or more user interface elements (e.g., physical and / or virtual controls) (e.g., a plurality of user interface elements 412, 414, 416, and 418, physical buttons 432 and 434, and / or rotatable input mechanism 436) contained in the media output data (e.g., user interface element data in 622). In some embodiments, the identification of one or more user interface elements includes a mapping (e.g., table 500) that includes one or more action identifiers mapped to one or more user interface elements (e.g., available actions in Figure 5). In some embodiments, the identification of one or more user interface elements includes an action mapped to each of the one or more user interface elements. In some embodiments, the identification of one or more user interface elements includes a plurality of actions mapped to the user interface elements. In some embodiments, the identification of one or more user interface elements includes a plurality of actions mapped to each of the user interface elements. In some embodiments, the mapping includes data describing the functional capabilities (e.g., inputs and / or actions) and / or location information (e.g., position, shape, and / or size) associated with one or more user interface elements. In some embodiments, one or more action identifiers include one or more identifiers of received inputs or actions associated with the user interface elements on a first computer system.

[0134] In some embodiments, in conjunction with (e.g., after, during, accordingly, and / or before) the first computer system transmits media output data via a second communication channel, the first computer system transmits (e.g., communicates and / or starts) via the first communication channel one or more action identifiers, including action identifiers (e.g., functional capabilities (e.g., actions) associated with one or more user interface elements) (e.g., action identifiers in 628 and / or action identifiers in 714), where one or more action identifiers correspond to one or more user interface elements. In some embodiments, the first computer system receives action identifiers via the first communication channel. In some embodiments, upon receiving action identifiers, the first computer system performs (e.g., executes, starts, and / or causes to start) an action corresponding to the action identifier (e.g., a function associated with at least one of the one or more user interface elements). In some embodiments, the action includes the selection of one or more user interface elements (e.g., tap input, swipe input, rotate input, and / or press input). In some embodiments, a first computer system transmits updated media output data (e.g., screen mirroring data, UI element data (e.g., physical and / or virtual controls)) different from the media output data via a second communication channel while maintaining a first communication channel, the updated media output data including one or more updated user interface elements and one or more updated action identifiers (e.g., one or more action identifiers) different from one or more action identifiers. In some embodiments, the updated media output data corresponds to performing an action (e.g., generated accordingly) corresponding to the action identifier. In some embodiments, the updated media output data is an update to the media output data (e.g., the next frame displayed via a display generation component, a change to the media output data).In some embodiments, one or more updated Action identifiers correspond to one or more updated user interface elements.

[0135] In some embodiments, the first communication channel uses a first communication protocol (e.g., Bluetooth protocol and / or Wi-Fi protocol), and the second communication channel does not use the first communication protocol. In some embodiments, the communication protocol is a wireless network protocol (e.g., Wi-Fi protocol, Bluetooth protocol, mobile communication protocol (e.g., 3G, 4G, LTE, 5G, and / or other 3G Partnership Project (3GPP®) related protocols)). In some embodiments, the communication protocol is one or more of the application layer protocol, presentation layer protocol, session layer protocol, transport layer protocol, network layer protocol, data link layer protocol, and / or physical layer protocol. In some embodiments, the second communication channel uses a second communication channel different from the first communication channel (e.g., the first communication channel exchanges data using the Wi-Fi protocol, and the second communication channel exchanges data using the Bluetooth protocol).

[0136] In some embodiments, both the first and second communication channels use a second communication protocol (e.g., a communication protocol described elsewhere in this specification).

[0137] In some embodiments, the first communication channel is encoded with a first encryption scheme (e.g., an encryption algorithm (e.g., symmetric and / or asymmetric)), and the second communication channel is encoded with a second encryption scheme (e.g., an encryption algorithm (e.g., symmetric and / or asymmetric)). In some embodiments, the first encryption scheme is different from the second encryption scheme. In some embodiments, the first and second encryption schemes are the same encryption scheme (e.g., the same encryption algorithm) but include different keys and / or cryptography.

[0138] It should be noted that the details of the process described above in relation to Method 800 (e.g., Figure 8) are also applicable in a manner similar to other methods described herein. For example, Method 800 may optionally include any one or more of the characteristics of the various methods described below with respect to Methods 900, 1000, and 1100. For example, Method 800 may include sending a mapping to a second computer system that includes one or more action identifiers mapped to one or more UI elements (e.g., from 1002 of Method 1000). In this example, by sending a mapping that includes one or more action identifiers, the second computer system is able to send action identifiers corresponding to actions performed by the first computer system. For brevity, these details will not be repeated below.

[0139] Figure 9 is a flowchart illustrating a method for communication between computer systems (e.g., Method 900) using several examples. In Method 900, some operations are combined in an optional manner, the order of some operations can be changed in an optional manner, and some operations can be omitted in an optional manner.

[0140] As described below, Method 900 provides a technique for communication between computer systems. Method 900 can provide a new and / or more effective method for communication between computer systems, thereby creating a more efficient interface. In the case of battery-powered computing devices, enabling faster and more efficient communication between computer systems saves power and extends the time between battery charges.

[0141] In some embodiments, Method 900 is performed on a first computer system (e.g., a phone, computer, tablet, and / or wearable) (e.g., smartphone 340, smartphone 440, second computer system 604, second computer system 704, computing system 100, and / or device 200). In some embodiments, the first computer system is a phone, watch, tablet, fitness tracking device, wearable device, television, multimedia device, accessory, speaker, head-mounted display (HMD), and / or personal computing device. In some embodiments, the first computer system communicates with input / output devices such as one or more cameras, speakers, microphones, sensors, and / or display components (e.g., display 408).

[0142] In 902, the first computer system sends a first request (e.g., a command and / or instruction) to the second computer system (e.g., a telephone, computer, tablet, and / or wearable device) (e.g., smartwatch 310, smartwatch 410, first computer system 602, first computer system 702, computing system 100, and / or device 200) for the second computer system to join a communication channel (e.g., a channel for sending and / or receiving data) (e.g., communication channel 430 in Figure 6, first communication channel 320, second communication channel 330, and / or first communication channel and / or second communication channel), the first request including first connection data (e.g., data that enables the receiving device to connect to the second computer system, such as an address, network information, and / or credential data) (e.g., first connection data in 608). In some embodiments, the second computer system is a telephone, watch, tablet, fitness tracking device, wearable device, television, multimedia device, accessory, speaker, head-mounted display (HMD), and / or personal computing device. In some embodiments, the second computer system communicates with one or more input / output devices such as a camera, speaker, microphone, sensor, and / or display component.

[0143] In 904, after sending the first request (e.g., accordingly and / or in accordance thereafter), and in accordance with the determination that a valid response (e.g., confirmation that the second device has joined the communication channel and / or confirmation that the first request has been received) (e.g., the response in 614) is not received from the second computer system within a threshold period (e.g., a predetermined amount of time such as one minute, 30 seconds, or 10 minutes) (e.g., the credentials are not up-to-date in 636), the first computer system sends to the second computer system a second request (e.g., a second command and / or instruction) (e.g., second connection data in 610) for the second computer system to join the communication channel, the second request including second connection data different from the first connection data (e.g., data that enables the receiving device to connect to the second computer system, such as address and / or credential data). In some embodiments, the threshold period is the period from when the first request was sent (for example, the threshold period is measured from the time the corresponding request was sent). In some embodiments, the first computer system sends a second request to the second computer system, and if it is determined that no valid response to the second request is received within a second threshold period (e.g., a predetermined time such as 1 minute, 30 seconds, or 10 minutes), the first computer system sends additional requests. In some embodiments, the first computer system continues to send additional requests until a valid response is received for each request, based on the determination that no valid response is received within each additional threshold period (e.g., a predetermined amount of time such as 1 minute, 30 seconds, or 10 minutes).

[0144] In 906, after sending a first request (e.g., in accordance with and / or in accordance therewith) and in accordance with the determination that a valid response has been received from the second computer system within a threshold period, the first computer system connects with the second computer system via a communication channel (e.g., initiates a screen mirroring session via the communication channel) (e.g., connects via the second communication channel in 618). In some embodiments, the threshold period is a dynamic period (e.g., the period lengthens after each request (e.g., 30 seconds or 1 minute) without receiving a valid response from the second computer system). In some embodiments, the communication channel is a peer-to-peer connection between the first computer system and the second computer system (e.g., a peer-to-peer Wi-Fi network and / or a peer-to-peer Bluetooth connection). In some embodiments, the communication channel is a connection via a third computer system (e.g., an intermediate computer system, a server, a network router, a telephone, a computer, a tablet, and / or a wearable device).

[0145] In some embodiments, the first computer system configures (e.g., establishes and / or starts) a communication channel to generate first connection data before the second computer system sends a first request to join the communication channel. In some embodiments, the first computer system configures the communication channel in response to the second computer system detecting a command (e.g., input) to join the communication channel. In some embodiments, configuring a communication channel includes one or more of the following: creating a network identifier (e.g., SSID), creating an address (e.g., network address), establishing a software access point, and / or starting a network communication channel. In some embodiments, the connection data includes one or more of the following: credentials (e.g., network credentials, password, passcode, and / or certificate) and / or information for connecting to the network.

[0146] In some embodiments, the first request includes a request to transmit (e.g., transmit and / or communicate) media output data (e.g., screen mirroring data, display data, audio data, and / or UI element data (e.g., physical and / or virtual control)) from a second computer system to a first computer system via a communication channel. In some embodiments, the first request is a request to establish a communication channel using first connection data.

[0147] In some embodiments, the first computer system connects to the second computer system via a communication channel and, after doing so (for example, accordingly and / or in accordance therewith), receives media output data (e.g., screen mirroring data, display and / or audio data, and / or UI element data (e.g., physical and / or virtual control)) from the second computer system via the communication channel.

[0148] In some embodiments, the determination that a valid response has been received from the second computer system within a threshold period includes the determination that the valid response contains valid credentials (e.g., the most recent credentials corresponding to the most recent request (network password and / or network identifier) ​​(e.g., checking that the credentials are current in 616)). In some embodiments, valid credentials are time-limited (e.g., expire after a predetermined amount of time). In some embodiments, valid credentials are the most recently issued credentials (e.g., if subsequent credentials are issued in succession, the earlier credentials become invalid in terms of time).

[0149] In some embodiments, the first connection data includes first credentials (e.g., a password, passcode, and / or a certificate). In some embodiments, the second connection data includes second credentials (e.g., a password, passcode, and / or a certificate) that are different from the first credentials. In some embodiments, after sending the first request, the first credentials are valid (e.g., a valid response includes receiving the first credentials within a threshold period). In some embodiments, after sending the second request, the second credentials are valid (e.g., a valid response includes receiving the second credentials within a threshold period (e.g., a threshold period, and / or a second threshold period different from the threshold period)), and the first credentials are not valid.

[0150] In some embodiments, the communication channel is a second communication channel (e.g., a channel for transmitting and / or receiving data). In some embodiments, while connected to the second computer system via the second communication channel (e.g., a channel for transmitting and / or receiving data) (e.g., after the connection), the first computer system maintains a connection with the second computer system via a different first communication channel (e.g., a communication channel as described elsewhere in this specification) (e.g., not terminating the channel, keeping the channel open, and / or continuing communication over the channel). In some embodiments, before transmitting the first request, the first computer system connects with the second computer system via the first communication channel. In some embodiments, after transmitting the first request, the first computer system connects with the second computer system via the first communication channel.

[0151] In some embodiments, while connected to a second computer system via a first and second communication channel, the first computer system receives media output data (e.g., screen mirroring data (e.g., display data in 620), display data, audio data, and / or UI element data (e.g., physical and / or virtual controls) (e.g., a plurality of user interface elements 412, 414, 416, and 418, and / or physical buttons 432 and 434, and / or rotatable input mechanism 436) (e.g., user interface element data in 622, first user interface element in 706)) from the second computer system via the first communication channel. In some embodiments, the first computer system receives identification of one or more user interface elements (e.g., physical and / or virtual controls) contained in the media output data (e.g., identification of one or more user interface elements as described elsewhere in this specification) from the second computer system via the first communication channel. In some embodiments, the identification of one or more user interface elements includes a mapping (e.g., an exemplary mapping in Table 500) that includes one or more action identifiers mapped to one or more user interface elements. In some embodiments, the identification of one or more user interface elements includes an action mapped to each of the one or more user interface elements. In some embodiments, the identification of one or more user interface elements includes multiple actions mapped to the user interface elements. In some embodiments, the identification of one or more user interface elements includes multiple actions mapped to each of the multiple user interface elements. In some embodiments, the mapping includes data describing the functional capabilities (e.g., inputs and / or actions) and / or location information (e.g., position, shape, and / or size) associated with one or more user interface elements.In some embodiments, one or more action identifiers include one or more identifiers of received inputs or actions associated with a user interface element on a first computer system.

[0152] In some embodiments, the first connection data includes a network identifier (e.g., network name and / or information for identifying the network) and first network credentials (e.g., password, passcode, and / or certificate). In some embodiments, the second connection data includes a network identifier and second network credentials (e.g., password, passcode, and / or certificate).

[0153] In some embodiments, the communication channel is a second communication channel (e.g., a channel for transmitting and / or receiving data) (e.g., a communication channel as described elsewhere in this specification). In some embodiments, transmitting a first request involves transmitting the first request to a second computer system via a first communication channel (e.g., a channel for transmitting and / or receiving data) (e.g., a communication channel as described elsewhere in this specification) that is different from the second communication channel. In some embodiments, transmitting a first request involves transmitting the first request to a second computer system via a third communication channel (e.g., a channel for transmitting and / or receiving data) (e.g., a communication channel as described elsewhere in this specification) that is different from the first and second communication channels.

[0154] In some embodiments, the second computer system is a wearable device (e.g., a wristwatch, a ring, a fitness tracking device, and / or an HMD).

[0155] In some embodiments, the threshold period is a first threshold period. In some embodiments, according to the determination that a valid response to a second request has been received from the second computer system within a second threshold period (e.g., a predetermined amount of time such as 1 minute, 30 seconds, or 10 minutes), the first computer system connects with the second computer system via a communication channel (e.g., accordingly and / or thereafter), and the first threshold period is different from the second threshold period. In some embodiments, the first threshold period occurs over a different time span than the second threshold period (e.g., they do not overlap in time and / or start and / or end at different times). In some embodiments, the first threshold period is a different length of time than the second threshold period (e.g., the first period is 30 seconds and the second period is 15 seconds, and / or the first period is 30 minutes and the second period is 1 hour). In some embodiments, the threshold period associated with a later transmitted request is shorter than the previous (e.g., immediately preceding) threshold period associated with the previous request (e.g., a successively shorter period). In some embodiments, each threshold period is the same. In some embodiments, the threshold period associated with a later transmitted request is longer (e.g., continuously longer) than the previous (e.g., immediately preceding) threshold period associated with the previous request. In some embodiments, the first computer system continues to transmit requests until a valid response is received. In some embodiments, the first computer system stops transmitting requests after certain criteria are met (e.g., a number of failed requests has been reached, a total time has elapsed without a valid response, and / or user input representing a request to cancel the communication channel setup).

[0156] It should be noted that the details of the process described above in relation to Method 900 (for example, Figure 9) are also applicable in a manner similar to other methods described herein. For example, Method 900 optionally includes one or more characteristics of the various methods described above with reference to Method 900. For example, Method 900 includes transmitting media output data from 804 of Method 800 to a second computer system via a second communication channel. In this example, by transmitting the media output data, the first computer system can display the user interface of the second computer system using the connected communication channel of Method 900. For brevity, these details will not be repeated below.

[0157] Figure 10 is a flowchart illustrating a method for communication between devices (e.g., Method 1000) according to several embodiments. Some operations in Method 1000 are arbitrarily combined, the order of some operations is arbitrarily changed, and some operations are arbitrarily omitted.

[0158] As will be described later, Method 1000 provides an intuitive method for communication between computer systems. Method 1000 can provide a new and / or more effective method for communication between computer systems, thereby creating a more efficient interface. In the case of battery-powered computing devices, enabling faster and more efficient communication between computer systems saves power and extends the time between battery charges.

[0159] In some embodiments, Method 1000 is performed on a first computer system (e.g., a phone, computer, tablet, and / or wearable) (e.g., smartwatch 310, smartwatch 410, first computer system 602, first computer system 702, computing system 100, and / or device 200). In some embodiments, the first computer system is a phone, wristwatch, tablet, fitness tracking device, wearable device, television, multimedia device, accessory, speaker, head-mounted display (HMD), and / or personal computing device. In some embodiments, the first computer system communicates with input / output devices such as one or more cameras, speakers, microphones, sensors, and / or display components (e.g., display 408).

[0160] In 1002, the first computer system transmits (e.g., transmits and / or communicates) to the second computer system (e.g., a telephone, computer, tablet, and / or wearable) one or more user interface (UI) elements of the first computer system (e.g., physical and / or virtual elements) (e.g., associated with and / or included in the UI of the first computer system and / or displayed on the display of the first computer system) and a mapping (e.g., data describing the functional capabilities (e.g., actions) and / or location information (e.g., position, shape, size, etc.) associated with the UI element) (e.g., data describing the functional capabilities (e.g., actions) and / or location information (e.g., position, shape, size, etc.) associated with the UI element). In some embodiments, the second computer system is a telephone, wristwatch, tablet, fitness tracking device, wearable device, television, multimedia device, accessory, speaker, head-mounted display (HMD), and / or personal computing device. In some embodiments, the second computer system communicates with input / output devices such as one or more cameras, speakers, microphones, sensors, and / or display components. In some embodiments, transmitting one or more UI elements includes transmitting a representation and / or identifier of each UI element. In some embodiments, a UI element is a representation of an element associated with a UI. In some embodiments, a UI element of the first computer system is a representation of an element associated with the UI of the first computer system. In some embodiments, a UI element is displayed as part of a displayed UI (for example, a UI element is a virtual control displayed within a user interface on the first computer system). In some embodiments, a UI element is an input control associated with (e.g., mapped to) a displayed UI (for example, a UI element is a physical control (e.g., a button, a rotatable input mechanism) that can be used to detect inputs associated with a displayed UI).In some embodiments, a mapping and / or one or more UI elements is an identification of one or more UI elements. In some embodiments, a mapping includes one or more action identifiers mapped to one or more user interface elements. In some embodiments, an identification of one or more user interface elements includes an action mapped to each of the one or more user interface elements. In some embodiments, an identification of one or more user interface elements includes a plurality of actions mapped to the user interface elements. In some embodiments, an identification of one or more user interface elements includes a plurality of actions mapped to each of the plurality of user interface elements. In some embodiments, a mapping includes data describing the functional capabilities (e.g., inputs and / or actions) and / or location information (e.g., position, shape, and / or size) associated with one or more user interface elements. In some embodiments, one or more action identifiers include one or more identifiers of received inputs or actions associated with the user interface elements on a first computer system.

[0161] In 1004, the first computer system receives from the second computer system one or more action identifiers (e.g., unique and / or descriptive identifiers of actions) (e.g., action identifiers as described elsewhere in this specification) (e.g., action identifiers in 714), and the action identifier is associated with one or more UI elements (e.g., receiving actions, indications of actions, and / or indications of specific UI elements to which the action corresponds (e.g., inputs representing taps on approval controls)).

[0162] In 1006, upon receiving an action identifier associated with a UI element, the first computer system performs an action associated with the action identifier (e.g., input and / or operation) (e.g., perform an action in 630, perform an action in 716). In some embodiments, the first computer system transmits one or more UI elements, each containing an identifier of a UI element displayed on the first computer system. In some embodiments, transmitting a mapping of one or more actions (e.g., a first mapping of actions in 708) includes an identifier of an action for each of the one or more UI elements of the first computer system. In some embodiments, performing an input includes causing the first computer system to register the input (e.g., to the UI) as if the input had been received directly by the first computer system. In some embodiments, performing an operation (e.g., associated with an input) includes performing an operation configured to be performed in response to an input (e.g., the result of the input).

[0163] In some embodiments, a first computer system communicates with display generation components (e.g., displays, touch-sensitive surfaces, projectors, and / or components capable of outputting visual content), and one or more UI elements are displayed in the user interface via the display generation components.

[0164] In some embodiments, the mapping includes a set of one or more locations (e.g., placement, coordinates, and / or multiple coordinates) (e.g., locations in Figure 5) of one or more UI elements within the user interface relative to the user interface, corresponding to the boundaries of each frame of one or more UI elements in the first computer system (e.g., the size of the element) of the user interface.

[0165] In some embodiments, one or more UI elements include one or more indications of physical UI elements of the first computer system (e.g., buttons, rotatable input mechanisms, and / or physical toggles) and one or more indications of virtual UI elements of the first computer system (e.g., affordances, graphics, and / or virtual controls).

[0166] In some embodiments, the action is the type of input. In some embodiments, the type of input includes one or more inputs selected from a group of inputs representing selection (e.g., touch input, press-and-hold input, and / or tap input) and inputs representing rotation.

[0167] In some embodiments, the mapping includes multiple action identifiers mapped to UI elements of one or more UI elements (e.g., at least one UI element). In some embodiments, the multiple action identifiers are mapped to multiple UI elements of one or more UI elements. In some embodiments, a selection of multiple action identifiers (e.g., a set, and / or two or more) is mapped to one or more UI elements of a set of one or more UI elements.

[0168] In some embodiments, in response to performing an action associated with an action identifier, the first computer system sends an update to one or more UI elements (e.g., changes to one or more UI elements that appear in response to the action) to the second computer system. In some embodiments, the update to one or more UI elements includes one or more (e.g., one, several, and / or all) of the modified UI elements.

[0169] In some embodiments, upon performing an action associated with an action identifier, the first computer system sends an update to a mapping containing one or more action identifiers corresponding to an update to one or more UI elements (e.g., a change to an action identifier) ​​to the second computer system. In some embodiments, the update to the mapping includes one or more (e.g., one, several, and / or all) of the mappings for the changed one or more action identifiers.

[0170] It should be noted that the details of the process described above in relation to Method 1000 (e.g., Figure 10) are also applicable in a manner similar to other methods described herein. For example, Method 1000 optionally includes one or more characteristics of the various methods described above and below, e.g., Methods 800, 900 and / or 1100. For example, the technique of Method 1100 may be used to detect an input associated with a UI element among one or more UI elements, the input being a second type of input distinct from a first type of input, and transmitting an action identifier associated with the first type of input to a second computer system. This characteristic may enable Method 1000 to detect an input of a different type than a mapping. For brevity, these details will not be repeated below.

[0171] Figure 11 is a flowchart illustrating a method for communication between devices (e.g., Method 1100) according to several embodiments. Some operations in Method 1100 are arbitrarily combined, the order of some operations is arbitrarily changed, and some operations are arbitrarily omitted.

[0172] As will be described later, Method 1100 provides an intuitive method for communication between computer systems. Method 1100 can provide a new and / or more effective method for communication between computer systems, thereby creating a more efficient interface. In the case of battery-powered computing devices, enabling faster and more efficient communication between computer systems saves power and extends the time between battery charges.

[0173] In some embodiments, Method 1100 is performed on a first computer system (e.g., a phone, computer, tablet, and / or wearable) (e.g., smartphone 340, smartphone 440, second computer system 604, second computer system 704, computing system 100, and / or device 200). In some embodiments, the first computer system is a phone, watch, tablet, fitness tracking device, wearable device, television, multimedia device, accessory, speaker, head-mounted display (HMD), and / or personal computing device. In some embodiments, the first computer system communicates with one or more input / output devices such as cameras, speakers, microphones, sensors, and / or display components.

[0174] In 1102, the first computer system receives from the second computer system (e.g., a telephone, computer, tablet, and / or wearable) (e.g., smartwatch 310, smartwatch 410, first computer system 602, first computer system 702, computing system 100, and / or device 200) one or more user interface elements of the second computer system and the second computer system (smartphone 340, smartphone 440, second computer system 604, second computer system 704, computing system 100, and / or The device 200) receives a mapping (e.g., data describing the functional capabilities (e.g., actions) and / or location information (e.g., position, shape, size, etc.) associated with a UI element) (e.g., a mapping as shown in Figure 5), where the action identifier is associated with a first type of input (e.g., a button press, rotation of a rotatable input mechanism, and / or touch) in the second computer system (e.g., a first mapping of actions in 708). In some embodiments, the second computer system is a telephone, a wristwatch, a tablet, a fitness tracking device, a wearable device, a television, a multimedia device, an accessory, a speaker, a head-mounted display (HMD), and / or a personal computing device. In some embodiments, the second computer system communicates with input / output devices such as one or more cameras, speakers, microphones, sensors, and / or display components. In some embodiments, the mapping of one or more action identifiers to one or more UI elements includes one or more identifiers of actions (e.g., behaviors) mapped to each of the one or more UI elements, multiple actions mapped to a user interface element, and multiple actions mapped to each of the multiple user interface elements.In some embodiments, the mapping includes one or more actions for each UI element (e.g., of the user interface of a second computer system). In some embodiments, the mapping includes one or more actions for fewer user interface elements than all user interface elements (e.g., of the user interface of a second computer system).

[0175] In 1104, the first computer system detects an input associated with one or more UI elements (e.g., directed towards it, located in its location, associated with its location, mentioned therein, identified therein, and / or in its direction) where the input is a second type of input (e.g., voice input, or hand gesture input (e.g., captured by a camera)) distinct from a first type of input (e.g., tap input, or tap-and-hold input) (e.g., receiving the input in 624 and 710). In some embodiments, the type of input is at least partially based on a mechanism used to detect the input (e.g., a touch-sensing surface, a physical button, a microphone, and / or a light-sensing device (e.g., a camera)). In some embodiments, the input type is at least partially based on the characteristics of the input (e.g., the length of time the touch or press is detected (e.g., a distinction can be made between tap input and tap-and-hold input), the amount of movement after the touch event (e.g., a distinction can be made between tap and swipe), the magnitude of the movement, the location of the input, the force of the input, the speed of the input, and / or the location of the start of the input (e.g., an edge swipe input can be distinguished from a non-edge swipe input)). In some embodiments, the input type is at least partially based on the characteristics of the input and at least partially based on the mechanism used to detect the input (e.g., a tap input on a touch-sensitive surface, a tap-and-hold input on a touch-sensitive surface, a press input on a physical button, a press-and-hold input on a physical button, and an audio input to a microphone are each considered different types of input).

[0176] In 1106, the first computer system uses mapping to determine that an input of a second type corresponds to an input of a first type in the second computer system (e.g., representing a request to be performed) (e.g., using mapping to translate an input of a second type into an identifier of an action of a first type associated with a UI element) (e.g., determining the action identifier in 626 and 712) (e.g., accordingly and / or thereafter), and in accordance with this determination, the first computer system transmits to the second computer system the action identifier associated with the input of the first type (e.g., the action identifier in 628 and 714). In some embodiments, using mapping to determine that an input of a (e.g., received) type corresponds to an input of a different type includes determining the intent that the input of that type represents an input of a second type (e.g., a touch on a virtual representation of a physical button may represent the intent to register an input for a physical button, and / or a voice input tapping a virtual control may represent the intent to register a tap input for a virtual control).

[0177] In some embodiments, one or more UI elements include one or more indications of physical UI elements of a second computer system (e.g., physical controls such as buttons, rotatable input mechanisms, and / or physical toggles) and one or more indications of virtual UI elements of a second computer system (e.g., virtual controls such as affordances, graphics, and / or virtual buttons).

[0178] In some embodiments, a second computer system communicates with display generation components (e.g., displays, touch-sensitive surfaces, projectors, and / or components capable of outputting visual content). In some embodiments, UI elements are displayed in the user interface via the display generation components.

[0179] In some embodiments, the mapping includes a set of one or more locations of UI elements within the user interface relative to the user interface (e.g., placement, coordinates, and / or multiple coordinates corresponding to the frame boundaries (e.g., the size of the element) of one or more UI elements in a second computer system) (e.g., one or more locations as shown in Table 500 in Figure 5).

[0180] In some embodiments, receiving one or more UI elements from the second computer system is done via a second communication channel between the first and second computer systems (e.g., a channel for sending and / or receiving data) (communication channel 430, second communication channel 330, and / or second communication channel in Figure 6). In some embodiments, receiving mappings from the second computer system is done via a first communication channel between the first and second computer systems (e.g., a channel for sending and / or receiving data) (communication channel 430, first communication channel 320, and / or first communication channel in Figure 6). In some embodiments, the first communication channel is different from the second communication channel. In some embodiments, before sending the first request, the first computer system connects with the second computer system via the first communication channel.

[0181] In some embodiments, displaying UI elements via a display generation component is performed simultaneously with a second computer system displaying UI elements via different display generation components.

[0182] In some embodiments, a mapping is used so that, upon determining (for example, accordingly and / or thereafter) that an input of a second type does not correspond to a first type input in the second computer system, the first computer system stops sending the action identifier associated with the first type input to the second computer system.

[0183] In some embodiments, the first type of input is touch input (e.g., tap input, tap-and-hold input, press input, select, and / or slide). In some embodiments, the second type of input is audio input (e.g., voice input, and / or voice command). In some embodiments, one or more UI elements include indications of physical UI elements (e.g., buttons, rotatable input mechanisms, and / or physical toggles), and the first type of input is input to at least one of the indications of the physical UI elements.

[0184] In some embodiments, the first computer system communicates with a display generation component (e.g., a display, a touch-sensitive surface, a projector, and / or a component capable of outputting visual content). In some embodiments, the first computer system displays a UI element from one or more UI elements via the display generation component.

[0185] It should be noted that the details of the process described above in relation to Method 1100 (for example, Figure 11) are also applicable to methods similar to those described herein. For example, Method 1100 may optionally include one or more characteristics of the various methods described above with reference to Method 1100. For example, Method 1100 may include techniques of Method 900, such as connecting to a second computer system via a communication channel after sending a first request, in accordance with the determination that a response to the first request has been received from the second computer system within a threshold period. For brevity, these details will not be repeated below.

[0186] The above description is based on the purpose of explanation and is written with reference to specific examples. However, the above illustrative description is not intended to be exhaustive or to limit this disclosure to the exact form disclosed. Many modifications and variations are possible in light of the above teachings. These examples have been selected and written to best illustrate the principles of the art and their practical applications. Thereafter, those skilled in the art can best utilize the art and various examples with various modifications suitable for their specific intended use.

[0187] While the present disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. Such changes and modifications should be understood to fall within the scope of the present disclosure and examples as defined by the claims.

[0188] As described above, one aspect of this technology involves collecting and using available data from various sources to improve communication data between devices. This disclosure intends that in some cases, this collected data may include personal information data that can be used to uniquely identify or contact or locate a particular individual. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, home addresses, or any other identifying information.

[0189] This disclosure acknowledges that such use of personal data in this technology may be for the benefit of the user. For example, personal data can be exchanged between computer systems. Thus, the use of such personal data enables computer systems to connect and provide additions and / or extensions to their functionality. Furthermore, other uses of personal data that may benefit the user are also conceivable in this disclosure.

[0190] This disclosure further assumes that any entity responsible for collecting, analyzing, disclosing, transferring, storing, or otherwise using such personal data will comply with appropriately defined privacy policies and / or privacy practices. Specifically, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for keeping personal data confidential and secure. For example, personal information from users should be collected for the entity's lawful and legitimate use and should not be shared or sold for any other purpose. Furthermore, such collection should only be carried out after informing and obtaining the user's consent. In addition, such entities will take all necessary steps to protect and secure access to such personal data and to ensure that others who have access to that personal data comply with those privacy policies and procedures. Furthermore, such entities may undergo third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices.

[0191] Notwithstanding the foregoing, the Disclosure also envisions embodiments that allow a user to selectively prevent the use of or access to personal data. That is, the Disclosure envisions that hardware and / or software elements may be provided to prevent or prevent access to such personal data. For example, in the case of imaging, the technology may be configured to allow a user to choose to “opt in” or “opt out” of participating in the collection of personal data while registering for the service.

[0192] Therefore, while this disclosure extensively covers the use of personal data to implement one or more of the disclosed embodiments, it is also conceivable that these embodiments could be implemented without requiring access to such personal data. In other words, the various embodiments of the Technology would not be rendered inoperable by the absence of all or part of such personal data.

Claims

1. It is a method, In the first computer system, While the first computer system is connected to the second computer system via the first communication channel, Receiving connection data for a second communication channel different from the first communication channel from the second computer system, Using the aforementioned connection data, connect to the second computer system via the second communication channel, The media output data is transmitted to the second computer system via the second communication channel, Methods that include...

2. The method according to claim 1, wherein receiving the connection data for the second communication channel includes receiving the connection data for the second communication channel via the first communication channel.

3. Connecting to the second computer system using the aforementioned connection data is The method according to claim 2, comprising connecting to the second computer system using the connection data in response to receiving the connection data via the first communication channel.

4. The aforementioned connection data, The method according to any one of claims 1 to 3, comprising one or more selected from the group of network information and authentication information.

5. The first computer system communicates with the display generation component, and the method is The media output data is displayed via the aforementioned display generation component, The method according to any one of claims 1 to 4, further comprising transmitting the media output data via the second communication channel, followed by maintaining the display of the media output data.

6. The method described above detects updated media output data, The method according to any one of claims 1 to 5, further comprising transmitting the updated media output data to the second computer system via the second communication channel in response to detecting the updated media output data.

7. In addition to transmitting the media output data via the second communication channel, the identification of one or more user interface elements included in the media output data is transmitted via the first communication channel. The method according to any one of claims 1 to 6, further comprising:

8. In addition to transmitting the media output data via the second communication channel, the first communication channel also transmits one or more action identifiers, each including an action identifier, wherein the one or more action identifiers correspond to the one or more user interface elements. Receiving the action identifier via the first communication channel, Upon receiving the aforementioned action identifier, the action corresponding to the aforementioned action identifier is executed, While maintaining the first communication channel, the updated media output data is transmitted via the second communication channel, wherein the updated media output data includes one or more updated user interface elements and one or more updated action identifiers that are different from the one or more action identifiers. The method according to claim 7, further comprising:

9. The method according to any one of claims 1 to 8, wherein the first communication channel uses a first communication protocol, and the second communication channel does not use the first communication protocol.

10. The method according to any one of claims 1 to 9, wherein both the first communication channel and the second communication channel use the second communication protocol.

11. The method according to any one of claims 1 to 10, wherein the first communication channel is encoded with a first encryption scheme, the second communication channel is encoded with a second encryption scheme, and the first encryption scheme is different from the second encryption scheme.

12. A non-temporary computer-readable medium for storing one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs include instructions for performing the method according to any one of claims 1 to 11.

13. A first computer system comprising one or more processors, A memory that stores one or more programs configured to be executed by one or more processors, A first computer system comprising, wherein one or more programs include instructions for performing the method according to any one of claims 1 to 11.

14. A first computer system, A computer system comprising means for performing the method described in any one of claims 1 to 11.

15. A computer program product comprising one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs include instructions for performing the method according to any one of claims 1 to 11.

16. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs are While the first computer system is connected to the second computer system via the first communication channel, The second computer system receives connection data for a second communication channel different from the first communication channel. Using the aforementioned connection data, connect to the second computer system via the second communication channel. A non-temporary computer-readable storage medium including instructions for transmitting media output data to the second computer system via the second communication channel.

17. A first computer system comprising one or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are While the first computer system is connected to the second computer system via the first communication channel, The second computer system receives connection data for a second communication channel different from the first communication channel. Using the aforementioned connection data, connect to the second computer system via the second communication channel. A first computer system including instructions for transmitting media output data to the second computer system via the second communication channel.

18. A first computer system, While the first computer system is connected to the second computer system via the first communication channel, Means for receiving connection data for a second communication channel different from the first communication channel from the second computer system, Means for connecting to the second computer system via the second communication channel using the aforementioned connection data, Means for transmitting media output data to the second computer system via the second communication channel, A first computer system equipped with the following:

19. A computer program product comprising one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs include instructions, While the first computer system is connected to the second computer system via the first communication channel, The second computer system receives connection data for a second communication channel different from the first communication channel. Using the aforementioned connection data, connect to the second computer system via the second communication channel. A computer program product including instructions for transmitting media output data to the second computer system via the second communication channel.

20. It is a method, In the first computer system, To send to a second computer system a first request for the second computer system to join a communication channel, wherein the first request includes first connection data, After sending the first request, In accordance with the determination that no valid response is received from the second computer system within the threshold period, the second computer system is instructed to send a second request for the second computer system to join the communication channel, the second request comprising second connection data different from the first connection data. In accordance with the determination that a valid response has been received from the second computer system within the threshold period, a connection is made to the second computer system via the communication channel. Methods that include...

21. Configuring the communication channel to generate the first connection data before the second computer system transmits the first request to join the communication channel, The method according to claim 20, further comprising:

22. The method according to any one of claims 20 to 21, wherein the first request includes a request to transmit media output data from the second computer system to the first computer system via the communication channel.

23. After connecting to the second computer system via the communication channel, receiving media output data from the second computer system via the communication channel. The method according to claim 22, further comprising:

24. The method according to any one of claims 20 to 23, wherein the determination that the valid response is received from the second computer system within the threshold period includes the determination that the valid response includes a valid credential.

25. The first connection data includes the first credentials, The second connection data includes a second credential different from the first credential, and after the first request is sent, the first credential is valid. After sending the second request, if the second credentials are valid and the first credentials are not valid, The method according to claim 24.

26. The communication channel is a second communication channel, and the method is The method according to any one of claims 20 to 25, further comprising maintaining a connection with the second computer system via a first communication channel different from the second communication channel while connected to the second computer system via the second communication channel.

27. While connected to the second computer system via the first and second communication channels, Receiving media output data from the second computer system via the second communication channel, The second computer system receives, via the first communication channel, the identification of one or more user interface elements included in the media output data, The method according to claim 26, further comprising:

28. The method according to any one of claims 20 to 27, wherein the first connection data includes a network identifier and a first network credential, and the second connection data includes the network identifier and a second network credential.

29. The method according to any one of claims 20 to 28, wherein the communication channel is a second communication channel, and transmitting the first request includes transmitting the first request to the second computer system via a first communication channel different from the second communication channel.

30. The method according to any one of claims 20 to 29, wherein the second computer system is a wearable device.

31. The threshold period is a first threshold period, and the method is The method according to any one of claims 20 to 29, further comprising connecting to the second computer system via the communication channel in accordance with the determination that a valid response to the second request has been received from the second computer system within a second threshold period, wherein the first threshold period is different from the second threshold period.

32. A non-temporary computer-readable medium storing one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs include instructions for performing the method according to any one of claims 20 to 31.

33. A first computer system comprising one or more processors, A memory that stores one or more programs configured to be executed by one or more processors, A first computer system comprising, wherein one or more programs include instructions for performing the method according to any one of claims 20 to 31.

34. A first computer system, A first computer system comprising means for performing the method described in any one of claims 20 to 31.

35. A computer program product comprising one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs include instructions for performing the method according to any one of claims 20 to 31.

36. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs are The second computer system receives a first request for the second computer system to join a communication channel, wherein the first request includes first connection data. After sending the first request, In accordance with the determination that no valid response is received from the second computer system within the threshold period, the second computer system sends a second request to the second computer system for the second computer system to join the communication channel, wherein the second request includes second connection data different from the first connection data. A non-temporary computer-readable storage medium containing instructions for connecting to the second computer system via the communication channel, in accordance with the determination that a valid response has been received from the second computer system within the threshold period.

37. A first computer system comprising one or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are The second computer system receives a first request for the second computer system to join a communication channel, wherein the first request includes first connection data. After sending the first request, In accordance with the determination that no valid response is received from the second computer system within the threshold period, the second computer system sends a second request to the second computer system for the second computer system to join the communication channel, wherein the second request includes second connection data different from the first connection data. A first computer system, including instructions, that connects to the second computer system via the communication channel, in accordance with the determination that a valid response has been received from the second computer system within the threshold period.

38. A first computer system, A second computer system includes means for transmitting a first request for the second computer system to participate in a communication channel, wherein the first request includes first connection data. After sending the first request, Means for transmitting to the second computer system a second request for the second computer system to join the communication channel, wherein the second request includes second connection data different from the first connection data, in accordance with the determination that no valid response is received from the second computer system within a threshold period. Means for connecting to the second computer system via the communication channel, in accordance with the determination that a valid response has been received from the second computer system within the threshold period, A first computer system equipped with the following:

39. A computer program product comprising one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs include instructions, The second computer system receives a first request for the second computer system to join a communication channel, wherein the first request includes first connection data. After sending the first request, In accordance with the determination that no valid response is received from the second computer system within the threshold period, the second computer system sends a second request to the second computer system for the second computer system to join the communication channel. The second request includes a second connection data that is different from the first connection data, A computer program product including instructions for connecting to the second computer system via the communication channel, in accordance with the determination that a valid response has been received from the second computer system within the threshold period.

40. It is a method, In the first computer system, In the second computer system, One or more user interface ("UI") elements of the first computer system, and Transmitting a mapping that includes one or more action identifiers corresponding to one or more UI elements of the first computer system, Receiving an action identifier from the second computer system, which is one or more action identifiers and which is associated with one or more UI elements; Upon receiving the action identifier associated with the UI element, the action associated with the action identifier is executed. Methods that include...

41. The method according to claim 40, wherein the first computer system communicates with a display generation component, and one or more UI elements are displayed on the user interface via the display generation component.

42. The method according to claim 41, wherein the mapping includes a set of one or more locations of the one or more UI elements in the user interface relative to the user interface.

43. The one or more UI elements mentioned above, The method according to any one of claims 40 to 42, comprising one or more indications selected from the group of one or more indications of physical UI elements of the first computer system and one or more indications of virtual UI elements of the first computer system.

44. The method according to any one of claims 40 to 43, wherein the action is of type input.

45. The method according to any one of claims 40 to 44, wherein the mapping includes a plurality of action identifiers mapped to one of the one or more UI elements.

46. In response to performing the action associated with the action identifier, the second computer system: Updating one or more UI elements, Sending an update to the mapping which includes one or more action identifiers corresponding to the update to one or more UI elements, The method according to any one of claims 40 to 45, further comprising:

47. A non-temporary computer-readable medium storing one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs include instructions for performing the method according to any one of claims 40 to 46.

48. A first computer system comprising one or more processors, A memory that stores one or more programs configured to be executed by one or more processors, A first computer system comprising, wherein one or more programs include instructions for performing the method according to any one of claims 40 to 46.

49. A first computer system, A first computer system comprising means for performing the method described in any one of claims 40 to 46.

50. A computer program product comprising one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs include instructions to perform the method according to any one of claims 40 to 46.

51. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs are In the second computer system, One or more user interface elements of the first computer system, A mapping including one or more action identifiers corresponding to one or more UI elements of the first computer system is transmitted. The second computer system receives an action identifier from the one or more action identifiers, wherein the action identifier is associated with one or more UI elements. A non-temporary computer-readable storage medium containing instructions that, upon receiving the action identifier associated with the UI element, execute the action associated with the action identifier.

52. A first computer system comprising one or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are In the second computer system, One or more user interface elements of the first computer system, A mapping including one or more action identifiers corresponding to one or more UI elements of the first computer system is transmitted. The second computer system receives an action identifier from the one or more action identifiers, wherein the action identifier is associated with one or more UI elements. A first computer system, including instructions, which, upon receiving the action identifier associated with the UI element, executes an action associated with the action identifier.

53. A first computer system, which, in relation to a second computer system, means for transmitting one or more user interface elements of the first computer system, Means for transmitting a mapping that includes one or more action identifiers corresponding to one or more UI elements of the first computer system, Means for receiving an action identifier from the second computer system, which is one or more action identifiers and which is associated with one or more UI elements, A means for executing the action associated with the action identifier in response to receiving the action identifier associated with the UI element, A first computer system equipped with the following:

54. A computer program product comprising one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs include instructions, In the second computer system, One or more user interface elements of the first computer system, A mapping including one or more action identifiers corresponding to one or more UI elements of the first computer system is transmitted. The second computer system receives an action identifier from the one or more action identifiers, wherein the action identifier is associated with one or more UI elements. A computer program product that includes instructions for executing an action associated with an action identifier in response to receiving the action identifier associated with the UI element.

55. It is a method, In the first computer system, From the second computer system, One or more user interface elements of the second computer system, Receiving a mapping that includes an action identifier mapped to one or more UI elements of the second computer system, wherein the action identifier is associated with a first type of input in the second computer system, To detect an input associated with one or more of the aforementioned UI elements, wherein the input is of a second type, different from the first type of input; Using the mapping, the action identifier associated with the first type of input is transmitted to the second computer system in accordance with the determination that the input, which is the second type of input, corresponds to the first type of input in the second computer system. Methods that include...

56. The one or more UI elements mentioned above, The method according to claim 55, comprising one or more indicators selected from the group consisting of one or more physical UI elements of the second computer system and one or more indicators of virtual UI elements of the second computer system.

57. The method according to any one of claims 55 and 56, wherein the second computer system communicates with a display generation component, and the UI elements are displayed on the user interface via the display generation component.

58. The method according to claim 57, wherein the mapping includes a set of one or more locations of the UI elements within the user interface relative to the user interface.

59. Receiving one or more UI elements from the second computer system is done via a second communication channel between the first computer system and the second computer system. Receiving the mapping from the second computer system is done via a first communication channel between the first computer system and the second computer system. The first communication channel is different from the second communication channel. The method according to claim 57.

60. The method according to claim 57, wherein displaying the UI element via the display generation component is performed simultaneously with the second computer system displaying the UI element via a different display generation component.

61. Using the mapping, in accordance with the determination that the input of the second type does not correspond to the first type of input in the second computer system, To stop transmitting the action identifier associated with the first type of input to the second computer system, The method according to any one of claims 55 to 60, further comprising:

62. The method according to any one of claims 55 to 61, wherein the first type of input is a touch input and the second type of input is an audio input.

63. The first computer system communicates with the display generation component, and the method is The method according to claim 55, further comprising displaying the UI elements of one or more UI elements via the display generation component.

64. A non-temporary computer-readable medium storing one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs include instructions for performing the method according to any one of claims 55 to 63.

65. A first computer system comprising one or more processors, A memory that stores one or more programs configured to be executed by one or more processors, A first computer system comprising, wherein one or more programs include instructions for performing the method according to any one of claims 55 to 63.

66. A first computer system, A first computer system comprising means for performing the method described in any one of claims 55 to 63.

67. A computer program product comprising one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs include instructions to perform the method according to any one of claims 55 to 63.

68. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs are From the second computer system, One or more user interface elements of the second computer system, A mapping including an action identifier, which is mapped to one or more UI elements of the second computer system, wherein the action identifier is associated with a first type of input in the second computer system, is received. An input is detected that is associated with one or more of the aforementioned UI elements, and that the input is of a second type, different from the first type of input. A non-temporary computer-readable storage medium including instructions for transmitting the action identifier associated with the first type of input to the second computer system, in accordance with the determination that the input, which is the second type of input, corresponds to the first type of input in the second computer system, using the mapping described above.

69. A first computer system comprising one or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are From the second computer system, One or more user interface elements of the second computer system, A mapping including an action identifier, which is mapped to one or more UI elements of the second computer system, wherein the action identifier is associated with a first type of input in the second computer system, is received. An input is detected that is associated with one or more of the aforementioned UI elements, and that the input is of a second type, different from the first type of input. A first computer system, including instructions for transmitting the action identifier associated with the first type of input to the second computer system, according to a determination that the input, which is the second type of input, corresponds to the first type of input in the second computer system, using the mapping described above.

70. A first computer system, which is accessed from a second computer system, means for receiving one or more user interface elements of the second computer system, and Means for receiving a mapping including an action identifier, which is mapped to one or more UI elements of the second computer system, wherein the action identifier is associated with a first type of input in the second computer system. means for detecting an input associated with one or more of the aforementioned UI elements, wherein the input is a second type of input different from the first type of input; Means for transmitting the action identifier associated with the first type input to the second computer system, in accordance with the determination that the input, which is the second type input, corresponds to the first type input in the second computer system, using the mapping described above; A first computer system equipped with the following:

71. A computer program product comprising one or more programs configured to be executed by one or more processors of a first computer system, wherein the one or more programs include instructions, From the second computer system, One or more user interface elements of the second computer system, A mapping including an action identifier, which is mapped to one or more UI elements of the second computer system, wherein the action identifier is associated with a first type of input in the second computer system, is received. An input is detected that is associated with one or more of the aforementioned UI elements, and that the input is of a second type, different from the first type of input. A computer program product including instructions that, using the mapping, transmit to the second computer system the action identifier associated with the first type of input, in accordance with the determination that the input, which is the second type of input, corresponds to the first type of input in the second computer system.

Images