Remote touch detection enabled by peripheral devices

Abstract

The present application relates to remote touch detection enabled by a peripheral device. The present disclosure as a whole relates to remote touch detection. In some examples, a first electronic device obtains first image data and second image data about an input, and performs an operation based on the input according to a group that satisfies one or more criteria determined based on the first image data and the second image data. In some examples, the first electronic device causes an infrared light source of a second electronic device to emit infrared light, obtains image data about the input, and performs an operation based on the input according to a group that satisfies one or more criteria determined based on the image data.

Description

[0001] This application is a divisional application of the invention patent application with the application date of August 30, 2019, application number 201910815262.6, and name “Remote touch detection enabled by peripheral device”. Technical Field

[0002] The present disclosure relates generally to computer generated reality (CGR) environments, and more particularly to techniques for remote touch detection. Background Art

[0003] A CGR environment is an environment in which some objects displayed for viewing by a user are generated by a computer. The user can interact with these virtual objects by activating hardware buttons or touching touch-enabled hardware. However, such techniques for interacting with virtual objects can be cumbersome and unintuitive for the user. Summary of the invention

[0004] Described herein are techniques for implementing remote touch detection using a system of multiple devices including peripheral devices placed on a physical surface, such as a table top. With these techniques, a user can interact with a virtual object by touching the physical surface.

[0005] In some embodiments, a method is described. The method includes: at a first electronic device: acquiring first image data about an input, wherein the first image data is acquired using one or more camera sensors of the first electronic device that is external to a second electronic device; acquiring second image data about the input, wherein the second image data is acquired using one or more camera sensors of the second electronic device, the first electronic device being different from the second electronic device; and, based on determining a group satisfying one or more criteria based on the first image data and the second image data, performing an operation based on the input, wherein the group of one or more criteria includes a criterion satisfied when the input is a touch input.

[0006] In some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be implemented by one or more processors of a first electronic device, the one or more programs including instructions for the following operations: obtaining first image data about an input, wherein the first image data is obtained using one or more camera sensors of the first electronic device outside a second electronic device; obtaining second image data about the input, wherein the second image data is obtained using one or more camera sensors of the second electronic device, the second electronic device being different from the first electronic device; and, performing an operation based on the input according to determining a group that satisfies one or more criteria based on the first image data and the second image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input is a touch input.

[0007] In some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be implemented by one or more processors of a first electronic device, the one or more programs including instructions for the following operations: obtaining first image data about an input, wherein the first image data is obtained using one or more camera sensors of the first electronic device outside a second electronic device; obtaining second image data about the input, wherein the second image data is obtained using one or more camera sensors of the second electronic device, the second electronic device being different from the first electronic device; and, performing an operation based on the input according to determining a group of one or more criteria that are satisfied based on the first image data and the second image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input is a touch input.

[0008] In some embodiments, a first electronic device is described. The first electronic device includes: one or more processors; and a memory storing one or more programs configured to be implemented by the one or more processors, the one or more programs including instructions for the following operations: obtaining first image data about an input, wherein the first image data is obtained using one or more camera sensors of the first electronic device outside a second electronic device; obtaining second image data about the input, wherein the second image data is obtained using one or more camera sensors of the second electronic device, the second electronic device being different from the first electronic device; and, performing an operation based on the input according to determining a group that satisfies one or more criteria based on the first image data and the second image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input is a touch input.

[0009] In some embodiments, a first electronic device is described. The first electronic device includes: a device for acquiring first image data about an input, wherein the first image data is acquired using one or more camera sensors of the first electronic device outside a second electronic device; a device for acquiring second image data about the input, wherein the second image data is acquired using one or more camera sensors of the second electronic device, the second electronic device being different from the first electronic device; and a device for performing an operation based on the input according to determining a group of one or more criteria that are satisfied based on the first image data and the second image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input is a touch input.

[0010] In some embodiments, a method is described. The method includes: at a first electronic device: causing an infrared light source of a second electronic device to emit infrared light; acquiring image data about an input, wherein the image data is acquired using one or more camera sensors of the first electronic device, the second electronic device being different from the first electronic device; and, based on determining that a group of one or more criteria is satisfied based on the image data, performing an operation based on the input, wherein the group of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

[0011] In some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be implemented by one or more processors of a first electronic device, the one or more programs including instructions for the following operations: causing an infrared light source of a second electronic device to emit infrared light; acquiring image data about an input, wherein the image data is acquired using one or more camera sensors of the first electronic device, the second electronic device being different from the first electronic device; and, performing an operation based on the input according to determining a group of one or more criteria that are satisfied based on the image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

[0012] In some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be implemented by one or more processors of a first electronic device, the one or more programs including instructions for the following operations: causing an infrared light source of a second electronic device to emit infrared light; acquiring image data about an input, wherein the image data is acquired using one or more camera sensors of the first electronic device, the second electronic device being different from the first electronic device; and, performing an operation based on the input according to determining that a group of one or more criteria is satisfied based on the image data, wherein the group of one or more criteria includes a criterion satisfied when the input contacts the infrared light.

[0013] In some embodiments, a first electronic device is described. The first electronic device includes: one or more processors; and a memory storing one or more programs configured to be implemented by the one or more processors, the one or more programs including instructions for the following operations: causing an infrared light source of a second electronic device to emit infrared light; acquiring image data about an input, wherein the image data is acquired using one or more camera sensors of the first electronic device, the second electronic device being different from the first electronic device; and, performing an operation based on the input according to determining that a group of one or more criteria is satisfied based on the image data, wherein the group of one or more criteria includes a criterion satisfied when the input contacts the infrared light.

[0014] In some embodiments, a first electronic device is described. The first electronic device includes: a device for causing an infrared light source of a second electronic device to emit infrared light; a device for acquiring image data about an input, wherein the image data is acquired using one or more camera sensors of the first electronic device, and the second electronic device is different from the first electronic device; and a device for performing an operation based on the input according to determining that a group of one or more criteria is satisfied based on the image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

[0015] In some embodiments, a method is described. The method includes: at a first electronic device: causing an infrared light source of the first electronic device to emit infrared light; acquiring image data about an input, wherein the image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; and, based on determining that a group of one or more criteria is satisfied based on the image data, performing an operation based on the input, wherein the group of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

[0016] In some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be implemented by one or more processors of a first electronic device, the one or more programs including instructions for the following operations: causing an infrared light source of the first electronic device to emit infrared light; acquiring image data about an input, wherein the image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; and, performing an operation based on the input according to determining a group of one or more criteria that are satisfied based on the image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

[0017] In some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be implemented by one or more processors of a first electronic device, the one or more programs including instructions for the following operations: causing an infrared light source of the first electronic device to emit infrared light; acquiring image data about an input, wherein the image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; and, performing an operation based on the input according to determining that a group of one or more criteria is satisfied based on the image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

[0018] In some embodiments, a first electronic device is described. The first electronic device includes: one or more processors; and a memory storing one or more programs configured to be implemented by the one or more processors, the one or more programs including instructions for the following operations: causing an infrared light source of the first electronic device to emit infrared light; acquiring image data about the input, wherein the image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; and, performing an operation based on the input according to determining that a group of one or more criteria is satisfied based on the image data, wherein the group of one or more criteria includes a criterion satisfied when the input contacts the infrared light.

[0019] In some embodiments, a first electronic device is described. The first electronic device includes: a device for causing an infrared light source of the first electronic device to emit infrared light; a device for acquiring image data about an input, wherein the image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; and a device for performing an operation based on the input according to determining that a group of one or more criteria is satisfied based on the image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In the following description, reference is made to the accompanying drawings which form a part of the following description and which illustrate several examples of the present disclosure. It should be understood that other examples may also be utilized and that structural and operational changes may be made without departing from the scope of the present disclosure. The same reference numerals are used in different drawings to indicate similar or identical items.

[0021] Figure 1A to Figure 1B Depicted are exemplary systems for use in various CGR techniques.

[0022] Figure 2 An example of a system including a head mounted display device and a peripheral device for enabling remote touch detection is shown.

[0023] Figure 3 An example of a notification prompting a user to correct an error condition is shown.

[0024] Figures 4 to 5 An example of remote touch detection in a CGR environment is shown.

[0025] Figures 6 to 10 Additional examples of remote touch detection in a CGR environment are shown.

[0026] Figure 11 to Figure 12 An alternative example of a system including a head mounted display device and a peripheral device for enabling remote touch detection is shown.

[0027] Fig.13 A flow chart illustrating a method for performing remote touch detection.

[0028] Fig.14 is a flow chart illustrating an alternative method for performing remote touch detection. DETAILED DESCRIPTION

[0029] Figure 1A and Figure 1B An exemplary system 100 for use in various CGR techniques is depicted.

[0030] In some embodiments, such as Figure 1A As shown, system 100 includes device 100a. Device 100a includes various components, such as processor 102, RF circuit 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch-sensitive surface 122. These components are optionally in communication via a communication bus 150 of device 100a.

[0031] In some embodiments, elements of system 100 are implemented in a base station device (e.g., a computing device such as a remote server, a mobile device, or a laptop computer), and other elements of system 100 are implemented in a head mounted display (HMD) device designed to be worn by a user, wherein the HMD device communicates with the base station device. In some embodiments, device 100a is implemented in either the base station device or the HMD device.

[0032] like Figure 1BAs shown, in some embodiments, system 100 includes two (or more) devices in communication, such as through a wired connection or a wireless connection. The first device 100b (e.g., a base station device) includes a processor 102, an RF circuit 104, and a memory 106. These components are optionally communicated through a communication bus 150 of the device 100b. The second device 100c (e.g., a head-mounted device) includes various components, such as a processor 102, an RF circuit 104, a memory 106, an image sensor 108, an orientation sensor 110, a microphone 112, a position sensor 116, a speaker 118, a display 120, and a touch-sensitive surface 122. These components are optionally communicated through a communication bus 150 of the device 100c.

[0033] In some embodiments, system 100 is a mobile device. In some embodiments, system 100 is a HMD device. In some embodiments, system 100 is a wearable HUD device.

[0034] System 100 includes processor 102 and memory 106. Processor 102 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some embodiments, memory 106 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions, which are configured to be implemented by one or more processors 102 to perform the following techniques.

[0035] System 100 includes RF circuitry 104. RF circuitry 104 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, an intranet), and / or wireless networks (such as cellular networks and wireless local area networks (LANs)). RF circuitry 104 optionally includes circuitry for using near field communications and / or short range communications (such as ) circuit for communication.

[0036] System 100 includes a display 120. In some embodiments, display 120 includes a first display (e.g., a left eye display panel) and a second display (e.g., a right eye display panel), each display being used to display images to a corresponding eye of a user. Corresponding images are displayed simultaneously on the first display and the second display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, thereby producing a parallax effect that provides the user with a stereoscopic effect of the object on the display. In some embodiments, display 120 includes a single display. For each eye of the user, the corresponding image is simultaneously displayed on a first area and a second area of ​​a single display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, thereby producing a parallax effect that provides the user with a stereoscopic effect of the object on the single display.

[0037] In some embodiments, system 100 includes a touch-sensitive surface 122 for receiving user input, such as tap input and swipe input. In some embodiments, display 120 and touch-sensitive surface 122 form a touch-sensitive display.

[0038] System 100 includes an image sensor 108. Image sensor 108 optionally includes one or more visible light image sensors (such as a charge coupled device (CCD) sensor) and / or a complementary metal oxide semiconductor (CMOS) sensor operable to acquire an image of a physical object from a physical environment. The image sensor also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the physical environment. For example, an active IR sensor includes an IR emitter, such as an IR point emitter, for emitting infrared light into the physical environment. Image sensor 108 also optionally includes one or more event cameras configured to capture the movement of a physical object in the physical environment. Image sensor 108 also optionally includes one or more depth sensors configured to detect the distance of a physical object from system 100. In some embodiments, system 100 uses a combination of a CCD sensor, an event camera, and a depth sensor to detect the physical environment surrounding system 100. In some embodiments, image sensor 108 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical objects in a physical environment from two different perspectives. In some embodiments, the system 100 uses the image sensor 108 to receive user input, such as a gesture. In some embodiments, the system 100 uses the image sensor 108 to detect the position and orientation of the system 100 and / or the display 120 in the physical environment. For example, the system 100 uses the image sensor 108 to track the position and orientation of the display 120 relative to one or more fixed objects in the physical environment. In some embodiments, the system 100 uses the image sensor 108 facing inward (e.g., facing the user) for gaze tracking and / or hand movement tracking, which can be used, for example, to control the user's avatar.

[0039] In some embodiments, the system 100 includes a microphone 112. The system 100 uses the microphone 112 to detect sounds from the user and / or the user's physical environment. In some embodiments, the microphone 112 includes a microphone array (including multiple microphones) that are optionally operated in series to identify ambient noise or locate a sound source in the space of the physical environment.

[0040] System 100 includes an orientation sensor 120 for detecting orientation and / or movement of system 110 and / or display 100. For example, system 100 uses orientation sensor 110 to track changes in position and / or orientation of system 100 and / or display 120, such as relative to physical objects in a physical environment. Orientation sensor 110 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0041] The physical environment (or real environment) refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical items (or physical objects or real objects), such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0042] In contrast, a computer generated reality (CGR) environment refers to a fully or partially simulated environment that a person perceives and / or interacts with via an electronic system. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect a person's head turning, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0043] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object, which creates a 3D or spatial audio environment that provides a perception of a point audio source in a 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0044] Examples of CGR include virtual reality and mixed reality. A virtual reality (VR) environment (virtual environment) refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment, and / or through a simulation of a subset of the person's physical movements within the computer-generated environment.

[0045] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0046] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0047] Examples of mixed reality include augmented reality and augmented virtuality. An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment through an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0048] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not true version of the original captured image. For another example, a representation of a physical environment may be transformed by graphically eliminating or blurring a portion thereof.

[0049] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more features of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the position of the sun in the physical environment.

[0050] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, silicon-based liquid crystal, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0051] Figure 2 A system 200 is shown that includes a peripheral device 200A and a head mounted display (HMD) device 200B. The peripheral device includes a camera sensor 210 (e.g., image sensor 108) and a motion sensor (e.g., orientation sensor 110). In addition, the HMD device 200B itself can be an embodiment of the system 100, or can be an embodiment of a portion of the system 100 (such as device 100a).

[0052] The system 200 including the peripheral device 200A and the HMD device 200B enables accurate remote touch detection on the surface 206 to interact (e.g., control, manipulate, activate, select) with UI elements displayed in the CGR environment displayed using the HMD device 200B. In some embodiments, the system 200 forms a triangulated system (e.g., one camera sensor on the peripheral device 200A and two camera sensors on the HMD device 200B). In some embodiments, the system 200 includes the peripheral device 200A (e.g., without a camera sensor) that emits infrared light for detection by the camera sensor on the HMD device 200B. In some embodiments, the system 200 does not include the peripheral device 200A and relies on the HMD device 200B to perform remote touch detection using depth sensing technology.

[0053] exist Figure 2 2, a user 202 is wearing an HMD device 200B that enables the user to view a CGR environment. The user 202 is sitting at a table 204 in a physical environment, wherein the table 204 includes a surface 206 on which physical objects such as a peripheral device 200A and an object 208 (e.g., a physical cup) are placed.

[0054] Figure 3 The perspective of the user 202 viewing the CGR environment via the transparent or semi-transparent display of the HMD device 200B is shown. In some embodiments, the display of the HMD device 200B is opaque, so the user views the physical environment using a pass-through video.

[0055] exist Figure 3 , the peripheral device 200A is located on a surface 206 adjacent to an object 208. For optimal operation, the peripheral device 200A can request additional space near or around itself so that one or more camera sensors 210 are not obstructed by occluders (e.g., obstacles). Figure 3In some embodiments, the system 200 determines whether a group of one or more operating conditions has been met. In some embodiments, the set of operating conditions includes requiring that there is a direct line of sight between a sensor (e.g., camera sensor 210) on the peripheral device 200A and a sensor on the HMD device 200B. In some embodiments, the set of operating conditions includes requiring that no object obstructs (e.g., obstructs) the camera sensor 210. For example, when no object is within a predefined threshold distance of the camera sensor, then no object obstructs the camera sensor 210. Due to the close proximity of the object 208 to the camera sensor 210, the system 200 determines that the camera sensor 210 is obstructed. The peripheral device 200A, the HMD device 200B, or a third device can determine whether the camera sensor 210 is obstructed. In some embodiments, the set of operating conditions includes requiring that the peripheral device 200A is in a suitable orientation (e.g., the surface of the peripheral device is roughly parallel to the ground, is horizontal, is vertical, etc.).

[0056] In some embodiments, determining whether the camera sensor 210 is blocked includes: acquiring image data about an object that is close to or adjacent to the camera sensor 210; and determining whether the camera sensor 210 is blocked based on the image data. In some embodiments, the HMD device 200B determines whether the camera sensor 210 is blocked. In some embodiments, determining by the HMD device 200B whether the camera sensor 210 is blocked includes receiving image data about an object in the field of view of the camera sensor 210 from the peripheral device 200A. In some embodiments, after receiving the image data from the peripheral device 200A (or in response thereto), the HMD device 200B determines whether the camera sensor 210 is blocked based on the image data. In some embodiments, the image data includes image data generated using the camera sensor 210 and / or a camera sensor at the HMD device 200B.

[0057] After determining that the camera sensor 210 is blocked, the HMD device 200B displays a notification 212 that prompts the user to correct the error state. The notification 212 includes an indication that the peripheral device 200A is blocked. The notification 212 also includes a prompt to move the peripheral device 200A to a new location. In some embodiments, the notification 212 includes a prompt to move the object that is causing the camera sensor 210 to be blocked.

[0058] exist Figure 4At , user 202 views the CGR environment after peripheral device 200A has moved to a new location. After peripheral device 200A moves to a new location (or in response thereto), system 200 determines whether camera sensor 210 is blocked. System 200 determines that camera sensor 210 is not blocked. For example, system 200 determines that object 208 is located outside a threshold distance from camera sensor 210. In some embodiments, system 200 determines whether camera sensor 210 is blocked in response to determining that peripheral device 200A has moved to a new location. Peripheral device 200A, HMD device 200B, or a third device may determine whether peripheral device 200A has moved to a new location. In some embodiments, data obtained from a motion sensor of peripheral device 200A is used to make this determination. After determining that a group of operating conditions for peripheral device 210A is met (e.g., camera sensor 210 is no longer blocked), HMD device 200B stops displaying notification 212.

[0059] exist Figure 4 At , after (or in response to) a set of operating conditions for the peripheral device 210A is satisfied, the HMD device 200B displays a user interface (UI). Figure 4 In the embodiment of the present invention, the user interface includes UI elements 214A, 214B, 214C, and 214D. In addition, the HMD device 200B displays a virtual trackpad 218, which is displayed in such a way that the virtual trackpad 218 appears to be placed on (or overlapped with) the surface 206. In some embodiments, the HMD device 200B does not display the virtual trackpad 218. In some embodiments, in addition to or instead of the virtual trackpad 218, the HMD device 200B displays a virtual keyboard. In some embodiments, when the user performs a tap gesture at the position of a key of the displayed virtual keyboard (or a position corresponding thereto), the system 200 receives input via the virtual keyboard.

[0060] Once the UI element is displayed, it is assumed that the user notices that the UI element 214A is in the wrong orientation and therefore wishes to correct its orientation. The user begins by preparing to perform a tap gesture on the surface 206. The peripheral device 200A uses the camera sensor 210 to obtain information about the object 216A (e.g., the index finger of the user's hand). In addition, the HMD device 200B obtains information about the object 216A via the camera sensor located at the HMD device 200B. Therefore, both the peripheral device 200A and the HMD device 200B use their corresponding camera sensors to obtain information about the object 216A (e.g., image data). Obtaining information about the object may include using the camera sensor to capture one or more images of the object in the field of view of the camera sensor. Capturing information about the object 216A from the perspective of both the device 200A and 200B allows the device to better resolve the position and movement of the object 216A, thereby contributing to the accuracy of the detection.

[0061] exist Figure 5 At , when UI element 214A is highlighted for selection (e.g., emphasized relative to other UI elements), the user selects UI element 214A for editing by performing a tap gesture on surface 206 using object 216A. As object 216A moves toward surface 206, peripheral device 200A and / or HMD device 200B uses the camera sensor of the corresponding device to capture information about object 216A. System 200 uses this information to determine whether the movement of object 216A meets one or more standard groups, and one or more standard groups include, for example, requiring the movement of the object to cause touch input. Peripheral device 200A, HMD device 200B, or a third device can make this determination. In some embodiments, determining whether the movement of object 216A meets one or more standard groups includes determining whether the movement of object 216A is a valid input (e.g., a valid input to system 200). In some embodiments, one or more standard groups include requiring the touch input to occur within the boundaries of virtual trackpad 218. In some embodiments, the system 200 uses the captured images to track an object (e.g., object 216A) using 3D object detection and / or template matching. In some embodiments, the system 200 extracts hand gesture information and / or finger gesture information from the captured images to determine whether the movement of the object 216A satisfies a set of one or more criteria.

[0062] In some embodiments, when the information obtained by the camera sensor indicates that the object has touched the surface (e.g., coincident with it), the movement of the object will cause touch input. In some embodiments, one or more standard groups include whether the movement of the object is a recognized gesture. It is worth noting that since the device 200B is also located on the surface, it can better detect the tap on the surface, so the sensor 210 may have a clear line of sight to the tap position. In contrast, the device 200A can also rely on depth analysis to determine whether the object has moved along the z-axis in this way to characterize it as a tap on the surface. In some embodiments, one or more standard groups include requiring the object to be a valid object. For example, when the object is a finger of a hand, the object is a valid object. In some embodiments, when the object is a writing tool (e.g., a pen, a pencil) or a stylus, the object is a valid object.

[0063] exist Figure 5 At , system 200 determines that the movement of object 216A satisfies the group of one or more criteria. For example, determining that the movement of object 216A satisfies the group of one or more criteria includes determining that the movement of object 216A is a recognized gesture (e.g., a tap gesture). After determining that the movement of object 216A satisfies the group of one or more criteria, system 200 selects UI element 214A based on the tap gesture.

[0064] exist Figure 6 In response to the selection of UI element 214A, HMD device 200B displays UI element 214A without displaying other UI elements. User 202 can use a rotation gesture to correct the orientation of UI element 214A. Peripheral device 200A and / or HMD device 200B use corresponding camera sensors to capture information about object 216A and / or object 216B (e.g., the thumb of the user's hand).

[0065] exist Figure 7 , when the user performs a rotation gesture, peripheral device 200A and HMD device 200B continue to use their respective camera sensors to capture information about object 216A and / or object 216B. Due to the position and orientation of object 216A, the information captured by the camera sensor of HMD device 200B is insufficient to determine whether the movement of objects 216A and 216B meets the set of one or more criteria. Specifically, the captured information does not include information about object 216A because object 216A is not in the field of view of the camera sensor of HMD device 200B. However, due to the position of peripheral device 200A, the information captured by camera sensor 210 of peripheral device 200A includes information about object 216A. This scenario demonstrates the advantages of a system with camera sensors that capture information from different perspectives (e.g., angles).

[0066] Using the captured information from the peripheral device 200A and the HMD device 200B, the system 200 determines that the movement of the object 216A and the object 216B satisfies a set of one or more criteria. For example, the system 200 determines based on the captured information that the movement of the objects 216A and 216B results in a touch input and is a recognized gesture (e.g., a rotation gesture). After determining that the movement of the objects 216A and the object 216B satisfies a set of one or more criteria, the HMD device 200B displays the image from its position in the image. Figure 6 The initial orientation in is rotated clockwise to Figure 7 The intermediate-oriented UI element 214A is shown in FIG.

[0067] exist Figure 8 At , as the user continues to perform the rotation gesture, peripheral device 200A and HMD device 200B continue to use their respective camera sensors to capture information about object 216A and / or object 216B. Using the captured information, system 200 determines whether the movement of objects 216A and 216B continues to satisfy the set of one or more criteria. After determining that the movement of objects 216A and 216B continues to satisfy the set of one or more criteria, HMD device 200B displays the image from its original position. Figure 7 The intermediate orientation in the further clockwise rotation to Figure 8 The UI element 214A is shown in the final orientation.

[0068] exist Fig. 9 Once UI element 214A is in the desired orientation, the user wishes to increase the size of displayed UI element 214A. To enlarge displayed UI element 214A, the user initiates an expand gesture. Figures 6 to 8 As described in the rotation gesture in FIG. 2 , the peripheral device 200A and the HMD device 200B use corresponding camera sensors to capture information about the object 216A and the object 216B.

[0069] As the user performs the expand gesture, peripheral device 200A and HMD device 200B continue to capture information about object 216A and object 216B using their respective camera sensors. Using the captured information from peripheral device 200A and HMD device 200B, system 200 determines that the movement of object 216A and object 216B satisfies a set of one or more criteria. For example, system 200 determines based on the captured information that the movement of object 216A and object 216B results in touch input and is a recognized gesture (e.g., an expand gesture). After determining that the movement of object 216A and object 216B satisfies a set of one or more criteria, HMD device 200B displays UI element 214A moved from its original position to the original position. Fig. 9 The initial size in the Fig.10 Animation of the final size shown in .

[0070] Fig.11 An alternative embodiment of system 200 is shown. Figures 4 to 10 In some embodiments, the peripheral device 200A includes a camera sensor 210 for capturing information about objects in its field of view. In some embodiments, the peripheral device 200A does not have a camera sensor. In some embodiments, the peripheral device 200A includes an infrared emitter 220 that emits infrared light to enable remote touch detection. In some embodiments, the infrared emitter 220 emits infrared light 222 as a light plane parallel (or substantially parallel) to the surface 206. To detect remote touch, the HMD device 200B obtains information about objects that coincide with (e.g., intersect) the infrared light 222.

[0071] For example, something like Figure 4 In the scenario, assuming Fig.11 User 202 in the embodiment of the present invention wishes to select UI element 214A in order to correct the orientation of the UI element. When UI element 214A is highlighted for selection (e.g., emphasized relative to other UI elements), the user selects UI element 214A for editing by performing a tap gesture on surface 206 using object 216A.

[0072] exist Fig.12 At , when object 216A moves toward the plane of infrared light 222, HMD device 200B uses its camera sensor to capture information about object 216A. For example, once object 216A coincides with the plane of infrared light 222, HMD device 200B detects reflection of infrared light caused by object 216A that coincides with the plane of infrared light 222 (e.g., identifies occlusion). System 200 uses this captured information to determine whether the movement of object 216A satisfies a set of one or more criteria, including, for example, that the movement of the object results in a touch input. Peripheral device 200A, HMD device 200B, or a third device may make this determination.

[0073] In some embodiments, the movement of the object results in a touch input when the system 200 determines that the object has touched (e.g., coincident with) the surface based on information acquired by the camera sensor. In some embodiments, the system 200 determines that the object has touched the surface when the object has coincided with the plane of infrared light 222. Fig.12 At this point, system 200 determines that the movement of object 216A satisfies a set of one or more criteria. After determining that the movement of object 216A satisfies a set of one or more criteria, system 200 selects UI element 214A based on the tap gesture, such as relative to Figure 5 as described.

[0074] Fig.131 is a flow chart illustrating a method 1300 for remote touch detection. In some embodiments, the method is performed by the peripheral device 200A. In some embodiments, the method is performed by the HMD device 200B. In some embodiments, the method is performed by a third device different from the peripheral device 200A and the HMD device 200B.

[0075] At box 1302, the first electronic device obtains first image data (e.g., a captured image) about an input (e.g., 216A, 216B) (or movement of an object (e.g., 216A, 216B)), wherein the first image data is obtained using one or more camera sensors (e.g., image sensor 108) of a second electronic device (e.g., a peripheral device (e.g., 200A), an HMD device (e.g., 200B)). In some embodiments, obtaining the first image data includes obtaining image data about a detected physical surface (e.g., 206). In some embodiments, before obtaining the first image data and the second image data about the input, the first electronic device determines whether there is a suitable environment for receiving touch input (e.g., the second electronic device is resting on a flat surface) based on orientation data (e.g., motion data) from the second electronic device. In some embodiments, the input (or movement of the object) occurs at a location other than the location of the second electronic device and the location of the first electronic device.

[0076] In some embodiments, based on determining that one or more camera sensors of the second electronic device are blocked, the first electronic device prompts (e.g., via a visual prompt (e.g., 212), an auditory prompt, and / or a tactile prompt) to move the second electronic device to a new position. In some embodiments, the method further includes: determining whether one or more camera sensors of the second electronic device are blocked based on image data (e.g., acquired using one or more camera sensors of the second electronic device and / or the first electronic device). In some embodiments, the image data includes information about an object (e.g., 208) in the field of view of one or more camera sensors of the second electronic device (or where no object exists) or information about an object adjacent to or close to the second electronic device (or where no object exists). In some embodiments, it is determined that one or more camera sensors of the second electronic device are blocked when the object (e.g., 208) in the field of view is within a threshold distance of one or more camera sensors. In some embodiments, the method includes prompting to move the second electronic device, or prompting to move the object causing the blockage, or a combination thereof.

[0077] At box 1304, the first electronic device obtains second image data (e.g., a captured image) about the input (e.g., 216A, 216B) (or the movement of the object (e.g., 216A, 216B)), wherein the second image data is obtained using one or more camera sensors (e.g., image sensor 108) of the first electronic device (e.g., a head-mounted device (e.g., 200B), a peripheral device (e.g., 200A)), and the second electronic device is different from the first electronic device. In some embodiments, the first electronic device may be the same as the second electronic device. In some embodiments, obtaining the second image data includes obtaining image data about the detected physical surface. In some embodiments, the first device communicates with the second device. In some embodiments, the first image data and the second image data correspond to the same time point or time period. In some embodiments, the input includes movement of a first object (e.g., a first finger of a hand) and a second object (e.g., a second finger of a hand), and at least one of the first image data and the second image data does not include information related to both the first object and the second object. In some embodiments, the first image data and the second image data provide sufficient data in combination for determining whether a group of one or more criteria is met.

[0078] At box 1306, determine whether one or more groups of standards are met based on the first image data and the second image data, wherein the one or more groups of standards include standards that are met when the input is a touch input. In some embodiments, the one or more groups of standards include standards that are met when the movement of the object is a valid input. In some embodiments, the valid input is a touch input. In some embodiments, the valid input is a touch input performed by a valid object (e.g., a finger, a finger-shaped portion, a pen, a stylus). In some embodiments, when the movement of the object is not a touch input, the one or more groups of standards are not met. In some embodiments, when the movement of the object is performed by an invalid object (e.g., an object other than a finger-shaped portion or a finger of the user), the one or more groups of standards are not met.

[0079] In some embodiments, the criteria include criteria that are met when the input (or movement of an object) is a gesture (e.g., a recognized gesture) (e.g., tap, double-click, drag, flick, pinch, expand, press, rotate). In some embodiments, the criteria include criteria that are met when the input (or movement of an object) is a tap gesture. In some embodiments, the criteria include criteria that are met when the input (or movement of an object) is a pinch gesture or an expand gesture. In some embodiments, the criteria include criteria that are met when the input (or movement of an object) is a rotation gesture. In some embodiments, the criteria include criteria that are met when the input (or movement of an object) corresponds to (e.g., is performed by) one or more fingers (e.g., a finger-like portion) of a user. In some embodiments, when the movement of an object is performed by an invalid object (e.g., an object other than a finger-like portion or finger of a user), a group of one or more criteria is not met. In some embodiments, the group that meets one or more criteria is determined by the first electronic device. In some embodiments, the second electronic device makes this determination.

[0080] At block 1308, based on determining that the group of one or more criteria is satisfied, the first electronic device performs an operation based on (e.g., in response to) the input (or movement of the object) (input in contact with the detected surface (e.g., 206) (e.g., the top of a table or desk). In some embodiments, based on determining that the group of one or more criteria is not satisfied based on the first image data and the second image data, the first electronic device abandons performing the operation based on the input.

[0081] In some embodiments, the first electronic device uses one or more camera sensors of the first electronic device to obtain image data about the second electronic device. In some embodiments, the first electronic device determines the position of the second electronic device based on the image data about the second electronic device. In some embodiments, technology (such as 3D object detection and / or template matching with image data) is used to determine the position of the second electronic device. In some embodiments, the first electronic device displays a virtual keyboard (or touchpad (e.g., 218)) based on the position of the second electronic device. In some embodiments, the position of the second electronic device is used as a reference point for determining where to display the virtual keyboard (or touchpad). In some embodiments, the virtual keyboard (or touchpad) is displayed in such a way that the virtual keyboard (or touchpad) appears to be resting on a physical surface (e.g., 206) (i.e., the same physical surface on which the second electronic device is resting).

[0082] Fig.1414 is a flow chart illustrating a method 1400 for remote touch detection. In some embodiments, the method is performed by device 200A. In some embodiments, the method is performed by device 200B. In some embodiments, the method is performed by a third device different from peripheral device 200A and HMD device 200B.

[0083] At box 1402, the first electronic device causes the infrared light source (e.g., 220) (e.g., one or more infrared light sources) of the second electronic device (e.g., 200A, 200B) to emit infrared light (e.g., 222). In some embodiments, based on determining that the infrared light source of the second electronic device is blocked, the first electronic device prompts (e.g., via a visual prompt (e.g., 212), an auditory prompt, and / or a tactile prompt) to move the second electronic device to a new position. In some embodiments, the method further includes: determining whether the infrared light source of the second electronic device is blocked based on image data (e.g., acquired using one or more camera sensors of the first electronic device). In some embodiments, the image data includes information about an object that is close to or adjacent to the second electronic device (or in which there is no object). In some embodiments, it is determined that the infrared light source of the second electronic device is blocked when the object is within a threshold distance of the second electronic device or the infrared light source. In some embodiments, the method includes prompting to move the second electronic device, or prompting to move the object that causes the blockage, or a combination thereof.

[0084] At box 1404, the first electronic device obtains image data about the input (e.g., 216A, 216B) (or the movement of the object (e.g., 216A, 216B)), wherein the image data is obtained using one or more camera sensors of the first electronic device (e.g., 200A, 200B), and the second electronic device is different from the first electronic device. In some embodiments, the input (or the movement of the object) occurs at a location other than the location of the second electronic device and the location of the first electronic device.

[0085] At box 1406, determine whether one or more groups of standards are met based on the image data, wherein the group of one or more standards includes a standard that is met when the input (e.g., 216A, 216B) contacts the infrared light (e.g., 222). In some embodiments, the group of one or more standards includes a standard that is met when the movement of the object is a valid input. In some embodiments, the valid input is a touch input. In some embodiments, the valid input is a touch input performed by a valid object (e.g., a finger, a finger-shaped portion, a pen, a stylus). In some embodiments, when the movement of the object is not a touch input, the group of one or more standards is not met. In some embodiments, when the movement of the object is performed by an invalid object (e.g., an object other than a finger-shaped portion or a finger of the user), the group of one or more standards is not met.

[0086] In some embodiments, the criteria include criteria that are met when the input (or movement of an object) is a gesture (e.g., a recognized gesture) (e.g., tap, double-click, drag, flick, pinch, expand, press, rotate). In some embodiments, the criteria include criteria that are met when the input (or movement of an object) is a tap gesture. In some embodiments, the criteria include criteria that are met when the input (or movement of an object) is a pinch gesture or an expand gesture. In some embodiments, the criteria include criteria that are met when the input (or movement of an object) is a rotation gesture. In some embodiments, the criteria include criteria that are met when the input (or movement of an object) corresponds to (e.g., is performed by) one or more fingers (e.g., a finger-like portion) of a user. In some embodiments, when the movement of an object is performed by an invalid object (e.g., an object other than a finger-like portion or finger of a user), a group of one or more criteria is not met. In some embodiments, the group that meets one or more criteria is determined by the first electronic device. In some embodiments, the second electronic device makes this determination.

[0087] At box 1408, based on determining that the group of one or more criteria is satisfied, the first electronic device performs an operation based on (e.g., in response to) the input (or movement of the object) (input in contact with the detected surface (e.g., 206) (e.g., the top of a table, desk). In some embodiments, based on determining that the group of one or more criteria is not satisfied based on the image data, the first electronic device abandons performing the operation based on the input.

[0088] In some embodiments, the first electronic device uses one or more camera sensors of the first electronic device (e.g., 200B) to obtain image data about the second electronic device (e.g., 200A). In some embodiments, the first electronic device determines the position of the second electronic device based on the image data about the second electronic device. In some embodiments, technology (such as 3D object detection and / or template matching with image data) is used to determine the position of the second electronic device. In some embodiments, the first electronic device displays a virtual keyboard (or touchpad (e.g., 218)) based on the position of the second electronic device. In some embodiments, the position of the second electronic device is used as a reference point for determining where to display the virtual keyboard (or touchpad). In some embodiments, the virtual keyboard (or touchpad) is displayed in such a way that the virtual keyboard (or touchpad) appears to be resting on a physical surface (e.g., 206) (i.e., the same physical surface on which the second electronic device is resting).

[0089] The details of the process described above with respect to method 1400 also apply in a similar manner to the method described with respect to method 1300. For example, method 1300 optionally includes one or more features of the method described above with reference to method 1400. For example, method 1300 may also include emitting infrared light to achieve accurate remote touch detection. For the sake of brevity, these details are not repeated below.

[0090] Advantageously, methods 1300 and 1400 improve the accuracy and robustness of remote touch detection. For example, methods 1300 and 1400 enable accurate detection of remote touch, including the ability to ignore spurious inputs (e.g., non-hand inputs, palm inputs). This can result in an efficient and intuitive technique for interacting with a CGR environment.

[0091] In addition, the present disclosure also relates to the following embodiments:

[0092] Embodiment 1. A method comprising:

[0093] acquiring first image data regarding the input, wherein the first image data is acquired using one or more camera sensors of a first electronic device external to a second electronic device;

[0094] acquiring second image data regarding the input, wherein the second image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; and

[0095] An operation is performed based on the input in accordance with determining a group of one or more criteria to be satisfied based on the first image data and the second image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input is a touch input.

[0096] Embodiment 2. The method according to embodiment 1 further includes:

[0097] Based on determining that the set of one or more criteria is not satisfied based on the first image data and the second image data, performing the operation based on the input is abandoned.

[0098] Embodiment 3. The method according to any one of embodiments 1 to 2, further comprising:

[0099] Before acquiring the first image data and the second image data regarding the input, determining whether there is a suitable environment for receiving the touch input based on the orientation data from the second electronic device.

[0100] Embodiment 4. The method of any one of Embodiments 1 to 3, wherein the input occurs at a location other than the location of the first electronic device and the location of the second electronic device.

[0101] Embodiment 5. The method of any one of Embodiments 1 to 4, wherein the criterion comprises a criterion satisfied when the input is a gesture.

[0102] Embodiment 6. The method of Embodiment 5, wherein the criteria include criteria that are satisfied when the input is a tap gesture.

[0103] Embodiment 7. The method of Embodiment 5, wherein the criteria include criteria that are satisfied when the input is a pinch gesture or an expand gesture.

[0104] Embodiment 8. The method of Embodiment 5, wherein the criteria include criteria that are satisfied when the input is a rotation gesture.

[0105] Embodiment 9. A method according to any one of Embodiments 1 to 8, wherein the criterion includes a criterion that is satisfied when the input corresponds to one or more fingers of a user.

[0106] Embodiment 10. The method of any one of Embodiments 1 to 9, wherein the determination of the group satisfying one or more criteria is made by the first electronic device.

[0107] Embodiment 11. The method according to any one of embodiments 1 to 10, wherein the first image data and the second image data correspond to the same time point or time period.

[0108] Embodiment 12. The method according to any one of embodiments 1 to 11, further comprising:

[0109] Based on determining that the one or more camera sensors of the second electronic device are blocked, a prompt is provided to move the second electronic device to a new location.

[0110] Embodiment 13. The method according to any one of embodiments 1 to 12, further comprising:

[0111] acquiring image data about the second electronic device using the one or more camera sensors of the first electronic device; and

[0112] determining a location of the second electronic device based on the image data regarding the second electronic device; and

[0113] A virtual keyboard is displayed based on the position of the second electronic device.

[0114] Embodiment 14. The method according to any one of embodiments 1 to 13, wherein:

[0115] The input includes movement of the first object and the second object, and

[0116] At least one of the first image data and the second image data does not include information about both the first object and the second object.

[0117] Embodiment 15. The method of Embodiment 14, wherein the first image data and the second image data in combination provide sufficient data for determining whether the set of one or more criteria is met.

[0118] Embodiment 16. A system comprising:

[0119] one or more processors; and

[0120] One or more memories storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of embodiments 1 to 15.

[0121] Embodiment 17. The system of Embodiment 16, wherein the one or more memories include a first memory of the first electronic device and a second memory of the second electronic device.

[0122] Embodiment 18. The system of any one of Embodiments 16-17, wherein the one or more memories include a third memory of a third electronic device different from the first electronic device and the second electronic device.

[0123] Embodiment 19. A method comprising:

[0124] At the first electronic device:

[0125] acquiring first image data regarding the input, wherein the first image data is acquired using one or more camera sensors of a first electronic device external to a second electronic device;

[0126] acquiring second image data regarding the input, wherein the second image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; and

[0127] An operation is performed based on the input in accordance with determining a group of one or more criteria to be satisfied based on the first image data and the second image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input is a touch input.

[0128] Embodiment 20. The method according to embodiment 19 further comprises:

[0129] Based on determining that the set of one or more criteria is not satisfied based on the first image data and the second image data, performing the operation based on the input is abandoned.

[0130] Embodiment 21. The method according to any one of embodiments 19 to 20, further comprising:

[0131] Before acquiring the first image data and the second image data regarding the input, determining whether there is a suitable environment for receiving the touch input based on the orientation data from the second electronic device.

[0132] Embodiment 22. The method of any one of Embodiments 19 to 21, wherein the input occurs at a location other than the location of the first electronic device and the location of the second electronic device.

[0133] Embodiment 23. A method according to any one of Embodiments 19 to 22, wherein the criterion includes a criterion that is satisfied when the input is a gesture.

[0134] Embodiment 24. The method of Embodiment 23, wherein the criteria include criteria that are satisfied when the input is a tap gesture.

[0135] Embodiment 25. The method of Embodiment 23, wherein the criteria include criteria that are satisfied when the input is a pinch gesture or an expand gesture.

[0136] Embodiment 26. The method of Embodiment 23, wherein the criteria include criteria that are satisfied when the input is a rotation gesture.

[0137] Embodiment 27. A method according to any one of Embodiments 19 to 26, wherein the criterion includes a criterion that is satisfied when the input corresponds to one or more fingers of a user.

[0138] Embodiment 28. The method of any one of Embodiments 19 to 27, wherein the determination of the group satisfying one or more criteria is made by the first electronic device.

[0139] Embodiment 29. A method according to any one of embodiments 19 to 28, wherein the first image data and the second image data correspond to the same time point or time period.

[0140] Embodiment 30. The method according to any one of embodiments 19 to 29, further comprising:

[0141] Based on determining that the one or more camera sensors of the second electronic device are blocked, a prompt is provided to move the second electronic device to a new location.

[0142] Embodiment 31. The method according to any one of embodiments 19 to 30, further comprising:

[0143] acquiring image data about the second electronic device using the one or more camera sensors of the first electronic device; and

[0144] determining a location of the second electronic device based on the image data regarding the second electronic device; and

[0145] A virtual keyboard is displayed based on the position of the second electronic device.

[0146] Embodiment 32. The method according to any one of embodiments 19 to 31, wherein:

[0147] The input includes movement of the first object and the second object, and

[0148] At least one of the first image data and the second image data does not include information about both the first object and the second object.

[0149] Embodiment 33. A method according to Embodiment 32, wherein the first image data and the second image data in combination provide sufficient data for determining whether the set of one or more criteria is met.

[0150] Embodiment 34. A non-transitory computer-readable storage medium storing one or more programs configured to be implemented by one or more processors of a first electronic device, wherein the one or more programs include instructions for executing a method according to any one of Embodiments 19 to 33.

[0151] Embodiment 35. A first electronic device, comprising:

[0152] one or more processors; and

[0153] A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of embodiments 19 to 33.

[0154] Embodiment 36. A first electronic device, comprising:

[0155] An apparatus for performing the method according to any one of embodiments 19 to 33.

[0156] Embodiment 37. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first electronic device, the one or more programs comprising instructions for performing the following operations:

[0157] acquiring first image data regarding an input, wherein the first image data is acquired using one or more camera sensors of the first electronic device external to a second electronic device;

[0158] acquiring second image data regarding the input, wherein the second image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; and

[0159] An operation is performed based on the input in accordance with determining a group of one or more criteria to be satisfied based on the first image data and the second image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input is a touch input.

[0160] Embodiment 38. A first electronic device, comprising:

[0161] one or more processors; and

[0162] A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations:

[0163] acquiring first image data regarding an input, wherein the first image data is acquired using one or more camera sensors of the first electronic device external to a second electronic device;

[0164] acquiring second image data regarding the input, wherein the second image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; and

[0165] An operation is performed based on the input in accordance with determining a group of one or more criteria to be satisfied based on the first image data and the second image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input is a touch input.

[0166] Embodiment 39. A first electronic device, comprising:

[0167] means for acquiring first image data regarding an input, wherein the first image data is acquired using one or more camera sensors of the first electronic device external to a second electronic device;

[0168] means for acquiring second image data regarding the input, wherein the second image data is acquired using one or more camera sensors of the second electronic device, the first electronic device being different from the second electronic device; and

[0169] Means for performing an operation based on the input according to determining a group of one or more criteria that are satisfied based on the first image data and the second image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input is a touch input.

[0170] Embodiment 40. A method comprising:

[0171] At the first electronic device:

[0172] acquiring first image data regarding the input, wherein the first image data is acquired using one or more camera sensors of a first electronic device external to a second electronic device;

[0173] acquiring second image data regarding the input, wherein the second image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; and

[0174] An operation is performed based on the input in accordance with determining a group of one or more criteria to be satisfied based on the first image data and the second image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input is a touch input.

[0175] Embodiment 41. The method according to embodiment 40 further comprises:

[0176] Based on determining that the set of one or more criteria is not satisfied based on the first image data and the second image data, performing the operation based on the input is abandoned.

[0177] Embodiment 42. The method according to any one of embodiments 40 to 41, further comprising:

[0178] Before acquiring the first image data and the second image data regarding the input, determining whether there is a suitable environment for receiving a touch input based on the orientation data from the first electronic device.

[0179] Embodiment 43. A method according to any one of embodiments 40 to 42, wherein the input

[0180] Occurs at a location other than the location of the first electronic device and the location of the second electronic device.

[0181] Embodiment 44. A method according to any one of Embodiments 40 to 43, wherein the criterion includes a criterion that is satisfied when the input is a gesture.

[0182] Embodiment 45. A method according to Embodiment 44, wherein the criteria include criteria that are met when the input is a tap gesture.

[0183] Embodiment 46 The method of Embodiment 44, wherein the criteria include criteria that are satisfied when the input is a pinch gesture or an expand gesture.

[0184] Embodiment 47. A method according to Embodiment 44, wherein the criteria include criteria that are met when the input is a rotation gesture.

[0185] Embodiment 48. A method according to any one of Embodiments 40 to 47, wherein the criterion includes a criterion that is satisfied when the input corresponds to one or more fingers of a user.

[0186] Embodiment 49. A method according to any one of Embodiments 40 to 48, wherein the determination of the group satisfying one or more criteria is made by the first electronic device.

[0187] Embodiment 50. A method according to any one of Embodiments 40 to 49, wherein the first image data and the second image data correspond to the same time point or time period.

[0188] Embodiment 51. The method according to any one of embodiments 40 to 50, further comprising:

[0189] Based on determining that the one or more camera sensors of the first electronic device are blocked, a prompt is provided to move the first electronic device to a new location.

[0190] Embodiment 52. The method according to any one of embodiments 40 to 51, further comprising:

[0191] acquiring image data about the first electronic device using the one or more camera sensors of the second electronic device; and

[0192] determining a location of the first electronic device based on the image data regarding the first electronic device; and

[0193] A virtual keyboard is displayed based on the position of the first electronic device.

[0194] Embodiment 53. The method according to any one of embodiments 40 to 52, wherein:

[0195] The input includes movement of the first object and the second object, and

[0196] At least one of the first image data and the second image data does not include information about both the first object and the second object.

[0197] Embodiment 54. A method according to Embodiment 53, wherein the first image data and the second image data in combination provide sufficient data for determining whether the group of one or more criteria is met.

[0198] Embodiment 55. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first electronic device, wherein the one or more programs include instructions for executing a method according to any one of Embodiments 40 to 54.

[0199] Embodiment 56. A first electronic device, comprising:

[0200] One or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for executing the method according to any one of embodiments 40 to 54.

[0201] Embodiment 57. A first electronic device comprising:

[0202] An apparatus for performing the method according to any one of embodiments 40 to 54.

[0203] Embodiment 58. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first electronic device, the one or more programs comprising instructions for performing the following operations:

[0204] acquiring first image data regarding an input, wherein the first image data is acquired using one or more camera sensors of the first electronic device external to a second electronic device;

[0205] acquiring second image data regarding the input, wherein the second image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; and

[0206] An operation is performed based on the input in accordance with determining a group of one or more criteria to be satisfied based on the first image data and the second image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input is a touch input.

[0207] Embodiment 59. A first electronic device comprising:

[0208] one or more processors; and

[0209] A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations:

[0210] acquiring first image data regarding an input, wherein the first image data is acquired using one or more camera sensors of the first electronic device external to a second electronic device;

[0211] acquiring second image data regarding the input, wherein the second image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; and

[0212] An operation is performed based on the input in accordance with determining a group of one or more criteria to be satisfied based on the first image data and the second image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input is a touch input.

[0213] Embodiment 60. A first electronic device comprising:

[0214] means for acquiring first image data regarding an input, wherein the first image data is acquired using one or more camera sensors of the first electronic device external to a second electronic device;

[0215] means for acquiring second image data regarding the input, wherein the second image data is acquired using one or more camera sensors of the second electronic device, the first electronic device being different from the second electronic device; and

[0216] Means for performing an operation based on the input according to determining a group of one or more criteria that are satisfied based on the first image data and the second image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input is a touch input.

[0217] Embodiment 61. A method comprising:

[0218] causing the infrared light source of the first electronic device to emit infrared light;

[0219] acquiring image data regarding the input, wherein the image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; and

[0220] An operation is performed based on the input in accordance with determining based on the image data that a set of one or more criteria is satisfied, wherein the set of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

[0221] Embodiment 62. The method according to embodiment 61, further comprising:

[0222] Based on determining based on the image data that the set of one or more criteria is not satisfied, performing the operation based on the input is abandoned.

[0223] Embodiment 63. A method according to any one of Embodiments 61 to 62, wherein the input occurs at a location other than the location of the first electronic device and the location of the second electronic device.

[0224] Embodiment 64. A method according to any one of Embodiments 61 to 63, wherein the criterion includes a criterion that is satisfied when the input is a gesture.

[0225] Embodiment 65. A method according to Embodiment 64, wherein the criteria include criteria that are met when the input is a tap gesture.

[0226] Embodiment 66 The method of Embodiment 64, wherein the criteria include criteria that are satisfied when the input is a pinch gesture or an expand gesture.

[0227] Embodiment 67. A method according to Embodiment 64, wherein the criteria include criteria that are met when the input is a rotation gesture.

[0228] Embodiment 68. A method according to any one of Embodiments 61 to 67, wherein the criterion includes a criterion that is satisfied when the input corresponds to one or more fingers of a user.

[0229] Embodiment 69. A method as described in any one of Embodiments 61 to 68, wherein the determination of the group that satisfies one or more criteria is made by the second electronic device.

[0230] Embodiment 70. The method according to any one of embodiments 61 to 69, further comprising:

[0231] According to determining that the infrared light source of the first electronic device is blocked, a prompt is provided to move the first electronic device to a new position.

[0232] Embodiment 71. The method according to any one of embodiments 61 to 70, further comprising:

[0233] acquiring image data about the first electronic device using the one or more camera sensors of the second electronic device; and

[0234] determining a location of the first electronic device based on the image data regarding the first electronic device; and

[0235] A virtual keyboard is displayed based on the position of the first electronic device.

[0236] Embodiment 72. A system comprising:

[0237] one or more processors; and

[0238] One or more memories storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method described in any one of Embodiments 61 to 71.

[0239] Embodiment 73. A system according to embodiment 72, wherein the one or more memories

[0240] The system includes a first memory of the first electronic device and a second memory of the second electronic device.

[0241] Embodiment 74. A system according to any one of embodiments 72 to 73, wherein the one

[0242] The one or more memories include a third memory of a third electronic device different from the first electronic device and the second electronic device.

[0243] Embodiment 75. A method comprising:

[0244] At the first electronic device:

[0245] causing the infrared light source of the second electronic device to emit infrared light;

[0246] acquiring image data regarding the input, wherein the image data is acquired using one or more camera sensors of the first electronic device, the first electronic device being different from the second electronic device; and

[0247] An operation is performed based on the input in accordance with determining based on the image data that a set of one or more criteria is satisfied, wherein the set of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

[0248] Embodiment 76. The method according to embodiment 75, further comprising:

[0249] Based on determining based on the image data that the set of one or more criteria is not satisfied, performing the operation based on the input is abandoned.

[0250] Embodiment 77. A method according to any one of Embodiments 75 to 76, wherein the input occurs at a location other than the location of the first electronic device and the location of the second electronic device.

[0251] Embodiment 78. A method according to any one of Embodiments 75 to 77, wherein the criterion includes a criterion that is satisfied when the input is a gesture.

[0252] Embodiment 79. A method according to Embodiment 78, wherein the criteria include criteria that are met when the input is a tap gesture.

[0253] Embodiment 80 The method of Embodiment 78, wherein the criteria include criteria that are satisfied when the input is a pinch gesture or an expand gesture.

[0254] Embodiment 81. A method according to Embodiment 78, wherein the criteria include criteria that are met when the input is a rotation gesture.

[0255] Embodiment 82. A method according to any one of Embodiments 75 to 81, wherein the criterion includes a criterion that is satisfied when the input corresponds to one or more fingers of a user.

[0256] Embodiment 83. A method according to any one of Embodiments 75 to 82, wherein the determination of the group satisfying one or more criteria is made by the first electronic device.

[0257] Embodiment 84. The method according to any one of embodiments 75 to 83, further comprising:

[0258] According to determining that the infrared light source of the second electronic device is blocked, a prompt is provided to move the second electronic device to a new position.

[0259] Embodiment 85. The method according to any one of embodiments 75 to 84, further comprising:

[0260] acquiring image data about the second electronic device using the one or more camera sensors of the first electronic device; and

[0261] determining a location of the second electronic device based on the image data regarding the second electronic device; and

[0262] A virtual keyboard is displayed based on the position of the second electronic device.

[0263] Embodiment 86. A non-transitory computer-readable storage medium, the non-transitory computer-readable

[0264] The storage medium stores one or more programs configured to be executed by one or more processors of the first electronic device, wherein the one or more programs include instructions for executing the method described in any one of Embodiments 75 to 85.

[0265] Embodiment 87. A first electronic device comprising:

[0266] one or more processors; and

[0267] A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method described in any one of Embodiments 75 to 85.

[0268] Embodiment 88. A first electronic device comprising:

[0269] An apparatus for performing the method according to any one of embodiments 75 to 85.

[0270] Embodiment 89. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first electronic device, the one or more programs comprising instructions for performing the following operations:

[0271] causing the infrared light source of the second electronic device to emit infrared light;

[0272] acquiring image data regarding the input, wherein the image data is acquired using one or more camera sensors of the first electronic device, the first electronic device being different from the second electronic device; and

[0273] An operation is performed based on the input in accordance with determining based on the image data that a set of one or more criteria is satisfied, wherein the set of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

[0274] Embodiment 90. A first electronic device comprising:

[0275] one or more processors; and

[0276] A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations:

[0277] causing the infrared light source of the second electronic device to emit infrared light;

[0278] acquiring image data regarding the input, wherein the image data is acquired using one or more camera sensors of the first electronic device, the first electronic device being different from the second electronic device; and

[0279] An operation is performed based on the input in accordance with determining based on the image data that a set of one or more criteria is satisfied, wherein the set of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

[0280] Embodiment 91. A first electronic device comprising:

[0281] means for causing the infrared light source of the second electronic device to emit infrared light;

[0282] means for acquiring image data regarding an input, wherein the image data is acquired using one or more camera sensors of the first electronic device, the first electronic device being different from the second electronic device; and

[0283] Means for performing an operation based on the input in accordance with determining based on the image data that a set of one or more criteria is satisfied, wherein the set of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

[0284] Embodiment 92. A method comprising:

[0285] At the first electronic device:

[0286] causing the infrared light source of the first electronic device to emit infrared light;

[0287] acquiring image data regarding the input, wherein the image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; and

[0288] An operation is performed based on the input in accordance with determining based on the image data that a set of one or more criteria is satisfied, wherein the set of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

[0289] Embodiment 93. The method according to embodiment 92, further comprising:

[0290] Based on determining based on the image data that the set of one or more criteria is not satisfied, performing the operation based on the input is abandoned.

[0291] Embodiment 94. A method according to any one of Embodiments 92 to 93, wherein the input occurs at a location other than the location of the first electronic device and the location of the second electronic device.

[0292] Embodiment 95. A method according to any one of Embodiments 92 to 94, wherein the criterion includes a criterion that is satisfied when the input is a gesture.

[0293] Embodiment 96. A method according to Embodiment 95, wherein the criteria include criteria that are met when the input is a tap gesture.

[0294] Embodiment 97. A method according to Embodiment 95, wherein the criteria include criteria that are met when the input is a pinch gesture or an expand gesture.

[0295] Embodiment 98. A method according to Embodiment 95, wherein the criteria include criteria that are met when the input is a rotation gesture.

[0296] Embodiment 99. A method according to any one of Embodiments 92 to 98, wherein the criterion includes a criterion that is satisfied when the input corresponds to one or more fingers of a user.

[0297] Embodiment 100. A method as described in any one of Embodiments 92 to 99, wherein the determination of the group that satisfies one or more criteria is made by the first electronic device.

[0298] Embodiment 101. The method according to any one of embodiments 92 to 100, further comprising:

[0299] According to determining that the infrared light source of the first electronic device is blocked, a prompt is provided to move the first electronic device to a new position.

[0300] Embodiment 102. The method according to any one of embodiments 92 to 101, further comprising:

[0301] acquiring image data about the first electronic device using the one or more camera sensors of the second electronic device; and

[0302] determining a location of the first electronic device based on the image data regarding the first electronic device; and

[0303] A virtual keyboard is displayed based on the position of the first electronic device.

[0304] Embodiment 103. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first electronic device, wherein the one or more programs include instructions for executing a method according to any one of Embodiments 92 to 102.

[0305] Embodiment 104. A first electronic device comprising:

[0306] one or more processors; and

[0307] A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of embodiments 92 to 102.

[0308] Embodiment 105. A first electronic device comprising:

[0309] An apparatus for performing the method according to any one of embodiments 92 to 102.

[0310] Embodiment 106. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first electronic device, the one or more programs comprising instructions for performing the following operations:

[0311] causing the infrared light source of the first electronic device to emit infrared light;

[0312] acquiring image data regarding the input, wherein the image data is acquired using one or more camera sensors of a second electronic device, the second electronic device being different from the first electronic device; and

[0313] An operation is performed based on the input in accordance with determining based on the image data that a set of one or more criteria is satisfied, wherein the set of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

[0314] Embodiment 107. A first electronic device comprising:

[0315] one or more processors; and

[0316] A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations:

[0317] causing the infrared light source of the first electronic device to emit infrared light;

[0318] acquiring image data regarding the input, wherein the image data is acquired using one or more camera sensors of a second electronic device, the second electronic device being different from the first electronic device; and

[0319] An operation is performed based on the input in accordance with determining based on the image data that a set of one or more criteria is satisfied, wherein the set of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

[0320] Embodiment 108. A first electronic device comprising:

[0321] means for causing the infrared light source of the first electronic device to emit infrared light;

[0322] means for acquiring image data regarding the input, wherein the image data is acquired using one or more camera sensors of a second electronic device, the second electronic device being different from the first electronic device; and

[0323] Means for performing an operation based on the input in accordance with determining based on the image data that a set of one or more criteria is satisfied, wherein the set of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

Claims

1. A method for input detection, comprising: At the first electronic device: acquiring first image data regarding an input, wherein the first image data is acquired using one or more camera sensors of the first electronic device that is external to the second electronic device, wherein the input occurs at a location other than a location of the first electronic device and a location of the second electronic device; acquiring second image data regarding the input, wherein the second image data is acquired using one or more camera sensors of the second electronic device, the first electronic device being different from the second electronic device; as well as An operation is performed based on the input in accordance with determining a group of one or more criteria to be satisfied based on the first image data and the second image data, wherein the group of one or more criteria includes a criterion that is satisfied when the input is a touch input.

2. The method according to claim 1, further comprising: Based on determining that the set of one or more criteria is not satisfied based on the first image data and the second image data, performing the operation based on the input is abandoned.

3. The method according to any one of claims 1 to 2, further comprising: Before acquiring the first image data and the second image data regarding the input, determining whether there is a suitable environment for receiving the touch input based on the orientation data from the second electronic device.

4. The method of any one of claims 1 to 2, wherein the criteria comprises criteria that are satisfied when the input is a gesture. The method of claim 4 , wherein the criteria comprises a criteria that is satisfied when the input is a tap gesture. The method of claim 4 , wherein the criteria include criteria that are satisfied when the input is a pinch gesture or an expand gesture. The method of claim 4 , wherein the criteria include a criteria that is satisfied when the input is a rotation gesture.

8. The method of any one of claims 1 to 2, wherein the criterion comprises a criterion that is satisfied when the input corresponds to one or more fingers of a user.

9. The method of any one of claims 1 to 2, wherein the determination of the group satisfying one or more criteria is made by the first electronic device.

10. The method according to any one of claims 1 to 2, wherein the first image data and the second image data correspond to the same time point or time period.

11. The method according to any one of claims 1 to 2, further comprising: Based on determining that the one or more camera sensors of the second electronic device are blocked, a prompt is provided to move the second electronic device to a new location.

12. The method according to any one of claims 1 to 2, further comprising: acquiring image data about the second electronic device using the one or more camera sensors of the first electronic device; as well as determining a location of the second electronic device based on the image data regarding the second electronic device; as well as A virtual keyboard is displayed based on the position of the second electronic device.

13. The method according to any one of claims 1 to 2, wherein: The input includes movement of the first object and the second object, and At least one of the first image data and the second image data does not include information about both the first object and the second object.

14. The method of claim 13, wherein the first image data and the second image data in combination provide sufficient data for determining whether the set of one or more criteria is satisfied.

15. A system for input detection, comprising: one or more processors; as well as One or more memories storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 14.

16. The system of claim 15, wherein the one or more memories include a first memory of the first electronic device and a second memory of the second electronic device.

17. The system of any one of claims 15 to 16, wherein the one or more memories include a third memory of a third electronic device different from the first electronic device and the second electronic device.

18. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 14.

19. A method for input detection, comprising: causing the infrared light source of the first electronic device to emit infrared light; acquiring image data regarding the input, wherein the image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; as well as An operation is performed based on the input in accordance with determining that a set of one or more criteria is satisfied based on the image data, wherein the set of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

20. The method according to claim 19, further comprising: Based on determining based on the image data that the set of one or more criteria is not satisfied, performing the operation based on the input is abandoned.

21. The method of any one of claims 19 to 20, wherein the input occurs at a location other than the location of the first electronic device and the location of the second electronic device.

22. A method according to any one of claims 19 to 20, wherein the criteria comprises criteria which are satisfied when the input is a gesture.

23. The method of claim 22, wherein the criteria comprises a criteria that is satisfied when the input is a tap gesture.

24. The method of claim 22, wherein the criteria comprises a criteria that is satisfied when the input is a pinch gesture or an expand gesture.

25. The method of claim 22, wherein the criteria comprises a criteria that is satisfied when the input is a rotation gesture.

26. A method according to any one of claims 19 to 20, wherein the criteria comprises a criterion which is satisfied when the input corresponds to one or more fingers of a user.

27. The method of any one of claims 19 to 20, wherein the determination of the group satisfying one or more criteria is made by the second electronic device.

28. The method according to any one of claims 19 to 20, further comprising: According to determining that the infrared light source of the first electronic device is blocked, a prompt is provided to move the first electronic device to a new position.

29. The method according to any one of claims 19 to 20, further comprising: acquiring image data about the first electronic device using the one or more camera sensors of the second electronic device; as well as determining a location of the first electronic device based on the image data regarding the first electronic device; as well as A virtual keyboard is displayed based on the position of the first electronic device.

30. A system for input detection, comprising: one or more processors; as well as One or more memories storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method according to any one of claims 19 to 29.

31. The system of claim 30, wherein the one or more memories include a first memory of the first electronic device and a second memory of the second electronic device.

32. The system of any one of claims 30 to 31, wherein the one or more memories include a third memory of a third electronic device different from the first electronic device and the second electronic device.

33. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs comprising instructions for executing the method according to any one of claims 19 to 29.

34. A method for input detection, comprising: At the first electronic device: causing the infrared light source of the first electronic device to emit infrared light; acquiring image data regarding the input, wherein the image data is acquired using one or more camera sensors of a second electronic device, the first electronic device being different from the second electronic device; as well as An operation is performed based on the input in accordance with determining that a set of one or more criteria is satisfied based on the image data, wherein the set of one or more criteria includes a criterion that is satisfied when the input contacts the infrared light.

35. The method of claim 34, further comprising: Based on determining based on the image data that the set of one or more criteria is not satisfied, performing the operation based on the input is abandoned.

36. The method of any one of claims 34 to 35, wherein the input occurs at a location other than the location of the first electronic device and the location of the second electronic device.

37. A method according to any one of claims 34 to 35, wherein the criteria comprises criteria which are satisfied when the input is a gesture.

38. The method of claim 37, wherein the criteria comprises a criteria that is satisfied when the input is a tap gesture.

39. The method of claim 37, wherein the criteria comprises criteria that are satisfied when the input is a pinch gesture or an expand gesture.

40. The method of claim 37, wherein the criteria comprises a criteria that is satisfied when the input is a rotation gesture.

41. A method according to any one of claims 34 to 35, wherein the criteria comprises a criterion which is satisfied when the input corresponds to one or more fingers of a user.

42. The method of any one of claims 34 to 35, wherein the determination of the group satisfying one or more criteria is made by the first electronic device.

43. The method according to any one of claims 34 to 35, further comprising: According to determining that the infrared light source of the first electronic device is blocked, a prompt is provided to move the first electronic device to a new position.

44. The method according to any one of claims 34 to 35, further comprising: acquiring image data about the first electronic device using the one or more camera sensors of the second electronic device; as well as determining a location of the first electronic device based on the image data regarding the first electronic device; as well as A virtual keyboard is displayed based on the position of the first electronic device.

45. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first electronic device, the one or more programs comprising instructions for executing the method according to any one of claims 34 to 44.

46. ​​A first electronic device, comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 34 to 44.

47. A first electronic device, comprising: Apparatus for carrying out the method according to any one of claims 34 to 44.

Images