Devices and methods for controlling electronic devices or systems using physical objects.
By detecting the movement of physical objects in the physical environment and using sensors to recognize user input, the problem that physical objects cannot control electronic devices in existing technologies has been solved, enabling a more flexible user interaction method.
Patent Information
- Application Number
- CN202310036468.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2023-01-06
- Filing Date
- 2023-01-10
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-01-10
AI Technical Summary
Existing technologies struggle to effectively utilize physical objects to control the functions of electronic devices without the need for communication or electronic circuits, resulting in limitations on user interaction methods.
By detecting the movement of physical objects in the physical environment and using sensors such as image sensors and microphones, user input can be identified and corresponding actions can be executed to control electronic devices.
It provides a way to control device functions using a physical object that does not need to communicate directly with the electronic device, enhancing the flexibility and convenience of user interaction.
Smart Images

Figure CN116414332B_ABST
Abstract
Description
[0001] Cross-references to related applications
[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 266,626, filed January 10, 2022, and U.S. Patent Application No. 18 / 151,206, filed January 6, 2023, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field
[0003] This disclosure relates in its entirety to a computer system that detects changes in a physical environment and performs actions at a corresponding computer system based on the detected changes in the physical environment. Background Technology
[0004] Users can interact with the computer system using one or more input devices (e.g., mouse, touch sensor, proximity sensor, image sensor, etc.). The computer system's sensors can be used to capture images of the physical environment surrounding the computer system (e.g., the computer system's operating environment). Summary of the Invention
[0005] Some of the examples described in this disclosure are performed at a first electronic device (e.g., a computer system) that communicates with a display and one or more input devices. In some examples, the first electronic device detects changes in the physical environment caused by movement of one or more physical objects in the physical environment that indicate user input. In some examples, the first electronic device performs a first action either at itself or at a second electronic device that communicates with it, based on changes in the physical environment caused by movement of one or more physical objects in the physical environment. Attached Figure Description
[0006] Figures 1A to 1B An exemplary block diagram of the architecture of a system or device according to some examples of this disclosure is shown.
[0007] Figures 2A to 2E Examples of electronic devices that adjust television volume based on the movement of one or more physical objects, according to some examples of this disclosure, are shown.
[0008] Figures 3A to 3D Examples of electronic devices are shown, according to some examples of this disclosure, that detect the movement of one or more physical objects to toggle on / off a system function at an electronic device or a second electronic device.
[0009] Figures 4A to 4BAn example is shown where an electronic device performs an action at the electronic device itself or at a second electronic device in response to the detection of a text-based input command.
[0010] Figures 5A to 5C Examples of electronic devices that detect one or more physical objects in a physical environment to construct a musical sequence (e.g., a sequence of musical notes) are shown, according to some examples of this disclosure.
[0011] Figure 6 A process 600 for performing an action at a first electronic device or a second electronic device, according to some examples of this disclosure, is shown. Detailed Implementation
[0012] The accompanying drawings, which form part of the description, are referenced in the following description and illustrate, by way of example, specific examples of optional implementations. It should be understood that other examples of the present disclosure may be used and structural changes may be made optionally without departing from the scope of the disclosure.
[0013] The terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms “an,” “a,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “includes,” “including,” “comprises,” and / or “comprising” as used in this specification specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.
[0014] Furthermore, although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the scope of the various examples described, a first device may be referred to as a second device, and similarly, a second device may be referred to as a first device. Both the first device and the second device are devices, but they are generally not the same device.
[0015] As described herein, depending on the context, the term "if" optionally means "when," "at," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if it is determined..." or "if [the stated condition or event] is optionally interpreted as "in response to determining..." or "in response to detecting [the stated condition or event]."
[0016] A physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic devices. A physical environment can include physical features such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with a physical environment through senses such as sight, touch, hearing, taste, and smell. Conversely, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment can include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. XR environments are generally referred to herein as computer-generated environments. In the case of an XR system, a subset of a person's physical motion or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the XR system are adjusted in a manner consistent with at least one physical law. For example, an XR system can detect head movement and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. For example, an XR system can detect movement of electronic devices (e.g., mobile phones, tablets, laptops, etc.) that present the XR environment, and in response, adjust the graphical content and sound field presented to the user in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust the characteristics of the graphical content in the XR environment in response to representations of physical motion (e.g., voice commands).
[0017] Many different types of electronic systems enable people to sense and / or interact with a variety of XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped like lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have an integrated opaque display and one or more speakers. Alternatively, head-mounted systems may be configured to receive external opaque displays (e.g., smartphones). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In some implementations, transparent or translucent displays can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology, which projects graphic images onto the human retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.
[0018] Figures 1A to 1B Exemplary block diagrams of the architecture of a system or device according to some examples of this disclosure are shown. Figure 1A As shown, device 100 optionally includes various sensors (e.g., one or more hand tracking sensors 102, one or more position sensors 104, one or more image sensors 106, one or more touch-sensitive surfaces 109, one or more motion and / or orientation sensors 110, one or more eye-tracking sensors 112, one or more microphones 113 or other audio sensors, etc.), one or more display generating components 114, one or more speakers 116, one or more processors 118, one or more memories 120, and / or communication circuitry 122. One or more communication buses 108 are optionally used for communication between the aforementioned components of device 100. In some examples, device 100 is a portable device, such as a mobile phone, smartphone, tablet computer, laptop computer, auxiliary device for communicating with another device, etc.
[0019] Communication circuit 122 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, intranets, wired and / or wireless networks, cellular networks, and wireless local area networks (LANs)). Communication circuit 122 optionally includes circuitry for using near-field communication (NFC) and / or short-range communication such as... The circuit used for communication.
[0020] Processor 118 optionally includes one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors (DSPs). In some examples, memory 120 is a non-transitory computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage device) that stores computer-readable instructions and / or programs configured to be executed by processor 118 to perform the techniques, processes, and / or methods described below. In some examples, memory 120 includes more than one non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium can be any medium (e.g., excluding signals) that can tangibly contain or store computer-executable instructions for use by or in connection with instruction execution systems, apparatuses, and devices. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media can include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include hard disks, optical discs based on CD, DVD, or Blu-ray technology, and persistent solid-state storage such as flash memory and solid-state drives.
[0021] Display generating component 114 optionally includes a single display (e.g., a liquid crystal display (LCD), an organic light-emitting diode (OLED), or other type of display). In some examples, display generating component 114 includes multiple displays. In some examples, display generating component 114 includes a display having a touch-sensitive surface (e.g., a touchscreen), a projector, a holographic projector, a retinal projector, etc.
[0022] In some examples, device 100 includes a touch-sensitive surface 109 configured to receive user input (touch and / or proximity input) (such as tap input and swipe input or other gestures). In some examples, display generating component 114 and touch-sensitive surface 109 together form a touch-sensitive display (e.g., a touchscreen integrated with device 100 or a touchscreen external to device 100 that communicates with device 100).
[0023] Image sensor 106 optionally includes one or more visible light image sensors (such as charge-coupled device (CCD) sensors) and / or complementary metal-oxide-semiconductor (CMOS) sensors operable to acquire images of physical objects from the real-world environment. Image sensor 106 optionally includes one or more infrared (IR) or near-infrared (NIR) sensors (such as passive or active IR or NIR sensors) for detecting infrared or near-infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. Image sensor 106 optionally includes one or more cameras configured to capture movement of physical objects in the real-world environment. Image sensor 106 optionally includes one or more depth sensors configured to detect the distance between the physical object and device 100. In some examples, information from one or more depth sensors may allow the device to identify objects in the real-world environment and distinguish them from other objects in the real-world environment. In some examples, one or more depth sensors may allow the device to determine the texture and / or shape of objects in the real-world environment.
[0024] In some examples, device 100 uses a combination of a CCD sensor, an event camera, and a depth sensor to detect the physical environment surrounding device 100. In some examples, image sensor 106 includes a first image sensor and a second image sensor. The first and second image sensors work together and are optionally configured to capture different information about physical objects in the real-world environment. In some examples, the first image sensor is a visible light image sensor, and the second image sensor is a depth sensor. In some examples, device 100 uses image sensor 106 to detect the position and orientation of device 100 and / or display generation component 114 in the real-world environment. For example, device 100 uses image sensor 106 to track the position and orientation of display generation component 114 relative to one or more stationary objects in the real-world environment.
[0025] In some examples, device 100 optionally includes a hand-tracking sensor 102 and / or an eye-tracking sensor 112. The hand-tracking sensor 102 is configured to track the position / location of a user's hand and / or fingers relative to a computer-generated environment, relative to a display generation component 114, and / or relative to another coordinate system, and / or the movement of the user's hand and / or fingers. The eye-tracking sensor 112 is configured to track the position and movement of a user's gaze (more generally, eyes, face, and / or head) relative to the real-world or computer-generated environment and / or relative to the display generation component 114. In some examples, the hand-tracking sensor 102 and / or the eye-tracking sensor 112 are implemented together with the display generation component 114 (e.g., in the same device). In some examples, the hand-tracking sensor 102 and / or the eye-tracking sensor 112 are implemented separately from the display generation component 114 (e.g., in different devices).
[0026] In some examples, the hand tracking sensor 102 uses an image sensor 106 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that captures three-dimensional information from the real world, including one or more hands. In some examples, the hand can be distinguished with sufficient resolution to differentiate the fingers and their corresponding positions. In some examples, one or more image sensors 106 are positioned relative to the user to define the image sensor's field of view and interaction space, in which the finger / hand positions, orientations, and / or movements captured by the image sensors are used as input (e.g., to differentiate from the user's resting hand or other hands of other people in the real-world environment). The advantage of tracking fingers / hands used for input (e.g., gestures) may be that using fingers / hands for input provides an input method that does not require the user to touch or hold the input device, and using an image sensor allows tracking without requiring the user to wear beacons or sensors on their hand / finger.
[0027] In some examples, the eye-tracking sensor 112 includes one or more eye-tracking cameras (e.g., IR cameras) and / or an illumination source (e.g., an IR light source / LED) that emits light toward the user's eyes. The eye-tracking camera may be pointed at the user's eyes to receive reflected light from the light source directly or indirectly from the eyes. In some examples, both eyes are tracked separately by the respective eye-tracking camera and illumination source, and gaze can be determined by tracking both eyes. In some examples, one eye (e.g., the dominant eye) is tracked by the respective eye-tracking camera / illumination source.
[0028] Device 100 optionally includes microphone 113 or other audio sensors. Device 100 uses microphone 113 to detect sound from the user and / or the user's real-world environment. In some examples, microphone 113 includes a microphone array that optionally operates together (e.g., to identify ambient noise or locate sound sources in the space of the real-world environment).
[0029] Device 100 optionally includes a position sensor 104 configured to detect the position of device 100 and / or display the position of generating component 114. For example, position sensor 104 optionally includes a GPS receiver that receives data from one or more satellites and allows device 100 to determine the absolute position of the device in the physical world.
[0030] Device 100 optionally includes a motion and / or orientation sensor 110 configured to detect the orientation and / or movement of device 100 and / or display generating component 114. For example, device 100 uses orientation sensor 110 to track changes in the position and / or orientation (e.g., relative to a physical object in a real-world environment) of device 100 and / or display generating component 114. Orientation sensor 110 optionally includes one or more gyroscopes, one or more accelerometers, and / or one or more inertial measurement units (IMUs).
[0031] It should be understood that Figure 1A The architecture described is exemplary, but system / device 100 is not limited to it. Figure 1A The components and configurations. For example, a device / system may include fewer, additional, or other components in the same or different configurations. In some examples, such as Figure 1BAs shown, system / device 150 can be divided among multiple devices. For example, first device 160 optionally includes a processor 118A, one or more memories 120A, and communication circuitry 122A that optionally communicates via a communication bus 108A. Second device 170 (e.g., corresponding to device 100) optionally includes various sensors (e.g., one or more hand tracking sensors 102, one or more position sensors 104, one or more image sensors 106, one or more touch-sensitive surfaces 109, one or more motion and / or orientation sensors 110, one or more eye-tracking sensors 112, one or more microphones 113 or other audio sensors, etc.), one or more display generation components 114, one or more speakers 116, one or more processors 118B, one or more memories 120B, and / or communication circuitry 122B. One or more communication buses 108B are optionally used for communication between the aforementioned components of device 170. The details of the components of devices 160 and 170 are similar to those of the corresponding components discussed above with respect to device 100, and will not be repeated here for the sake of brevity. The first device 160 and the second device 170 may optionally communicate via a wired or wireless connection between the two devices (e.g., via communication circuits 122A to 122B).
[0032] As described herein, in some examples, physical objects, even those without communication or electronic circuitry, can be used to enable electronic devices to perform actions (e.g., control functionality of the electronic device). For example, movement of a physical object or the result of such movement (or more generally, changes in a physical environment device caused by the movement of one or more physical objects in the physical environment) can indicate user input, which can be detected by sensors of the electronic device (e.g., image sensors, proximity sensors, etc.). In some examples, the electronic device performs a first action based on changes in the physical environment caused by the movement of one or more physical objects in the physical environment. In some examples, a configuration process (also referred to herein as a “registration process”) can be used to associate physical objects, one or more boundaries, and / or associate actions before physical objects are available to enable the electronic device to perform actions. Thus, the use of physical objects can be used to implement the functionality of electronic devices, providing convenient or alternative ways to control the device using physical objects that are not otherwise communicatively coupled to the electronic device and / or do not directly interact with the user interface displayed to the user of the electronic device.
[0033] Figures 2A to 2E Examples of electronic devices that adjust television volume based on the movement of one or more physical objects, according to some examples of this disclosure, are shown. Figure 2A The diagram shows a component that generates a display (e.g., Figure 1A and Figure 1B The display generation component 114 and one or more image sensors 214 (e.g., Figure 1A and Figure 1B Electronic device 201 comprising one or more sensors 102 to 112 as described herein. In some examples, one or more image sensors 214 include one or more external sensors (e.g., sensors facing outwards from the user) for detecting objects in the physical environment 222 of electronic device 101 and movement of the user's hand and / or include one or more internal sensors (e.g., sensors facing inwards from the user's face) for detecting the user's attention (e.g., gaze). In some examples, physical objects in the physical environment are rendered by electronic device 201 (e.g., displayed using display generation component 114).
[0034] In some examples, physical objects in the physical environment 222 of electronic device 201 are presented by electronic device 201 via a transparent or semi-transparent display (e.g., the display of device 201 does not obscure the user's view of objects in the physical environment, thus allowing these objects to be visible). For example, as... Figure 2A As shown, electronic device 201 is presenting objects in physical environment 222 via transparent or semi-transparent display generating component 114, including television 206 currently playing television program A at first playback volume level 210, table 204, and physical objects 202 (e.g., pen, coffee cup, flashlight, comb, computer, telephone, tablet, etc.).
[0035] In some examples, electronic device 201 receives input from a user corresponding to a request to control the electronic device (e.g., electronic device 201 or a second electronic device, such as television 206) using a physical object in the physical environment 222 (as described below in process 600 and...). Figures 3A to 3D (As described in more detail below). The user is requesting that the physical object used to control the electronic device optionally exclude electronic circuitry and / or communication circuitry that communicates with electronic device 201 (or a second electronic device (e.g., television 206)). In some examples, the input received by electronic device 201 for controlling the corresponding electronic device with a physical object includes predetermined gesture / movement input (as will be described in more detail below). Figures 3A to 3D (as described in the text), voice input, and / or eye contact input. For example, in Figure 2A In this context, electronic device 201 is receiving voice input 224 stating, "I want to control the volume of my TV using the object I am currently looking at," while simultaneously detecting that the user's attention is currently directed at object A202. In some examples, in response to electronic device 201 detecting via one or more image sensors 214 that the user's gaze (e.g., represented by an ellipse 208 with a plus sign) is currently directed at object A202, electronic device 201 detects... Figure 2A The user's attention is directed to object A 202 (e.g., Figure 2A (As shown). Additionally or alternatively, in response to electronic device 201 detecting via one or more image sensors 214 and / or orientation sensors (e.g., orientation sensor 110) that the user's head is currently oriented toward object A 202, electronic device 201 optionally detects that the user's attention is directed toward object A 202. Additionally or alternatively, electronic device 201 optionally receives voice input stating "I want to use object A to control the volume of my TV," such that electronic device 201 can identify object A 202 without requiring the user to gaze at object A 202. It should be understood that, although Figure 2A An example is described where electronic device 201 is receiving input to control television 206 (e.g., a second electronic device), but electronic device 201 may optionally receive input for controlling system functions of electronic device 201 with object A 202, such as controlling the brightness of electronic device 201, the playback position of media item being played by electronic device 201, etc. It should also be understood that the invention described herein is not limited to... Figures 2A to 2E The example described herein is not intended to be applied to other devices / systems that communicate with electronic device 201, such as Internet of Things (IoT) devices / systems.
[0036] In response to the electronic device 201 receiving Figure 2A The inputs include voice input 224 and eye contact (e.g., represented by ellipse 208). The electronic device 201 optionally initiates a process for controlling the volume of the television 206 using object A 202, such as... Figure 2B As shown. In Figure 2B In the middle, electronic device 201 is displaying a voice-assisted user interface 216 (e.g., system functions of electronic device 201 that allow hands-free operation of electronic device 201), the voice-assisted user interface including Figure 2A The provided voice input 224 includes instructions 212 and 218 indicating the next operation when configuring object A 202 to control the volume of television 206. As shown, in some examples, this instruction is a text-based command prompting the user to provide / select the movement boundaries of object A 202 for volume control. It should be understood that, although... Figure 2B This shows the response of electronic device 201 receiving Figure 2A The electronic device 201 displays a voice-assisted user interface 216 based on the input described herein, but the electronic device 201 can also initiate a process for configuring object A 202 to control the volume of the television 206 without displaying the voice-assisted user interface 216 (e.g., transmitting the remaining operations / instructions of the configuration process via the speaker of the electronic device 201 without displaying the voice-assisted user interface 216). Although Figure 2BThe diagram illustrates a visual representation of interaction with a voice-assisted user interface, but it should be understood that the interaction may use only audio input (e.g., using a speaker and microphone), partially use audio input (optionally, using other inputs / outputs such as visual notifications, haptic feedback, etc.), and / or not use audio (e.g., using text-driven or other user interfaces).
[0037] In some examples, device 201 can initiate a registration process where the user can indicate the movement boundaries of object A 202. For example... Figure 2B As shown, in some examples, configuring object A 202 to control the volume of television 206 includes indicating the minimum and / or maximum movement boundaries of object A 202 (optionally relative to another physical object in physical environment 222, such as table 204, as described below). The minimum and maximum movement boundaries may represent endpoints of a slider-type control, where object A 202 acts as a slider. In some examples, during and / or after successful registration, device 201 may provide visual indicators (e.g., a virtual line between the minimum and maximum movement boundaries, virtual indications of the maximum and minimum boundaries, a virtual representation of scale markings along the virtual line, and / or text labels for "maximum" and "minimum" at the maximum movement boundary, etc.) to demonstrate the movement path of the slider-type control.
[0038] In some examples, device 201 may prompt the user to select a minimum movement boundary and a maximum movement boundary. For example, device 201 may display a first text prompt (and / or audio prompt) asking the user to move object A 202 to the minimum movement boundary position. After completion, device 201 may display a second text prompt (and / or audio prompt) asking the user to move object A 202 to the maximum movement boundary position.
[0039] As will be described in more detail below, after successfully configuring object A 202 to control the volume of television 206 (e.g., after successful registration), when electronic device 201 detects that object A 202 is at the minimum movement boundary, electronic device 201 optionally sets the volume of television 206 to 0%, and when electronic device 201 detects that object A 202 is at the maximum movement boundary, it optionally sets the volume of television 206 to 100%. In some examples, electronic device 201 detects eye input from the user to indicate the minimum and maximum movement boundaries of object A 202. For example, in Figure 2C In the middle, on the electronic device 201 display Figure 2BAfter instruction 218 in the electronic device 201 indicates to the user the minimum and maximum movement boundaries of object A 202, the electronic device 201 detects the first input provided by the user, which includes a gaze directed at the left front corner of table 204 (represented by ellipse 208) to select the left front corner of table 204 as the minimum movement boundary of object A 202, and a second input, which includes a gaze directed at the right front corner of table 204 (represented by ellipse 208) to select the right front corner of table 204 as the maximum movement boundary of object A 202.
[0040] It should be understood that when electronic device 201 detects eye input from a user for selecting the minimum and maximum movement boundaries of object A 202, electronic device 201 may also additionally or alternatively detect other forms of input indicating the minimum and maximum movement boundaries of object A 202, such as detecting, via one or more image sensors 214, that the user taps the left and right front corners of table 204 to indicate the minimum and maximum movement boundaries of object A 202 respectively, or detecting that the user performs a predefined gesture (e.g., a pinch or tap gesture without touching or contacting electronic device 201) when the user's attention (e.g., eye contact represented by ellipse 208) is directed to the left and right front portions of table 204 to indicate the minimum and maximum movement boundaries of object A 202.
[0041] In some examples, electronic device 201 detects changes in the physical environment 222 via one or more image sensors 214. For example, in Figure 2D In the middle, upon receiving the selection Figure 2C After receiving one or more inputs of the minimum and maximum movement boundaries, the electronic device 201 now detects that the user's hand (e.g., represented by hexagon 226) has moved within range of one or more sensors 214 and is grasping (e.g., holding) an object A 202 in the physical environment 222. Figure 2C In contrast, where the hand (e.g., represented by hexagon 226) is not within the range of one or more image sensors 214 and is not in the grasping object A 202. As a result of (e.g., in response to) one or more image sensors 214 capturing one or more images of the user (or a part of the user, such as the user's hand) while the user is interacting with the physical environment 222, the electronic device 201 optionally detects Figure 2D The user's hand (e.g., represented by hexagon 226) is grasping object A202. Figure 2E In the process, when the user's hand (e.g., represented by hexagon 226) continues to grasp / hold object A 202, the electronic device 201 detects that the user's hand (e.g., represented by hexagon 226) has moved object A 202 away from the object. Figure 2D The position in the physical environment 222 shown (e.g., the front middle part of table 204) is moved to Figure 2E The position shown in the physical environment 222 (e.g., slightly offset to the left of the right front portion of table 204) corresponds to the maximum movement boundary of object A 202 (as previously mentioned). Figure 2C The above)).
[0042] In some examples, the volume level of television 206 is adjusted (e.g., incrementally / gradually) as the position of object A 202 changes relative to the selected minimum and maximum movement boundaries of object A 202. For example, as Figure 2E As shown, in response to electronic device 201 detecting that object A202 is located at an equidistant (e.g., midpoint) distance from the physical environment 222 between the minimum and maximum movement boundaries of object A202, Figure 2D As shown, the device moves to a position within physical environment 222 that is 5 units (e.g., inches, feet, yards, etc.) from the maximum movement boundary (e.g., the front right portion of table 204) of object A 202 and 95 units (e.g., inches, feet, yards, etc.) from the minimum movement boundary (e.g., the front left portion of table 204) of object A 202, and the electronic device 201 adjusts the volume of television 206 from... Figure 2D The volume level 210 shown (e.g., corresponding to 50% of the volume level of TV 206) has been modified / updated to... Figure 2E The volume level 220 shown is (for example, corresponding to 95% of the volume level of television 206). When electronic device 201 is detecting object A 202 from... Figure 2D Move the position in the physical environment 222 shown to Figure 2E In the physical environment 222 shown, the electronic device 201 optionally updates the volume of the television 206 continuously from 50% to 95% at 1%, 2%, 3%, 5%, or 10% (or any other suitable amount)—as opposed to updating the volume of the television 206 only after the electronic device 201 detects that object A 202 is no longer moving in the physical environment 222. Despite Figure 2D and Figure 2E The diagram illustrates adjusting the volume of television 206 based on the horizontal movement of object A 202. However, it should be understood that the volume of television 206 can also be adjusted based on the vertical movement of object A 202 within the physical environment 222 (e.g., if the minimum and maximum movement boundaries of object A 202 are located at different positions along the Y-axis (e.g., vertical position relative to the user's viewpoint) and along the X-axis (e.g., horizontal position relative to the user's viewpoint), such as... Figures 2A to 2EAs shown. Furthermore, in some examples where the minimum and maximum movement boundaries of object A 202 are located at different positions along the Y-axis, when object A 202 moves in the X and / or Z directions while the Y position of object A 202 does not change, electronic device 201 optionally abandons updating the volume of television 206 according to the movement of object A 202.
[0043] When object A 202 is detected to have moved... Figure 2E Following the location shown in the physical environment, electronic device 201, in response to detecting further movement of object A 202 in physical environment 222, optionally updates / modifies the volume of television 206. For example, after electronic device 201 updates the volume of television 206 to volume level 220 (e.g., corresponding to 95% of the volume level at television 206), electronic device 201 optionally detects further movement of the user's hand toward object A 202. If the detected further movement of object A 202 includes object A 202 moving toward... Figure 2E If the physical position of object A 202 is moved 1, 2, 3, 4, or 5 units (e.g., inches, feet, yards) to the right, electronic device 101 optionally increases the volume of television 206 by 1%, 2%, 3%, 4%, or 5% respectively to 96%, 97%, 98%, 99%, or 100% of the volume level of television 206. Conversely, if further movement of object A 202 involves object A 202 moving to the right... Figure 2E If the physical position of the object A 202 shown is moved to the left by 5, 10, 15, 20 or 25 units (e.g., inches, feet, yards), the electronic device 201 may optionally reduce the volume of the television 206 by 5%, 10%, 15%, 20% or 25% to 90%, 80%, 75%, 70% or 65% of the volume level of the television 206.
[0044] although Figures 2A to 2E The examples primarily describe a linear scale between the minimum and maximum movement boundaries and the corresponding volume adjustment; however, it should be understood that in some examples, a non-linear correspondence between position and volume can be implemented. Furthermore, although... Figures 2A to 2E This section primarily describes slider-type functionalities (e.g., volume adjustment, brightness adjustment, etc.) defined using maximum and minimum movement boundaries. As described herein, other types of functionality can be implemented (e.g., using a rotating knob with a handle, where the handle optionally indicates the knob's position, as referenced below). Figures 3A to 3D (Description of toggle switches, etc.). Additionally... Figures 2A to 2E This section primarily describes the configuration process initiated using audio and / or eye input. As discussed in this article, other inputs can be used to initiate the configuration process.
[0045] In some examples, the functionality can be a switch-type functionality (e.g., a toggle switch). In some examples, the electronic device detects the movement of one or more physical objects to toggle the system function on / off at the electronic device (e.g., the device that is detecting the movement of one or more physical objects) and / or toggle the system function on / off at a second electronic device (e.g., the device that is optionally not detecting the movement of one or more physical objects), as will now be discussed regarding Figures 3A to 3D As stated above.
[0046] Figure 3A An electronic device 301 is shown, including a display generation component 114 and one or more image sensors 314. Figure 3A The electronic device 301, display generating component 114, and one or more image sensors 314 shown are optionally connected to, respectively Figures 2A to 2E The electronic device 201, display generating component 114, and one or more image sensors 214 depicted and shown are similar to or the same as those in the illustration. Also, Figure 3A As shown, electronic device 301 is presenting objects in physical environment 322 of electronic device 301 via transparent or semi-transparent display generating component 114, including currently turned-off television 306, table 304 and physical objects 302 (e.g., pen, coffee cup, flashlight, comb, computer, telephone, tablet, etc.).
[0047] In some examples, electronic device 301 initiates a process of controlling system functions of electronic device 301 or another electronic device having a physical object in physical environment 322 in response to audio and / or gaze input, as referenced in [reference]. Figures 2A to 2EThe configuration process may be initiated using, for example, eye contact and / or audio input. In some examples, in response to electronic device 301 detecting that a physical object is moving in a predetermined manner, electronic device 301 initiates a process for controlling a first system function of electronic device 301 or a first system function of a second electronic device (e.g., television 306) with the physical object in physical environment 322. For example, when electronic device 301 detects that a physical object has moved in a zigzag motion from left to right and then from left to right, electronic device 301 optionally initiates a process for controlling the on / off state of television 306 with the physical object. Conversely, when electronic device 301 detects that a physical object has moved in a zigzag motion from right to left and then from right to left, electronic device 301 optionally initiates a process for controlling the on / off state of a lighting system (e.g., different functions) communicating with electronic device 301 with the physical object (or optionally toggling the "Do Not Disturb" mode of electronic device 301 on / off). It should be understood that the above-described processes for movement and initiating control associated with functionality are examples, and other functionalities can be controlled by different movements. For example, electronic device 301 is optionally configured to detect a plurality of predetermined movements, which optionally correspond to requests to configure different system functions of electronic device 301 or a second electronic device (e.g., a device / system communicating with electronic device 301). In some examples, a specific predetermined movement of any object can be detected to initiate a process of control using that object, and the associated functionality can be selected as part of the initiated process (e.g., using audio and / or eye input to indicate the associated functionality), rather than being determined based on one of a plurality of predetermined movements. Figure 3B In this example, when the television 306 is currently off, the electronic device 301 detects that object A 302 is moving in a zigzag motion from left to right and then from left to right again. In this example, the zigzag motion corresponds to a predetermined movement required by the electronic device 301 to initiate a process of controlling the on / off state of the television 306 with object A 302. In response to the electronic device 301 detecting the zigzag motion of object A 302 in the physical environment 322, the electronic device 301 optionally initiates a process configured to control the on / off state of the television 306 with object A 302. In some examples, the process of configuring the on / off state of the television 306 with object A 302 includes generating an audio output 308 that indicates to the user that the next operation in the configuration process is to select the movement boundary of object A 302, such as... Figure 3BAs shown. The selected movement boundary is a position in the physical environment 322. If the electronic device 301 detects that the movement of object A 302 exceeds the selected movement boundary, this position optionally causes the electronic device 301 to turn on the television 306, and if the electronic device 301 does not detect that the movement of object A 302 exceeds the selected movement boundary, this position optionally causes the electronic device 301 to turn off the television 306 (or if the television 306 is already turned off, it remains off), as will be described in more detail below. It should be understood that the audio output 308 can also be extended or replaced by other types of information (e.g., haptic information, visual / text notifications, etc.).
[0048] In some examples, electronic device 301 is used in conjunction with previously... Figures 2A to 2E The method described in [the text] receives input for selecting the movement boundary in a similar manner. For example, in [the text] Figure 3C In the process of generating electronic device 301 Figure 3B After the audio output 308 prompts the user to indicate the moving boundary of object A302, the electronic device 301 detects input including the user's gaze (e.g., represented by an ellipse 310 with a plus sign) pointing towards the front center portion of table 304, to select the front center portion of table 304 as the moving boundary of object A302. It should be understood that when the electronic device 301 detects gaze input from the user for selecting the moving boundary of object A301, the electronic device 301 may also additionally or alternatively detect other forms of input for selecting the moving boundary of object A302, such as detecting, via one or more image sensors 314, a tapping of the user's hand on the front center portion of table 304 to select the front center portion of table 304 as the moving boundary of object A302, or detecting that the user performs a predefined "air" gesture (e.g., a pinch or tap gesture without touching or contacting the electronic device 301) when the user's attention (e.g., gaze represented by ellipse 310) is directed towards the front center portion of table 304, to select the front center portion of table 304 as the moving boundary of object A302.
[0049] Figure 3D The diagram illustrates the state transition using object A 302 (TV 306 is turned on) after the configuration process. Figure 3D In the process, electronic device 301 detects that the user's hand (e.g., represented by hexagon 326) has moved object A 302 from the front right portion of table 304 to the front left portion of table 304, which exceeds the... Figure 3C The selected movement boundary (front middle portion of table 304). In response to electronic device 301 detecting that object A 302 has moved beyond... Figure 3CUpon selecting the movement boundary, electronic device 301 transmits a signal to television 306 to turn on television 306, and in response to television 306 receiving the transmitted signal from electronic device 301, television 306 is turned on (e.g., ...). Figure 3D (As shown). Although in Figure 3D In the middle, the television 306 responds to the electronic device 301 detecting that the movement of object A 302 exceeds the limit. Figure 3C Select the movement boundary and from such Figure 3C The state shown is changed from being off to being as follows Figure 3D The indicated on state, but it should be understood that if electronic device 301 detects that the movement of object A 302 does not exceed [a certain threshold], [the on state will be affected]. Figure 3C If the selected moving boundary is used, then the TV 306 will optionally remain as follows: Figure 3C The off state is shown (the opposite of the on state). In some examples, after transmitting a signal to the television 306 to turn it on, the electronic device 301 detects further movement of the object A 302, and in response, the electronic device 301 modifies the on / off state of the television 306 in a similar manner as described above. In some examples, a hysteresis may be applied to avoid switching states when the object approaches a movement boundary. In some examples, the object A 302 must move more than a threshold distance beyond the movement boundary in order to switch states.
[0050] In some examples, the electronic device performs an action at itself or at a second electronic device (e.g., a television, an IoT system, or other system / device communicating with the electronic device) in response to the detection of a written or drawn input command (e.g., a text-based and / or image-based input command), as will be discussed later. Figure 4A and Figure 4B As stated above.
[0051] Figure 4A An electronic device 401 is shown, including a display generation component 114 and one or more image sensors 414. The electronic device 401, the display generation component 114, and the one or more image sensors 414 are optionally integrated with previously described... Figures 2A to 3D The electronic devices 201 and / or 301, display generating components 114 and / or one or more image sensors 214 and / or 314 described herein are the same as or similar to those described herein. Figure 4AAs shown, the physical environment 422 of the electronic device 401 includes an annotable object 402 (e.g., a diary, a piece of paper, a notebook, etc.), a writing device 404 (e.g., a pen, pencil, marker, or any other type of object capable of annotating the annotable object 402), and optionally a television 406 currently playing media content at volume level 410. As described herein, in some examples, the writing device may be used to physically inscribe ink on the annotable object (e.g., graphite on a pencil on paper, ink on a pen / marker in a notebook, etc.). In some examples, text-based or image-based input commands enable the user not only to control the electronic device but also to maintain a physical "written record" of all settings of the user interface elements (e.g., on the annotable object), which the user can retain and transmit for future reference or use.
[0052] exist Figure 4A In this case, electronic device 401 has not yet detected / received a text-based or image-based input command because writing device 404 has not yet annotated the annotable object 402 to include a text-based or image-based input command. Figure 4BIn this text, the writing device 404 has annotated the annotable object 402 to include text-based or image-based input commands corresponding to a request to adjust the volume level of the television 406. Specifically, the text-based or image-based input commands include a slider track (indicated by dashed lines). The “-” and “+” symbols indicate the minimum and maximum ends of the slider control (e.g., the area near the “-” symbol corresponds to a lower volume level compared to the area near the “+” symbol). The text-based or image-based input commands also include a slider bar represented by the “|” symbol corresponding to the desired volume level of the television 405 (e.g., indicating slider type functionality), and its relative placement along the slider track can indicate the volume level between the maximum and minimum ends. In some examples, the user can adjust the volume by erasing the “|” symbol at a first position and redrawing the “|” symbol at a second, different position along the slider track. The text-based or image-based input commands also include the character / word “television volume” indicating that the text-based input command is associated with modifying the volume of the television 406 (e.g., a device / system that communicates with and is visible to the electronic device 401). In some examples, elements of the text-based or image-based input commands (e.g., identification of drawn sliders, maximum and minimum boundary positions, and corresponding functionality) can be confirmed as part of the registration process before using text-based or image-based input commands to control device actions. In some examples, text and / or audio prompts can be used to prompt the user to input, clarify, and / or confirm some or all of the information used for registration before using text-based or image-based input commands for control. For example, prompts can be given for uninputted aspects of text-based or image-based commands (e.g., instructing the user to add "+" and "-" symbols to indicate maximum and minimum points) or to clarify the semantic meaning of aspects of text-based or image-based commands (e.g., instructing the user to redraw the position of the "+" symbol when the system cannot distinguish the user's drawing of the "+" and "-" symbols).
[0053] After the writing device 404 annotates the annotable object 402 to include a text-based or image-based input command for controlling the volume of the television 406 (e.g., in response to the annotation of the annotable object), the electronic device 401 (e.g., via one or more image sensors 414) detects the text-based or image-based input command and transmits a signal to the television 406 to control and / or modify the volume level of the television 406 to correspond to the volume level indicated by the "|" symbol in the text-based or image-based input command (as previously described). The television 406 receives the signal transmitted by the electronic device 401 and adjusts the volume of the television 406 from... Figure 4A The volume level shown is reduced to 420. Figure 4BThe volume level 432 shown corresponds to the volume level indicated by the "|" symbol in a text-based or image-based input command. It should be understood that the above example is one possible example of text-based or image-based input commands that can be detected by the electronic device 401, but without departing from the scope of this disclosure, the electronic device 401 may also detect other text-based or image-based input commands. For example, the electronic device 401 may optionally be configured to detect text-based or image-based input commands for modifying the lighting of an IoT-based lighting system communicating with the electronic device 401, or for modifying the functionality of the electronic device 401, and other possibilities.
[0054] although Figures 4A to 4B This primarily describes text-based or image-based input commands with slider-type functionality (e.g., volume adjustment) defined using maximum and minimum boundaries. However, as described herein, other types of functionality can be implemented (e.g., knobs that indicate rotation using changes in the drawn position of a knob pointer, toggle switches, non-linear sliders, etc.). Additionally, although... Figures 4A to 4B The functionality of a slider type, including a slider track and a slider bar, is shown, but other user interface implementations of the slider control can be implemented. For example, the control could be a rectangle with a minimum and maximum value represented at opposite ends, and a shadow within the rectangle relative to the minimum end of the rectangle could represent a volume level.
[0055] In addition, although Figures 4A to 4B The description primarily focuses on text-based or image-based input commands, where volume adjustment is achieved through the ink application of the "|" symbol within the text-based or image-based input command. However, it should be understood that in some examples, a virtual slider control can be used to achieve volume adjustment. For instance, in response to detecting a slider control as part of a text-based or image-based input command (e.g., a slider track and slider bar annotated on an annotable object), the system can render a virtual slider control. In some examples, the virtual slider control may include a virtual slider track and a virtual slider bar that can obscure the ink application of the slider control to the text-based or image-based input command. In some examples, when rendering the virtual slider control, the system can detect the position of the virtual slider bar being manipulated using the writing device 404 by touching the top of the writing device 404 to the virtual slider bar and dragging the virtual slider bar using the writing device 404 (where touch and movement are detected, for example, using an image sensor of device 401).
[0056] In some examples, electronic devices detect one or more physical objects in the physical environment to construct a musical sequence (e.g., a sequence of musical notes), as will be referred to now. Figures 5A to 5C Described. In some examples, physical objects can be used to implement a Musical Instrument Digital Interface (MIDI). In Figure 5A In the process, electronic device 501 is transmitting data via one or more image sensors 515 (e.g., with...). Figure 4A and Figure 4B One or more image sensors 414 (identical or similar) detect three physical objects in the physical environment 522 of the electronic device 501: object A 502, object B 504, and object C 506 (presented by the electronic device 501 via a transparent or semi-transparent display generation component 114). In some examples, one or more physical objects in the physical environment 522 correspond to one or more musical notes. Although not in Figures 5A to 5C Described in the context of, but the object is optionally first configured to represent a note and the surface on which the object is placed is optionally first configured to represent the timing and frequency boundaries of the MIDI interface (e.g., time is assigned on a first axis (e.g., the x-axis) and frequency is assigned on a second axis (e.g., the y-axis). For example, in Figure 5A In response to the detection of object A 502, object B 504, and object C 506 by electronic device 501, electronic device 501 constructs / generates music sequence 508. For example... Figure 5A As shown, the music sequence 508 includes a first note 510 corresponding to object A 502, a second note 512 corresponding to object B 504, and a third note 514 corresponding to object C 506.
[0057] In some examples, the notes corresponding to the corresponding physical objects in physical environment 522 are based on one or more properties of that physical object. For example, the first note 510 may optionally be different from the second note 512 and / or the third note 514 because object A 502—the physical object corresponding to the first note 510—is different from object B 504 and / or object C 506 (the physical objects corresponding to the second note 512 and the third note 514, respectively) (for example, object A 502 is a first-type physical object, and object B 504 and / or object C 506 are not first-type physical objects).
[0058] In some examples, the notes corresponding to the corresponding physical objects in physical environment 522 are based on the visual appearance of the corresponding physical object and / or the symbols included at the corresponding physical object (as a supplement or alternative to using xy position to determine the frequency and time of each note). For example, object A 502 optionally corresponds to the first note 510 because electronics 501 determines that the first symbol is located at object A 502 (e.g., including a drawing of the first instrument or note) and / or because electronics determine that object A 502 is a first color (e.g., yellow, red, blue, etc.). Similarly, for similar reasons as described above, object B 504 and object C 506 optionally correspond to the second note 512 and the third note 514, respectively. Thus, in some examples, the first note 510, the second note 512, and the third note 514 optionally correspond to different / dissimilar notes or the same note (e.g., depending on the position along the y-axis).
[0059] In some examples, the timing of the notes in the music sequence 508 is based on the position of the corresponding physical objects in the physical environment 522 of the electronic device 501. For example, note 510 may optionally appear before note 512, and note 512 may appear before note 514 in the music sequence 508, because the electronic device 501 detects object A 502 before object B 504 and object B 504 before object C 506 in a left-to-right manner. It should be understood that if the electronic device 501 were to detect physical objects in the physical environment in a right-to-left manner instead, the notes in the music sequence 508 would optionally be arranged such that the third note 514 is first, the second note 512 is second, and the first note 510 is third.
[0060] In some examples, the time distance between notes in music sequence 508 is based on (e.g., corresponding to) the distance between corresponding physical objects in physical environment 522. For example, note 512 may optionally appear 0.1 seconds, 0.5 seconds, 1 second, or 2 seconds after note 510 in music sequence 508 because electronic device 501 has determined that object B 504 will be 1 inch, 2 inches, 4 inches, or 6 inches to the right of object A 502. Similarly, the time between notes 512 and 514 in music sequence 508 may optionally be longer than the time between notes 510 and 512 in music sequence 508 because the physical distance between object A 502 and object B 504 is less than the physical distance between object B 504 and object C 506 (e.g., ...). Figure 5A (As shown). In some examples, placing two (or more) objects at the same point along the time axis (e.g., the x-axis) can be used to implement chords.
[0061] In some examples, in response to an electronic device 501 constructing a music sequence 508, the electronic device 501 plays the music sequence 508 or causes the music sequence 508 to be played at a second electronic device (e.g., a smart speaker, television, computer, etc.) communicating with the electronic device 501.
[0062] In some examples, electronic device 501 constructs a new music sequence in response to detecting movement of a physical object within its physical environment 522. For example, in Figure 5B In the process, the electronic device detects that the user's hand (e.g., represented by hexagon 516) has grasped / picked up object B 504 in the physical environment 522. After the electronic device detects that the user's hand (e.g., represented by hexagon 516) has grasped / picked up object B 504, in Figure 5C In the middle, electronic device 501 detected object B 504 from Figure 5B The location in the physical environment 522 indicated in the middle is to Figure 5C The movement of the object B 504 within the physical environment 522 is indicated by the movement of the object B 504. In response to the electronic device 501 detecting the movement of the object B 504 within the physical environment 522, the electronic device 501 constructs a new music sequence 518, which differs from the previously described sequence in a similar manner. Figure 5A The music sequence shown is 508.
[0063] Figure 6 Exemplary processes 600 for performing actions at a first or second electronic device according to some examples of this disclosure are shown. Process 600 is optionally performed at electronic devices such as device 100, device 201, 301, 401, and / or 501. Some operations in process 600 are optionally combined and / or optionally omitted. In some examples, the order of some operations in process 600 is optionally changed.
[0064] In some examples, process 600 is performed at a first electronic device that communicates with a display and one or more input devices. In some examples, the first electronic device detects (602) a change in the physical environment caused by the movement of one or more physical objects in the physical environment that indicate user input. In some examples, the first electronic device performs (604) a first action at itself or at a second electronic device that communicates with the first electronic device based on the change in the physical environment caused by the movement of one or more physical objects in the physical environment.
[0065] In some examples, detecting changes in the physical environment of the first electronic device includes detecting the movement of a first physical object among one or more physical objects relative to a second physical object in the physical environment of the first electronic device that is different from one or more physical objects.
[0066] In some examples, the distance between the first physical object and the second physical object is less than a threshold distance before the first electronic device detects a change in the physical environment. In some examples, after detecting a change in the physical environment including movement of the first object relative to the second object, the first electronic device performs a first action at either the first or second electronic device based on determining that the distance between the first and second objects is greater than the threshold distance. In some examples, after detecting a change in the physical environment including movement of the first object relative to the second object, the first electronic device abandons performing the first action at either the first or second electronic device based on determining that the distance between the first and second objects does not exceed the threshold distance.
[0067] In some examples, after performing a first action at a first electronic device or a second electronic device and when the distance between the first physical object and the second physical object is greater than a threshold distance, the first electronic device detects movement of the first physical object relative to the second physical object. In some examples, in response to detecting movement of the first physical object relative to the second physical object, based on determining that the distance between the first physical object and the second physical object is less than a threshold distance, the first electronic device performs a second action different from the first action at the first electronic device or the second electronic device. In some examples, in response to detecting movement of the first physical object relative to the second physical object, based on determining that the distance between the first physical object and the second physical object is greater than a threshold distance, the first electronic device abandons performing the second action at the first electronic device or the second electronic device.
[0068] In some examples, performing the first action includes toggling a first function of either the first or second electronic device to enable or disable the first function of either the first or second electronic device. In some examples, the one or more physical objects do not include electronic circuitry. In some examples, changes in the physical environment are captured via one or more input devices of the first electronic device. In some examples, the one or more input devices include cameras that capture the movement of one or more physical objects over multiple time periods.
[0069] In some examples, performing the first action includes displaying a user interface object at a first electronic device. In some examples, in response to selecting a user interface object at the first electronic device, the first electronic device performs a second action, different from the first action, at either a first electronic device or a second electronic device based on the selection of the user interface object at the first electronic device.
[0070] In some examples, the first physical object and the second physical object are at a first distance before the first electronic device detects a change in the physical environment. In some examples, after detecting a change in the physical environment, including movement of the first object relative to the second object, the first electronic device performs a first action at either the first or second electronic device based on determining that the first object and the second physical object are at a second distance different from the first distance; and based on determining that the first object and the second physical object are at a third distance different from the first distance, the first electronic device performs a second action at either the first or second electronic device, different from the first action.
[0071] In some examples, after a first action is performed at a first electronic device or a second electronic device and when the first physical object is a second distance away from the second physical object, the first electronic device detects movement of the first physical object relative to the second physical object. In some examples, in response to detecting movement of the first physical object relative to the second physical object: based on determining that the first object and the second physical object are a fourth distance different from the second distance, the first electronic device performs a third action at the first or second electronic device; and based on determining that the first object and the second physical object are a fifth distance different from the second distance, the first electronic device performs a fourth action different from the third action at the first or second electronic device.
[0072] In some examples, performing a first action includes modifying a first function of a first electronic device or a second electronic device by a first amount; performing a second action includes modifying the first function of the first electronic device or the second electronic device by a second amount different from the first amount; performing a third action includes modifying the first function of the first electronic device or the second electronic device by a third amount different from the second amount; and performing a fourth action includes modifying the first function of the first electronic device or the second electronic device by a fourth amount different from the third amount (e.g., adjusting the volume or brightness level based on the position of a first physical object acting as a slider).
[0073] In some examples, before detecting movement of the first physical object relative to the second physical object (e.g., before performing a first action based on changes in the physical environment caused by the movement of one or more physical objects), the first electronic device receives input via one or more input devices for configuring the first action to be performed based on the one or more physical objects. In some examples, in response to receiving the input, the first electronic device initiates a process for configuring the first action to be performed based on the movement of the one or more physical objects.
[0074] In some examples, after detecting a change in the physical environment including the movement of one or more physical objects, the first electronic device abandons the action of responding to the detection of the movement of one or more physical objects, based on the determination that the first electronic device is not yet configured to perform a first action in response to the movement of one or more physical objects.
[0075] In some examples, the input for configuring the first action to be performed based on one or more physical objects includes the user's gaze directed at one or more physical objects by a first electronic device for a threshold amount of time. In some examples, the input for configuring the first action to be performed based on one or more physical objects includes a voice command (e.g., audio input) directed at one or more physical objects. In some examples, the input for configuring the first action to be performed based on a first physical object and a second physical object includes a predetermined gesture directed at one of the one or more physical objects.
[0076] In some examples, one or more physical objects include multiple physical objects and performing the first action includes constructing a note sequence based on the positions of the multiple physical objects, wherein the frequency of each note is determined based on the corresponding position of one of the multiple physical objects in the physical environment (e.g., constructing a first note based on determining a first position of a first physical object in the physical environment; and constructing a second note different from the first note based on determining a first position of the first physical object in the physical environment).
[0077] In some examples, after constructing the note sequence, the first electronic device detects movement of one or more physical objects among a plurality of physical objects. In some examples, in response to detecting movement of one or more physical objects among the plurality of physical objects, the first electronic device constructs a different note sequence (e.g., based on the updated positions of the plurality of physical objects).
[0078] In some examples, the timing of each note is determined based on the position of one of a number of physical objects in the physical environment (e.g., based on the position of the multiple objects along an axis representing time).
[0079] In some examples, one or more physical objects include writing devices and / or annotable objects. In some examples, the first electronic device detects (602) changes in the physical environment caused by the movement of one or more physical objects indicating user input (e.g., detecting user interface elements written with ink on a physical surface by one or more physical objects in the physical environment, and detecting changes in the appearance of user interface elements caused by the movement of one or more physical objects indicating user input). In some examples, the first electronic device performs a first action at itself or at a second electronic device in communication with the first electronic device based on changes in the physical environment caused by the movement of one or more physical objects (e.g., adjusting the volume when the user draws a slider bar at different positions on paper along a drawn slider user interface element).
[0080] It should be understood that Figure 6 The specific order in which the operations described herein are presented is merely exemplary and is not intended to indicate that the described order is the only possible order in which the operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein.
[0081] The operation of the above-described process is optionally implemented by an electronic device including: one or more processors; a memory; and one or more programs stored in the memory. One or more programs stored in the memory and configured to be executed by one or more processors cause the processors to perform any of the operations described above. The operation of the above-described process is optionally implemented by storing one or more programs. One or more programs stored in a non-transitory computer-readable storage medium and configured to be executed by one or more processors of the electronic device cause the processors to perform any of the operations described above.
[0082] The above process can optionally be implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., as per [reference to...]). Figures 1A to 1B (as described above) or a dedicated chip. Furthermore, see the above references. Figure 6 The operation is optionally performed by Figures 1A to 1B The components described herein shall be used for implementation.
[0083] For purposes of explanation, the foregoing description has been given by reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible based on the teachings above. The examples were chosen and described to best illustrate the principles of the invention and its practical application, thereby enabling others skilled in the art to best utilize the invention with various modifications suitable for the particular intended use, as well as the various described examples.
Claims
1. A method for controlling an electronic device or system using a physical object, comprising: At the first electronic device that communicates with the display and one or more input devices: The movement of a first physical object relative to a second physical object from a first position in the physical environment of the first electronic device to a second position in the physical environment is detected, wherein the first physical object and the second physical object are less than a threshold distance before the movement of the first physical object relative to the second physical object is detected, wherein the movement indicates user input, and wherein the first physical object is different from the user's hand and different from a writing tool; as well as After detecting the movement of the first physical object relative to the second physical object: Based on the determination that the distance between the first physical object and the second physical object is greater than the threshold distance, a first action is performed at the first electronic device or at the second electronic device communicating with the first electronic device based on the movement of the first physical object relative to the second physical object; as well as If it is determined that the distance between the first physical object and the second physical object does not exceed the threshold distance, the first action is abandoned at the first electronic device or the second electronic device.
2. The method according to claim 1, further comprising: After the first action is performed at the first electronic device or the second electronic device, and when the distance between the first physical object and the second physical object is greater than the threshold distance, a second movement of the first physical object relative to the second physical object is detected; and In response to detecting a second movement of the first physical object relative to the second physical object: Based on the determination that the distance between the first physical object and the second physical object is less than the threshold distance, a second action different from the first action is performed at the first electronic device or the second electronic device. as well as If it is determined that the distance between the first physical object and the second physical object is greater than the threshold distance, the execution of the second action at the first electronic device or the second electronic device is abandoned.
3. The method of claim 2, wherein performing the first action includes switching on a first function of the first electronic device or the second electronic device, and performing the second action includes switching off the first function of the first electronic device or the second electronic device.
4. The method according to claim 1, wherein the first physical object does not include electronic circuits.
5. The method of claim 1, wherein the movement of the first physical object relative to the second physical object is captured via the one or more input devices of the first electronic device.
6. The method of claim 5, wherein the one or more input devices include a camera that captures the movement of the first physical object relative to the second physical object over multiple time periods.
7. The method of claim 1, wherein performing the first action includes displaying a user interface object at the first electronic device, the method further comprising: In response to selecting the user interface object at the first electronic device: Based on the selection of the user interface object at the first electronic device, a second action different from the first action is performed at either the first electronic device or the second electronic device.
8. The method of claim 1, wherein before the first electronic device detects the movement of the first physical object relative to the second physical object, the first physical object and the second physical object are at a first distance apart, the method further comprising: After detecting the movement of the first physical object relative to the second physical object: Based on determining that the first physical object and the second physical object are at a second distance that is different from the first distance, the first action is performed at the first electronic device or the second electronic device. as well as Based on determining that the first physical object and the second physical object are at a third distance different from the first distance, a second action different from the first action is performed at the first electronic device or the second electronic device.
9. The method of claim 8, further comprising: After the first action is performed at the first electronic device or the second electronic device, and when the first physical object is at the second distance from the second physical object, a second movement of the first physical object relative to the second physical object is detected; and In response to detecting the second movement of the first physical object relative to the second physical object: Based on determining a fourth distance between the first physical object and the second physical object that is different from the second distance, a third action is performed at the first electronic device or the second electronic device. as well as Based on determining a fifth distance between the first physical object and the second physical object that is different from the second distance, a fourth action, different from the third action, is performed at the first electronic device or the second electronic device.
10. The method according to claim 9, wherein: Performing the first action includes modifying the first function of the first electronic device or the second electronic device by a first amount. Performing the second action includes modifying the first function of the first electronic device or the second electronic device to a second amount that is different from the first amount. Performing the third action includes modifying the first function of the first electronic device or the second electronic device to a third quantity that is different from the second quantity, and Performing the fourth action includes modifying the first function of the first electronic device or the second electronic device to a fourth quantity that is different from the third quantity.
11. The method of claim 1, further comprising: Before performing the first action based on the movement of the first physical object relative to the second physical object, input for configuring the first action to be performed based on the first physical object is received via the one or more input devices; as well as In response to receiving the input: The process of initiating the first action to be executed is based on the movement of the first physical object relative to the second physical object.
12. The method according to claim 1, further comprising: Before the movement of the first physical object relative to the second physical object is detected: Based on the determination that the first electronic device has not been configured to perform the first action in response to movement of the first physical object relative to the second physical object, the execution of the first action in response to the detection of movement of the first physical object relative to the second physical object is abandoned.
13. The method of claim 11, wherein the input for configuring the first action to be performed based on the first physical object includes the gaze of a user of the first electronic device directed at the first physical object for a threshold amount of time.
14. The method of claim 1, wherein the first physical object is a first corresponding physical object among a plurality of physical objects, and wherein performing the first action comprises: A musical note sequence is constructed based on the positions of the plurality of physical objects, wherein the frequency of each note in the musical note sequence is determined based on the position of one of the plurality of physical objects in the physical environment.
15. The method of claim 14, further comprising: After constructing the note sequence, the movement of one or more physical objects among the plurality of physical objects is detected; as well as In response to detecting movement of one or more of the plurality of physical objects, different note sequences are constructed.
16. The method of claim 14, wherein the timing of each note in the note sequence is determined based on the corresponding position of one of the plurality of physical objects in the physical environment.
17. The method of claim 1, wherein the first physical object and the second physical object do not include electronic circuits.
18. A non-transitory computer-readable storage medium storing one or more programs, said one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform any one of the methods according to claims 1 to 17.
19. An electronic device comprising: monitor; as well as One or more processors, the one or more processors being configured to perform any one of the methods according to claims 1 to 17.
Citation Information
Patent Citations
Utilizing real world objects for user input
US20150309629A1
Mediated reality
WO2017009529A1
