Interacting with smart devices using a pointing controller

Abstract

A pointing controller worn by a user enables intuitive control of various connected smart devices. A user can indicate a device to be controlled by pointing toward the smart device (or a surrogate object) and performing a predetermined action to select the device for control. Once selected, the user can interact with the smart device by performing various gestures or interactions with one or more control elements integrated into the pointing controller. The smart device can provide feedback using visual or auditory indicators on the smart device or by sending control signals to a pointing controller or other device in proximity to the user that provides tactile, auditory, or visual feedback.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Provisional Application No. 62 / 807,094, filed on February 18, 2019, which is incorporated herein by reference.

[0002] The present disclosure relates to controlling smart devices, and more particularly to controlling interactions with smart devices using a pointing controller.

Background Art

[0003] Users have conventionally configured smart devices such as thermostats, speakers, and lighting systems through control on the device itself or through a user interface accessible on a smartphone application or web portal. Accessing these interfaces is not always convenient for the user and can be inefficient. Voice - controlled devices provide an additional level of control but may not be desirable in quiet situations or in noisy environments.

Summary of the Invention

Means for Solving the Problems

[0004] A method, a non-transitory computer-readable storage medium, and a tracking device control the interaction with a smart device using a pointing controller. Sensor data is obtained from a state sensing device of the pointing controller. The movement of a pointing vector is tracked through a three-dimensional space based on the sensor data and a stored arm model. An intersection between the pointing vector and coordinates in a three-dimensional space associated with the smart device is detected to place the smart device in a selected state. An augmented reality display device displays a virtual menu associated with the smart device. A control interaction between the virtual menu and the pointing controller associated therewith is detected when the smart device is in a selected state, and a command is generated based on the control interaction to control the operation of the smart device.

[0005] In one embodiment, the coordinates associated with the smart device comprise the physical location of the smart device. In another embodiment, the coordinates associated with the smart device comprise the location of an actual or virtual proxy device associated with the smart device.

[0006] In one embodiment, the step of detecting an intersection between the pointing vector and the coordinates associated with the smart device includes generating a pointing cone having a central axis aligned with the pointing vector, an origin proximate to the location of the pointing controller, and a radius that increases with the distance from the origin of the pointing vector. The intersection is detected in response to the pointing cone overlapping the coordinates associated with the smart device.

[0007] In one embodiment, the step of detecting a control interaction includes detecting activation of an inter-digit button of the pointing controller. Further, in one embodiment, the step of detecting a control interaction includes detecting an interaction with a slider control interface of the pointing controller, navigating between different menu items within a virtual menu in response to the interaction with the slider control interface, and selecting a menu item in response to detecting activation of an inter-digit button of the pointing controller. In another embodiment, the step of detecting a control interaction includes detecting a gesture performed using the pointing controller that indicates a control function of the smart device.

[0008] In one embodiment, the step of tracking movement of the pointing vector includes performing the tracking based on a camera integrated with the pointing controller.

[0009] In one embodiment, the step of tracking movement of the pointing vector includes detecting whether the pointing controller is indoors or outdoors and adjusting parameters of the arm model according to whether the pointing controller is indoors or outdoors. In another embodiment, the step of tracking movement of the pointing vector includes detecting whether the user of the pointing controller is sitting or standing and adjusting parameters of the arm model according to whether the user of the pointing controller is sitting or standing. In another embodiment, the step of tracking movement of the pointing vector includes detecting a fatigue level associated with the user of the pointing controller and adjusting parameters of the arm model according to the detected fatigue level.

[0010] In one embodiment, the step of tracking the movement of the pointing vector includes determining that coordinates in a three-dimensional space associated with the smart device exceed a threshold distance from the pointing controller, generating parameters of an arm model corresponding to the arm as if it were in an outstretched position, and tracking the movement based on the parameters of the arm model.

[0011] In one embodiment, the step of tracking the movement of the pointing vector includes determining that coordinates in a three-dimensional space associated with the smart device are below a threshold distance from the pointing controller, generating parameters of an arm model corresponding to the arm as if it were in a relaxed position near the body, and tracking the movement based on the parameters of the arm model.

[0012] In one embodiment, the tracking device recognizes the smart device as smart lighting and includes detecting a swiping gesture on the touch interface of the pointing controller, detecting a control interaction with the pointing controller, and generating a command including controlling the dimming of the lighting according to the direction of the swiping gesture.

[0013] Additional embodiments will be apparent to those skilled in the art. This specification also provides, for example, the following items. (Item 1) A method for controlling interaction with a smart device using a pointing controller, the method comprising: obtaining sensor data from a state sensing device of the pointing controller; tracking movement of a pointing vector through three-dimensional space based on the sensor data and a stored arm model; detecting an intersection between the pointing vector and coordinates in the three-dimensional space associated with the smart device to place the smart device in a selected state; causing an augmented reality display device to display a virtual menu associated with the smart device; detecting a control interaction between the pointing controller associated with the virtual menu when the smart device is in the selected state; generating a command for controlling the operation of the smart device based on the control interaction A method comprising. (Item 2) The method of item 1, wherein the coordinates associated with the smart device comprise a physical location of the smart device. (Item 3) The method of item 1, wherein the coordinates associated with the smart device comprise a location of an actual or virtual proxy device associated with the smart device. (Item 4) Detecting the intersection comprises: generating a pointing cone having a central axis aligned with the pointing vector, an origin proximate to the location of the pointing controller, and a radius that increases with the distance from the origin of the pointing vector; detecting the intersection with the pointing vector in response to the pointing cone overlapping the coordinates associated with the smart device The method of item 1, comprising. (Item 5) The method of item 1, wherein detecting the control interaction comprises detecting activation of an inter-digit button of the pointing controller. (Item 6) Detecting the control interaction comprises: detecting an interaction with a slider control interface of the pointing controller; navigating between different menu items within the virtual menu in response to the interaction with the slider control interface selecting a menu item in response to detecting activation of an inter-finger button of the pointing controller The method according to item 1, comprising: (Item 7) The method according to item 1, wherein detecting the control interaction includes detecting a gesture performed using the pointing controller indicating a control function of the smart device. (Item 8) The method according to item 1, wherein tracking the movement of the pointing vector includes performing the tracking based on a camera integrated with the pointing controller. (Item 9) Tracking the movement of the pointing vector includes: detecting whether the pointing controller is indoors or outdoors; and adjusting parameters of the arm model according to whether the pointing controller is indoors or outdoors The method according to item 1, comprising: (Item 10) Tracking the movement of the pointing vector includes: detecting whether the user of the pointing controller is sitting or standing; and adjusting parameters of the arm model according to whether the user of the pointing controller is sitting or standing The method according to item 1, comprising: (Item 11) Tracking the movement of the pointing vector includes: detecting a fatigue level associated with the user of the pointing controller; and adjusting parameters of the arm model according to the detected fatigue level The method according to item 1, comprising: (Item 12) Tracking the movement of the pointing vector includes: determining that the coordinates in the three-dimensional space associated with the smart device exceed a threshold distance from the pointing controller; generating parameters of the arm model corresponding to the arm in a position as if extended; and tracking the movement based on the parameters of the arm model The method according to item 1, comprising: (Item 13) Tracking the movement of the pointing vector includes: determining that the coordinates in the three-dimensional space associated with the smart device are below a threshold distance from the pointing controller; generating parameters of the arm model corresponding to the arm in a relaxed position near the body Tracking the movement based on the parameters of the wrist model and The method according to item 1, comprising: (Item 14) Further comprising recognizing the smart device as smart lighting, and Detecting the control interaction with the pointing controller includes detecting a swiping gesture on the touch interface of the pointing controller, Generating the command includes controlling the dimming of the lighting according to the direction of the swiping gesture, The method according to item 1. (Item 15) A non-transitory computer-readable storage medium storing instructions for controlling the interaction with a smart device using a pointing controller, wherein the instructions, when executed by one or more processors, cause the one or more processors to Obtain sensor data from the state sensing device of the pointing controller, Tracking the movement of the pointing vector through three-dimensional space based on the sensor data and the stored wrist model, Detecting the intersection between the pointing vector and the coordinates in the three-dimensional space associated with the smart device to set the smart device to a selected state, Causing an augmented reality display device to display a virtual menu associated with the smart device, When the smart device is in the selected state, detecting a control interaction between the virtual menu and the pointing controller associated therewith, Generating a command for controlling the operation of the smart device based on the control interaction A non-transitory computer-readable storage medium for performing the steps including. (Item 16) The non-transitory computer-readable storage medium according to item 15, wherein the coordinates associated with the smart device comprise the physical location of the smart device. (Item 17) The non-transitory computer-readable storage medium according to item 15, wherein the coordinates associated with the smart device comprise the location of an actual or virtual proxy device associated with the smart device. (Item 18) Detecting the intersection includes Generating a pointing cone having a central axis aligned with the pointing vector, an origin proximate to the location of the pointing controller, and a radius that increases with the distance from the origin of the pointing vector; Detecting the intersection with the pointing vector in response to the pointing cone overlapping the coordinates associated with the smart device; The non-transitory computer-readable storage medium according to item 15, comprising: (Item 19) The non-transitory computer-readable storage medium according to item 15, wherein detecting the control interaction includes detecting activation of an inter-finger button of the pointing controller. (Item 20) Detecting the control interaction includes: Detecting an interaction with a slider control interface of the pointing controller; Navigating between different menu items in the virtual menu in response to the interaction with the slider control interface; Selecting a menu item in response to detecting activation of an inter-finger button of the pointing controller. The non-transitory computer-readable storage medium according to item 15, comprising: (Item 21) The non-transitory computer-readable storage medium according to item 15, wherein detecting the control interaction includes detecting a gesture performed using the pointing controller indicating a control function of the smart device. (Item 22) Tracking movement of the pointing vector includes performing the tracking based on a camera integrated with the pointing controller. The non-transitory computer-readable storage medium according to item 15. (Item 23) Tracking movement of the pointing vector includes: Detecting whether the pointing controller is indoors or outdoors; Adjusting parameters of the arm model according to whether the pointing controller is indoors or outdoors. The non-transitory computer-readable storage medium according to item 15, comprising: (Item 24) Tracking movement of the pointing vector includes: Detecting whether the user of the pointing controller is sitting or standing; Adjusting the parameters of the arm model according to whether the user of the pointing controller is sitting or standing The non-transitory computer-readable storage medium according to item 15, comprising: (Item 25) Tracking the movement of the pointing vector includes detecting a fatigue level associated with the user of the pointing controller; and adjusting the parameters of the arm model according to the detected fatigue level. The non-transitory computer-readable storage medium according to item 15, comprising: (Item 26) Tracking the movement of the pointing vector includes determining that the coordinates in the three-dimensional space associated with the smart device exceed a threshold distance from the pointing controller; generating parameters of the arm model corresponding to the arm as if in an outstretched position; and tracking the movement based on the parameters of the arm model. The non-transitory computer-readable storage medium according to item 15, comprising: (Item 27) Tracking the movement of the pointing vector includes determining that the coordinates in the three-dimensional space associated with the smart device are less than a threshold distance from the pointing controller; generating parameters of the arm model corresponding to the arm as if in a relaxed position near the body; and tracking the movement based on the parameters of the arm model. The non-transitory computer-readable storage medium according to item 15, comprising: (Item 28) further comprising recognizing the smart device as smart lighting, wherein detecting the control interaction with the pointing controller includes detecting a swiping gesture on a touch interface of the pointing controller, and generating the command includes controlling the dimming of the lighting according to the direction of the swiping gesture. The non-transitory computer-readable storage medium according to item 15. (Item 29) A tracking device, comprising: one or more processors; and a non-transitory computer-readable storage medium storing instructions for controlling an interaction with a smart device using a pointing controller, the instructions, when executed by the one or more processors, cause the one or more processors to Obtaining sensor data from the state sensing device of the pointing controller; Tracking the movement of a pointing vector through three-dimensional space based on the sensor data and a stored arm model; Detecting an intersection between the pointing vector and coordinates in the three-dimensional space associated with the smart device to place the smart device in a selected state; Causing an augmented reality display device to display a virtual menu associated with the smart device; Detecting a control interaction between the pointing controller associated with the virtual menu when the smart device is in the selected state; Generating a command for controlling the operation of the smart device based on the control interaction; A non-transitory computer-readable storage medium for performing steps including; A tracking device comprising; (Item 30) The tracking device according to item 29, wherein the coordinates associated with the smart device comprise a physical location of the smart device. (Item 31) The tracking device according to item 29, wherein the coordinates associated with the smart device comprise a location of an actual or virtual proxy device associated with the smart device. (Item 32) Detecting the intersection comprises: Generating a pointing cone having a central axis aligned with the pointing vector, an origin proximate to the location of the pointing controller, and a radius that increases with the distance from the origin of the pointing vector; Detecting the intersection with the pointing vector in response to the pointing cone overlapping the coordinates associated with the smart device; The tracking device according to item 29, including. (Item 33) Detecting the control interaction includes detecting activation of an inter-digit button of the pointing controller, the tracking device according to item 29. (Item 34) Detecting the control interaction includes: Detecting an interaction with a slider control interface of the pointing controller; Navigating between different menu items in the virtual menu in response to the interaction with the slider control interface; Selecting a menu item in response to detecting activation of an inter-finger button of the pointing controller The tracking device according to item 29, comprising: (Item 35) The tracking device according to item 29, wherein detecting the control interaction includes detecting a gesture performed using the pointing controller indicating a control function of the smart device. (Item 36) The tracking device according to item 29, wherein tracking the movement of the pointing vector includes performing the tracking based on a camera integrated with the pointing controller. (Item 37) Tracking the movement of the pointing vector includes: Detecting whether the pointing controller is indoors or outdoors; Adjusting parameters of the arm model according to whether the pointing controller is indoors or outdoors; The tracking device according to item 29, comprising: (Item 38) Tracking the movement of the pointing vector includes: Detecting whether the user of the pointing controller is sitting or standing; Adjusting parameters of the arm model according to whether the user of the pointing controller is sitting or standing; The tracking device according to item 29, comprising: (Item 39) Tracking the movement of the pointing vector includes: Detecting a fatigue level associated with the user of the pointing controller; Adjusting parameters of the arm model according to the detected fatigue level; The tracking device according to item 29, comprising: (Item 40) Tracking the movement of the pointing vector includes: Determining that the coordinates in the three-dimensional space associated with the smart device exceed a threshold distance from the pointing controller; Generating parameters of the arm model corresponding to the arm in a position where the finger is extended; Tracking the movement based on the parameters of the arm model; The tracking device according to item 29, comprising: (Item 41) Tracking the movement of the pointing vector includes: Determining that the coordinates in the three-dimensional space associated with the smart device are below a threshold distance from the pointing controller; generating parameters of the arm model corresponding to the arm in a relaxed position adjacent to a body; tracking the movement based on parameters of the arm model; Item 30. The tracking device of item 29, comprising: (Item 42) further comprising recognizing the smart device as a smart light; detecting the control interaction with the pointing controller includes detecting a swiping gesture on a touch interface of the pointing controller; generating the command includes controlling dimming of the light in response to a direction of the swiping gesture; Item 29. The tracking device according to item 29.

Brief Description of the Drawings

[0014] The disclosed embodiments have other advantages and features, which will become more readily apparent from the following modes for carrying out the invention of the present invention and the appended claims when considered in conjunction with the accompanying drawings.

[0015] [Figure 1] Figure (or "FIG.") 1 illustrates an exemplary embodiment of a smart device control system.

[0016] [Figure 2] Figure 2 illustrates an exemplary embodiment of a pointing controller.

[0017] [Figure 3] Figure 3 illustrates an exemplary embodiment of a tracking device.

[0018] [Figure 4] Figure 4 illustrates an exemplary embodiment of a control processing module for processing interactions from a pointing controller.

[0019] [Figure 5] Figure 5 illustrates an exemplary embodiment of a process for controlling a smart device based on interactions using a pointing controller.

[0020] [Figure 6] Figure 6 illustrates an exemplary embodiment of a pointing controller with a form factor for gripping between adjacent fingers.

DETAILED DESCRIPTION OF THE INVENTION

[0021] The figures and the following description relate to preferred embodiments by way of illustration only. Note that from the following discussion, it will be readily recognized that alternative embodiments of the structures and methods disclosed herein can be employed as viable alternatives without departing from the principles claimed.

[0022] Here, several embodiments are referred to in detail, and their examples are illustrated in the accompanying figures. Note that whenever possible, similar or like reference numbers may be used in the figures and may indicate similar or like functionality. The figures depict embodiments of a system (or method) disclosed for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein can be employed without departing from the principles described herein.

[0023] A pointing controller worn by a user enables intuitive control of various connected smart devices. The user may indicate the device to be controlled by pointing in the direction of the smart device (or proxy object) and performing a predetermined action to select the device for control. Once selected, the user may interact with the smart device by performing various gestures or interacting with one or more control elements integrated into the pointing controller. The smart device may provide feedback by using visual or auditory indicators on the smart device or by sending control signals to a pointing controller or other device proximate to the user that provides tactile, auditory, or visual feedback.

[0024] FIG. 1 is a block diagram of a smart device control system 100 according to one embodiment. The smart device control system 100 includes a smart device 140, a tracking device 110, and a pointing controller 120 connected via a network 130. In alternative configurations, different and / or additional components may be included within the smart device control system 100.

[0025] The smart device 140 comprises a connectivity-enabled electronic device capable of performing one or more functions based on various configuration settings. Examples of the smart device 140 include, for example, a smart thermostat, a smart lock, a smart refrigerator, a smart speaker, a smart lighting controller, a smart medical device, or other devices. The smart device 140 generally comprises hardware including at least a processor, a storage medium, and a communication interface. Additionally, the smart device 140 may include one or more sensors for detecting various environmental conditions related to its operation. For example, a smart thermostat may include a temperature sensor and a humidity sensor. The smart device 140 generally provides one or more control outputs in response to direct user input, detected environmental conditions, detected events, or combinations thereof. For example, a smart thermostat may control a heating and / or cooling system to control the ambient temperature within a desired range. A smart lighting system may control turning a connected light bulb on or off, the color of the light output, or the on / off pattern associated with the light bulb. The smart device 140 may execute software or firmware that enables it to provide a certain level of automated control for intelligently predicting the desired actions of the user. The smart device 140 may further include an application programming interface (API) that enables other connected devices to provide command inputs to the smart device 140 or query status information or other information from the smart device 140. The API may be accessible via an application interface or a web interface of another network-connected device such as a smartphone, a remote controller, or a backend server.

[0026] The pointing controller 120 includes a control device that captures user gestures and interactions with control elements integrated into the pointing controller 120. In certain embodiments, the pointing controller 120 has a form factor that enables it to be worn on the hand, wrist, or forearm. For example, in one embodiment, the pointing controller 120 has a form factor that enables it to be held between two adjacent fingers, as illustrated in FIG. 6. In other embodiments, the pointing controller 120 may comprise a ring form factor that enables it to be worn on a single finger. In another embodiment, the pointing controller 120 may comprise a knuckle duster that is worn across multiple fingers. In yet other embodiments, the pointing controller 120 comprises a band that can be worn around the wrist or arm.

[0027] The pointing controller 120 includes various sensors for enabling position and orientation sensing of the pointing controller 120, and a control interface for receiving direct input from a user wearing the pointing controller 120. For example, the pointing controller 120 may capture human gestures such as pointing or waving of the hand, and may capture interactions with control elements on the pointing controller 120. Different gestures may be utilized to provide different control inputs to the smart device 140. Advantageously, the pointing controller 120 enables the user to interact with the smart device 140 in a natural manner, as will be described in further detail below.

[0028] The tracking device 110 comprises a computing device that operates in conjunction with the pointing controller 120 to process gestures and other interactions detected by the pointing controller 120 and generate control inputs for controlling the smart device 140. In one embodiment, the tracking device 110 interfaces with an API of the smart device 140 to provide the control inputs. For example, the tracking device 110 may receive position tracking data from a sensor of the pointing controller 120 and, based on the tracking data, determine the pointing direction of the pointing controller 120 and / or a particular gesture performed by a user wearing the pointing controller 120. The tracking device 110 may further obtain information about the location of the smart device 140 and / or other objects in the environment of the tracking controller 120. In addition, the tracking device 110 may receive control data indicative of user interactions with control elements on the pointing controller 120. The tracking device 110 then generates control outputs to control aspects of the smart device 140 based on the detected pointing direction, the location of the smart device 140 and / or other objects, and interactions with the control elements of the pointing controller 120.

[0029] In an embodiment, tracking device 110 comprises a smartphone, tablet, head-mounted display device, or other device that interfaces with pointing controller 120 and runs an application for interfacing with smart device 140, including a computing device that does not necessarily include a display. Alternatively, tracking device 110 may be integrated with pointing controller 120 in any of the form factors of pointing controller 120 described above.

[0030] In certain embodiments, the tracking device 110 may include a display system that presents digital content such as audio, images, video, or combinations thereof. Here, the tracking device 110 may comprise an augmented reality display device embodied as, for example, a head-mounted device having an integrated display or a separate display such as a smartphone or a tablet. In augmented reality applications, the tracking device 110 enables the presentation of information and / or virtual objects along with the view of a real-world viewer. This overlay may be implemented, for example, via a translucent display that enables a user to view a presentation rendered in parallel with the real-world view, a projection system that projects virtual objects or information onto the real-world view, or a camera feed that captures the real-world view, combines it with the overlaid presentation, and presents the combined view to the user via the display.

[0031] In an exemplary use case, the pointing controller 120 and the tracking device 110 may enable a user to interact with the smart device 140 using natural and intuitive movements. For example, the user may point at the smart device 140 and perform a predetermined interaction with the pointing controller 120 to activate a function of the smart device 140, such as turning on the lighting, setting the thermostat, changing the volume of the speaker, or other functions. In other embodiments, the user may control the smart device by pointing at a proxy device that may include a real-world object or a virtual object associated with the smart device 140. Here, for example, the user may point at a radiator associated with a smart thermostat and cause the smart thermostat to control the temperature of the radiator. In another example, the user may point at an incandescent light bulb associated with a smart switch and cause the smart switch to control the operation of the incandescent light bulb. In yet other embodiments, a virtual object within the augmented reality display of the tracking controller 110 may serve as a proxy device for controlling the smart device 140. For example, the user may point at a virtual menu or a virtual icon associated with a smart thermostat and control the functionality of the smart thermostat.

[0032] Network 130 may include any combination of local area and / or wide area networks that use both wired and / or wireless communication systems. In one embodiment, network 130 uses standard communication technologies and / or protocols and may include one or more than one of Bluetooth®, Bluetooth Low Energy, WiFi Direct, WiFi, cellular network technologies, or wired communication protocols. Network 130 may include different types of connections between different devices. For example, tracking device 110 may communicate with a pointing controller via a Bluetooth® connection and may communicate with smart device 140 via a WiFi connection. In some embodiments, all or some of the communication links of network 130 may be encrypted using any suitable technique.

[0033] FIG. 2 is a block diagram illustrating an exemplary embodiment of a pointing controller 120. In one embodiment, pointing controller 120 includes a control unit 210, a state sensing module 220, a control element 230, a power subsystem 240, a wireless interface 250, and an output device 260. In alternative embodiments, pointing controller 120 includes additional or different components.

[0034] The state sensing module 220 comprises an electronic device for capturing data that enables sensing of the state of the pointing controller 120, which may include, for example, position, orientation, motion, environmental conditions, or other information about the state of the pointing controller 120. For example, in one embodiment, the state sensing module 220 may comprise a six-degree-of-freedom (6DOF) inertial measurement unit (IMU) having a gyroscope for sensing orientation or angular velocity and an accelerometer for sensing acceleration. In another embodiment, the state sensing module 220 may comprise a nine-degree-of-freedom (9DOF) IMU that includes a gyroscope and an accelerometer as described above, and further includes a magnetometer for detecting a magnetic field (e.g., the Earth's magnetic field). The magnetometer may be utilized as a compass for detecting the orientation of the pointing controller 120 relative to geographical orientation. The IMU may further process data obtained by direct sensing and convert measurements to other useful data, such as calculating velocity or position from acceleration data.

[0035] In another embodiment, the state sensing module 220 may comprise one or more cameras for capturing an image of the environment suitable for tracking the position and orientation of the pointing controller 120 and correcting any drift that may be accumulated within the IMU data. Here, the image data may be processed using a scale-invariant feature transform (SIFT) algorithm and an existing map of the space, using a simultaneous localization and mapping (SLAM) technique, using specifically created tracking markers visible to the camera, or using other image-based tracking techniques. The tracking algorithm for deriving the position and orientation of the pointing controller 120 based on the captured image may be implemented with respect to the pointing controller 120 itself, or the image may be provided to the tracking device 110 for processing in order to reduce the power consumption of the pointing controller 120.

[0036] In another embodiment, the state sensing module 220 may include a radio frequency (RF) transceiver that detects beacons from anchor devices at known positions within the environment or from the tracking device 110. An accurate position in three-dimensional space can be calculated using triangulation techniques based on the time of flight of various beacon signals or can be calculated from the received signal strength indication (RSSI) from an array of anchor devices.

[0037] In another embodiment, the state sensing module 220 may include a Bluetooth® direction finding module that obtains the position of the pointing controller 120 relative to the tracking device 110 or other external device (e.g., using an array of antennas in the pointing controller 120, the tracking device 110, or both to determine the direction of radio waves).

[0038] In one embodiment, the state sensing module 220 may include a barometric pressure sensor that measures barometric pressure. The height of the pointing controller 120 may be estimated based on the detected pressure, as described in more detail below.

[0039] In one embodiment, the state sensing module 220 may utilize Bluetooth® direction finding to obtain the position of the pointing controller 120 relative to the tracking device 110 (e.g., using an array of antennas in the pointing controller 120, the tracking device 110, or both to determine the direction of radio waves), as described in more detail below.

[0040] In a further embodiment, the state sensing module 220 may include an ultrasonic pulse transmitter and / or a microphone that can be used to determine the acoustic time of flight representing the distance between the pointing controller 120 and the tracking device 110 or other reference device, as described in more detail below.

[0041] In another embodiment, the state sensing module 220 may be omitted entirely, and alternative techniques may be used to determine the pointing direction of the pointing controller 120. For example, an infrared (IR) module (not shown) may be included in place of the state sensing module 220, emitting an IR signal that is detectable by a receiver integrated with or mounted on the smart device 140 or a surrogate object (e.g., as a stick-on, low-cost, low-power device).

[0042] The control elements 230 include one or more controls for detecting control inputs from a user. The control elements 230 may include, for example, touch sensors (e.g., capacitive touch sensors), other sensors or transducers, or physical buttons, dials, switches, or other control mechanisms. In particular embodiments, the control elements 230 include a slider control interface 232 and inter-finger buttons 234. In other embodiments, different or additional control elements 230 may be employed.

[0043] The slider control interface 232 comprises a touch-sensitive pad accessible by the user's thumb or other finger. The touch-sensitive pad may comprise an array of sensing elements that detect changes in capacitance or resistance that occur in response to a touch, thereby enabling the touch-sensitive pad to detect the presence or absence of a touch and the location of the touch within the area of the pad. In some embodiments, the touch-sensitive pad may additionally include a touch force sensor to enable sensing of the force applied by the touch. The user may interact with the slider control interface 232 by performing various gestures such as tapping or swiping using the thumb or other finger. Swiping may be performed in a forward or backward direction along the axis of the finger (e.g., parallel to the pointing direction), along an axis substantially perpendicular to the axis of the finger (e.g., perpendicular to the pointing direction), or in a circular motion in a clockwise or counterclockwise direction. In the form factor of FIG. 6, the slider controller interface 232 may be positioned on the bottom side of the pointing controller 120 on a surface that extends across the bottoms of the index finger and middle finger.

[0044] The inter-finger button 234 may comprise a touch-sensitive and / or pressure-sensitive pad positioned such that it can be selected by pressing two fingers together. For example, in the form factor of FIG. 6, the inter-finger button 234 may be on the interior of a curved surface such that it is adjacent to the side of the index finger or middle finger when the pointing controller 120 is held between the index finger and middle finger. In one embodiment, the inter-finger button 234 comprises a force-sensitive resistor that detects the force applied to the touch-sensitive pad. Alternatively, the inter-finger button 234 may operate in a manner similar to the touch-sensitive pad of the slider controller interface 232 discussed above. Due to its placement, the inter-finger button may be used to detect a "pinching gesture" in which the middle finger and index finger (or other pair of adjacent fingers) are pressed towards each other and at least a threshold pressure is applied to the touch-sensitive and / or pressure-sensitive pad of the inter-finger button.

[0045] Power subsystem 240 stores power and supplies it to pointing controller 120. For example, power subsystem 240 may include a battery, a charging circuit for charging the battery, and one or more voltage regulators for controlling the voltage supplied to other components of pointing controller 120. In certain embodiments, power subsystem 240 may control pointing controller 120 to switch between different power modes (e.g., full power mode, low power mode, and sleep mode) to efficiently utilize the battery.

[0046] Wireless interface 250 communicates wirelessly with tracking device 110 via network 130. In certain embodiments, wireless interface 250 may include, for example, a Bluetooth® interface, a Bluetooth low energy interface, a WiFi link, or other wireless interface. Wireless interface 250 may communicate directly with tracking device 110 via a peer-to-peer connection, or may communicate with tracking device 110 via one or more intermediate devices over a local area network, a wide area network, or a combination thereof. In certain embodiments, wireless interface 250 may communicate directly with smart device 140.

[0047] Output device 260 includes various devices for providing an output from pointing controller 120 in response to a control signal from tracking device 110 or directly in response to an action on pointing controller 120. Output device 260 may include, for example, a haptic feedback device (e.g., a linear resonant actuator or an eccentric mass vibration motor), one or more light emitting diodes (LEDs), or an audio output device.

[0048] The control unit 210 processes inputs from the state sensing module 220, the control element 230, the power subsystem 240, and the wireless interface 250, and controls various functions of the pointing controller 120. In certain embodiments, the control unit 210 includes a processor and a non-transitory computer-readable storage medium that stores instructions which, when executed by the processor, cause the processor to perform the functions attributable to the controller 210 described herein. Alternatively, or in addition, the control unit 210 may comprise digital logic embodied as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0049] The control unit 210 processes raw data from the state sensing module 220 and the control element 230, detects motion events or interaction events, and then transmits the processed events to the tracking device 110 instead of the raw data, thereby reducing the bandwidth over the communication link 130. For example, the control unit 210 may obtain raw accelerometer, gyroscope, and / or magnetometer data from the IMU of the state sensing module 220, apply a sensor fusion algorithm, and determine the detected orientation (e.g., roll, pitch, and yaw values). Further, the control unit 210 may process raw touch data (e.g., capacitance or resistance sensing), perform processing such as analog-to-digital conversion and filtering, and generate a touch detection event indicating the detection of a touch and the position or force of the touch, which is transmitted to the tracking device 110.

[0050] Alternatively, the control unit 210 may transmit only the raw data from the state sensing module 220 and the control element 230 to the tracking device 110, and the processing described above may instead be performed on the tracking device 110. In another embodiment, the control unit 210 may transmit both raw and processed event data to the tracking device 110.

[0051] In one embodiment, other components of the pointing controller 120 may be coupled to the control unit 210 via a data bus such as a Serial Peripheral Interface (SPI) bus, a parallel bus, or an I2C bus. Further, components of the pointing controller 120 may generate an interrupt signal detectable by the control unit, enabling a short latency response to user input.

[0052] FIG. 3 is a block diagram illustrating an embodiment of the tracking device 110. In the illustrated embodiment, the tracking device 110 includes a processor 310, a storage medium 320, a wireless interface 330, a sensor 340 including a camera 345 and a state sensing module 342, and an output device 350 including a display 352 and an audio output device 354. Alternative embodiments may include additional or different components. For example, in some embodiments, the tracking device 110 without display capabilities does not necessarily include a display, a camera 345, or a content presentation module 322.

[0053] The wireless interface 330 communicates wirelessly with the pointing controller 120 via the network 130. In one embodiment, the wireless interface 330 may comprise, for example, a Bluetooth® interface, a WiFi interface, or both. The wireless interface 330 may communicate directly with the pointing controller 120 via a peer-to-peer connection, or communicate with the pointing controller 120 via one or more intermediate devices via a local area network, a wide area network, or a combination thereof. The wireless interface 330 may further communicate with the smart device 140 via the network 130.

[0054] In one embodiment, the wireless interface 330 receives transmission information and commands to the pointing controller 120 and controls the pointing controller 120 to enter various power modes, i.e., requests detailed information about the status of the pointing controller 120 such as battery status, temperature, or other diagnostic information, updates the firmware of the pointing controller 120, activates the tactile actuator on the pointing controller 120 according to a specific vibration pattern, or configures the tactile actuator on the pointing controller 120 to directly respond to events detected on the pointing controller 120 such as activating a specific button or control input on the pointing controller 120. The wireless interface 330 may further periodically receive transmissions from the pointing controller 120, including information such as tracking data from the status sensing module 220 of the pointing controller 120, control data from the control element 230 of the pointing controller 120, or battery information from the power subsystem 240 of the pointing controller 120.

[0055] The sensor 340 detects various conditions associated with the operating environment of the tracking device 110. For example, the camera 345 captures real-time video of the real-world environment within the view of the tracking device 110. The image data from the camera may be combined with virtual objects or information to present an augmented reality view of the world. The camera 345 may include a conventional image camera, a non-visual camera such as a depth camera or a LIDAR camera, or a combination thereof.

[0056] Sensor 340 may also include a state sensing module 342 to sense the movement and orientation of tracking device 110. The state sensing module 342 may include similar components and may operate in a similar manner to the state sensing module 220 of pointing controller 120 discussed above. For example, the state sensing module 342 may include one or more of an IMU, a radio frequency (RF) transceiver, a Bluetooth® direction finding module, a barometric pressure sensor, an ultrasonic pulse transmitter and / or a microphone, or other sensors.

[0057] Sensor 340 may optionally include other sensors for detecting various conditions, such as, for example, a location sensor (e.g., a global positioning system) or a temperature sensor.

[0058] The output device 350 includes various devices for providing an output from the tracking device 110 for presenting digital content. In certain embodiments, the output device 350 may include at least a display 352 and an audio output device 354. In alternative embodiments, the output device 350 may include additional output devices for providing feedback to the user, such as, for example, a tactile feedback device and one or more light emitting diodes (LEDs). The audio output device 354 may include a port for connecting one or more integrated speakers or one or more external speakers to reproduce audio associated with the presented digital content. The display device 352 comprises an electronic device for presenting image or video content, such as an LED display panel, an LCD display panel, or other types of displays. The display device 352 may be configured in a manner to present digital content in an immersive way to simulate a virtual or augmented reality environment. For example, the display device 352 may comprise a stereoscopic display that presents different images to the left and right eyes to create the appearance of a three-dimensional environment. In certain embodiments, the display device 352 may present digital content that combines rendered graphics depicting virtual objects and / or environments with content captured from the camera 345, enabling an augmented reality presentation with virtual objects overlaid on a real-world scene.

[0059] The memory medium 320 (e.g., a non-transitory computer-readable storage medium) stores instructions executable by the processor 310 to perform functions resulting from the tracking device 110 described herein. In certain embodiments, the memory medium 320 includes a content presentation module 322 and a control processing module 324. In alternative embodiments, the memory medium 320 may include additional or different modules.

[0060] The content presentation module 322 presents digital content via the display 352 and / or the audio output device 354. The content to be displayed may include a virtual reality or augmented reality environment within a three-dimensional space. The content to be displayed may include virtual objects that can be combined with real-world images captured by the camera 345. The content presentation module 322 may adapt the content based on the information received from the control processing module 324.

[0061] The control processing module 324 processes the input received from the pointing controller 120 via the wireless interface 330, may generate control commands for the smart device 140, and / or may control the output of the content presentation module 322, generating processed input data. For example, the control processing module 324 may track the position of the pointing controller 120 within the virtual environment displayed by the content presentation module 322 based on the tracking data received from the state sensing modules 220, 342. Further, the control processing module 324 may process the input from the control element 230 and detect gestures performed on the control element 230. The control processing module 324 may determine commands for output to the smart device 140 based on the detected tracking and detected gestures of the pointing controller 120, and / or may cause the content presentation module 322 to update the presentation in response to the action. Examples of the control processing module 324 are described in further detail below.

[0062] FIG. 4 illustrates an exemplary embodiment of the control processing module 324. The control processing module 324 includes a tracking module 402, an arm model 404, a gesture recognition module 406, an object interaction module 408, a menu navigation module 410, and a calibration module 412. Alternative embodiments may include different or additional modules.

[0063] The tracking module 402 infers the position and orientation of the pointing controller 120 with respect to the user's hand. In embodiments where the tracking device 110 is integrated into a head-mounted display, the position of the actor's head can be directly inferred from the position of the tracking device 110 because the tracking device 110 is fixed relative to the head position. In particular, the tracking module 402 determines the orientation of the pointing controller 120 based on the tracking data from the state awareness module 220, and obtains the position and orientation of the tracking device 110 with respect to the environment based on the sensor data from the tracking device 110 (e.g., the tracking data from the state awareness module 342). The tracking module 402 then estimates the position of the pointing controller 120 with respect to the environment based on the orientation of the pointing controller 120, the position and orientation of the tracking device 110, and an arm model 404 that models the posture of the user operating the pointing controller 120.

[0064] Based on the orientation and calculated position of the pointing controller 120, the tracking module 402 generates and continuously updates a pointing vector that originates from the position of the pointing controller 120 and extends in the direction corresponding to the detected orientation. In the case of a pointing controller 120 worn on one or more fingers, the pointing vector may extend along the central axis through the pointing controller 120 that is aligned with the finger. The pointing vector may be defined according to three-dimensional coordinates within a virtual environment tracked by the tracking device 110. Thus, the pointing vector provides a pointing direction with respect to the scene within the virtual environment. The pointing vector may comprise, for example, a pair of angles including a first angle (i.e., pitch angle) with respect to the ground plane and a second angle (i.e., yaw angle) with respect to a vertical plane perpendicular to the ground plane. In certain embodiments, an orientation angle (i.e., roll angle) about the axis of the pointing vector may also be tracked along with the pointing vector.

[0065] In one embodiment, the tracking module 402 may calculate a pointing cone around the pointing vector. Here, the cone originates from the pointing controller 120, has a central axis aligned with the pointing vector, and has a diameter that increases with the distance from the pointing controller 120. The cone angle may be adjustable by the user or developer, or may be a hard-coded parameter. Additionally, the cone angle may be automatically updated based on the context of the detected interaction with the object. For example, when interacting with an environment with a large number of objects in close proximity, the cone angle may be automatically reduced relative to an environment with a small number of objects that are far away. The tracking module 402 updates the pointing vector, the point cone, and the orientation angle when the user moves the pointing controller 120.

[0066] In one embodiment, the tracking module 402 performs tracking based at least in part on IMU data from the state sensing module 220 of the pointing controller 120.

[0067] In one embodiment, the tracking module 402 may perform tracking based at least in part on barometric pressure data from the state sensing module 220 and / or the barometric pressure sensor of the tracking device 110. For single-ended sensing, a reference pressure value may be determined corresponding to the baseline height during the calibration process. The tracking module 402 may subsequently acquire a barometric pressure reading and calculate a vertical offset from the baseline height based on the change in pressure. In another embodiment, the tracking module 402 uses differential sensing to estimate the vertical position of the pointing controller 120. In this embodiment, the differential pressure is calculated between the barometric pressure measurement obtained from the pressure sensor of the pointing controller 120 and the barometric pressure measurement obtained from the pressure sensor within the external tracking device 110. The differential sensor measurements may be filtered to compensate for natural atmospheric variations due to weather or other factors.

[0068] In another embodiment, the tracking module 402 may track the pointing controller 120, at least in part, based on the relative RSSI of the wireless signals received at both the pointing controller 120 and the tracking device 110. The relative RSSI may be used to estimate the distance between the tracking device 110 and the pointing controller 120. The distance estimation may be further improved by modeling the radiation and sensitivity patterns of the antennas within the pointing controller 120 and the tracking device 110 (or between multiple devices such as the pointing controller 120, the AR headset, and a mobile phone).

[0069] In another embodiment, the tracking module 402 may utilize Bluetooth® direction finding data to obtain the position of the pointing controller 120 relative to the tracking device 110 (e.g., using an array of antennas in the pointing controller 120, the tracking device 110, or both to determine the direction of radio waves). In one embodiment, the roll and pitch components of the pointing direction are obtained from the integrated IMU, and the yaw direction is obtained from Bluetooth® direction finding. In another embodiment, roll, pitch, and yaw may be obtained from other components of the pointing controller 120, and Bluetooth® direction finding may be used to perform corrections if there are discrepancies between other measurements. In another embodiment, the statistical error properties may be determined (e.g., if the error is consistent in some relative configurations), and information about the relative orientation may be determined based on the statistical error properties. In yet another embodiment, Bluetooth® direction finding may be utilized to determine multiple points on a rigid body (e.g., from two or more antenna arrays within the AR viewer), and in addition, the distance between the pointing controller 120 and the tracking device 110 may be estimated without necessarily relying on RSSI.

[0070] In a further embodiment, the tracking module 402 may perform tracking based on the acoustic flight time representing the distance between the ultrasonic pulse transmitter and microphone within the pointing controller 120 and the tracking device 110. In one embodiment, the tracking module 402 utilizes the estimated distance from the acoustic flight time in the tracking calculation only when the detected distance is below a maximum threshold distance (e.g., 1.5 meters). In another embodiment, the Doppler shift effect may be detected to estimate the velocity of the pointing controller 120 relative to the tracking device 110. Here, the velocity estimate may be utilized to compensate for errors in the velocity estimate determined from the IMU data using dead reckoning. In another embodiment, the distance estimated based on the acoustic flight time may be adjusted based on barometric data to compensate for variations in the speed of sound due to pressure differences.

[0071] The parameters of the wrist model 404 may be determined in the initialization process and may be updated during tracking, as will be explained below. The input parameters of the wrist model 404 may include, for example, the user's height, a standardized model of human dimensional ratios, a joint angle model, and various operating conditions that may change over time. The user's height may be manually obtained from the user during the initialization process in response to a user prompt that requests the user to enter their height. Alternatively, the height may be automatically estimated based on the estimated position of the tracking device 110 relative to the ground. For example, visual analysis may be performed on the image data captured by the camera 345 of the tracking device 110 to estimate the height. Based on the user's height, the tracking module 402 may perform a lookup in a pre-loaded lookup table that maps the height to the sizes of the hand, forearm, arm, shoulder, and neck based on a standardized model of human dimensional ratios. Then, using the combined dimensions of the human model and the detected orientation of the pointing controller 120, the tracking module 402 can apply a joint angle model and predict the relative probabilities of various wrist postures. The most probable posture may be selected, and the tracking module 402 may estimate the position of the pointing controller 120 relative to the tracking device 110 from the posture.

[0072] In certain embodiments, additional information derived by the tracking module 402 can be incorporated to more accurately predict the user's posture and eliminate undesirable results. For example, if the most likely predicted posture generated by the joint angle model predicts a user's arm (an impossible result) that intersects the known location of a detected real-world object, the tracking module 402 may instead select the next most probable prediction that does not predict an arm that intersects the detected object.

[0073] In another embodiment, the tracking module 402 may utilize information about the user's current location and / or movement history to improve the accuracy of tracking by applying different parameters of the arm model 404 in different contexts. For example, since people tend to use more extensive gestures when outdoors than when indoors, the tracking module 402 may adjust the parameters of the arm model 404 depending on whether the user is indoors or outdoors. The tracking module 402 may detect whether the user is indoors or outdoors based on image analysis of the captured images or other sensor data. In one technique, the tracking module 402 may determine whether the user is indoors or outdoors based on the presence or absence of a ceiling surface within a certain distance of the user (e.g., within 5 meters above the user), which may be detected based on image analysis from the captured images or from other sensors. In another embodiment, the tracking module 402 may measure the number of flat surfaces within a defined distance of the tracking device 110, and if the number exceeds a predetermined threshold, determine that the user is indoors, and if the number does not exceed the threshold, determine that the user is outdoors. In yet another embodiment, a location sensor (e.g., a global positioning system device) may be used to determine the geographical location of the tracking device 110. Then, using map data from a map service, the tracking module 402 may determine that the user is indoors if the location matches a building, or otherwise determine that the user is outdoors. In yet another embodiment, the radio signal strength of a radio signal (e.g., a GPS signal or a cellular data signal) received by the tracking device 110 from a remote source may be used to determine whether the user is indoors or outdoors. For example, when the radio signal strength exceeds a predetermined threshold, the tracking module 402 may determine that the user is outdoors, and when the radio signal strength is below the threshold, the tracking module 402 may determine that the user is indoors. In yet another embodiment, the tracking module 402 may perform an analysis of the luminance and / or wavelength of local light sources detected by the camera 345 to detect whether the user is indoors or outdoors.For example, high-intensity light around the color temperature of sunlight indicates that the user is likely outdoors, while a color temperature consistent with an incandescent bulb indicates that the user is indoors.

[0074] In another embodiment, the parameters of the arm model 404 may be adapted based on whether the user is sitting or standing. Here, the tracking module 402 may detect the height of the tracking device 110 relative to the ground as described above and determine whether the user is sitting or standing by detecting whether the height is significantly lower (e.g., exceeding a threshold difference) than the user's standing height.

[0075] In one embodiment, the tracking module 402 may further estimate the user's fatigue level in order to better predict the position of the pointing controller 120. Here, the tracking module 402 may model the fatigue level by tracking the amount of time the user spends holding their wrist at a certain threshold height, and the level of fatigue increases over time. As the user may prefer to keep their arm lower as fatigue increases, the parameters of the arm model 404 may cause the tracking module 402 to adjust the detected position downward as the predicted fatigue level increases. In one embodiment, the tracking module 402 may apply a machine learning approach to model the fatigue characteristics of a particular user.

[0076] In one embodiment, the tracking module 402 may utilize the image data from the camera 345 to sense the position of the pointing controller 120, the hand, the forearm, or the arm. The tracking module 402 may utilize the sensed position to re-calibrate the orientation and position of the pointing controller 120 relative to the tracking device 110 and account for accumulated drift in the tracking data as described in further detail below. Further, the tracking module 402 may apply the sensed position from the image data to improve the accuracy of the arm model 404 by updating estimated parameters such as the length of the arm or the predicted joint angles. The position of the arm may further be estimated from the integration of successive acceleration values from the accelerometer of the state sensing module 220.

[0077] In one embodiment, the tracking module 402 may further utilize position information about a virtual object (or a real-world object at a known location) to infer the position of the pointing controller 120. For example, if the object is approaching (e.g., below a threshold distance), it may be inferred that the hand is in a relaxed position close to the body. On the other hand, if the object is far away (e.g., above a threshold distance), it may be inferred that the hand is in an extended position. The tracking module 402 may adjust the parameters of the arm model 404 based on the inferred arm position.

[0078] If the tracking device 110 is not head-mounted (e.g., the tracking device 110 is embodied as a handheld smartphone or tablet), the position of the user's head may be unknown relative to the tracked position of the tracking device 110. In this case, calibration techniques may be applied to estimate the position of the user's head relative to the position of the tracking device 110. For example, in one embodiment, a user interface on the tracking device 110 prompts the user to touch the tracking device 110 to the user's nose during a calibration phase of the application. Alternatively, a camera of the tracking device 110 may capture an image of the user's face, and a face tracking algorithm may be applied to detect a center point of the face corresponding to the initial head position. In yet another embodiment, the vertical component of the head position can be manually obtained by prompting the user to enter their height, or the user's height may be obtained from a linked health tracking application or online service accessible by the tracking device 110.

[0079] Once calibrated, the tracking module 402 may estimate the vertical component of the head position to be fixed within the three-dimensional space, and the vertical movement of the tracking device 110 may be tracked within the three-dimensional space relative to this position. Alternatively, a camera 345 of the tracking device 110 may capture images that are processed to detect changes in the height of the terrain. The estimated head position of the user may be updated based on the detected changes in the height of the terrain such that it is at a generally fixed vertical position above the ground.

[0080] In the horizontal plane, the tracking module 402 may estimate the head position to be a fixed horizontal offset from the tracked position of the tracking device 110. Thus, as the tracking device 110 moves and rotates in the horizontal plane, the head position is estimated at a fixed horizontal distance from the tracked position of the tracking device 110.

[0081] Recalibration may be performed when the user changes from a sitting position to a standing position (or vice versa). This change may be indicated manually by the user or automatically detected when a suitable offset in the vertical position of the tracking device 110 (and / or pointing controller 120) is detected. For example, the camera 345 of the tracking device 110 may be processed to detect the height of the tracking device 110 relative to the ground and may capture an image that can be used to detect when the user sits or stands up.

[0082] In an alternative embodiment, the user's head position may be assumed to be completely fixed. Here, instead of estimating the head position in the horizontal plane to track the horizontal movement of the tracking device 110 at a fixed offset, the head position may instead be estimated to remain at both a fixed vertical and horizontal position within the three-dimensional space without tracking the movement of the tracking device 110.

[0083] In yet another embodiment, a hybrid model that combines the techniques described above may be used. Here, the initial head location with respect to the tracking device 110 is first calibrated using the calibration techniques described above (e.g., by prompting the user to touch the display device to the user's nose). The tracking module 402 may first be set to a "stationary" mode that estimates the head position such that it is maintained at a fixed position within the three-dimensional space. The position of the tracking device 110 is tracked using the state sensing module 342 as it moves through the three-dimensional space, and the distance between the tracking device 110 and the estimated head position that is fixed thereto is calculated. When the distance between the estimated head location and the tracking device 110 exceeds a predetermined activation radius (e.g., approximately equal to the estimated length of the user's fully extended arm), the tracking module 402 switches to a "walking" mode. In the "walking" mode, the head position is instead estimated to be a fixed distance behind the detected position of the tracking device 110. When the tracking device 110 detects that its motion has dropped below a threshold speed and remains below the threshold speed for a threshold time period, the tracking module 402 switches to return to a "stationary mode" where the estimated position of the head becomes fixed and is no longer updated based on the position of the tracking device 110.

[0084] Alternatively, when in the "walking mode", the head position with respect to the tracking device 110 may instead be estimated using a spring mass or spring mass damper model. In this embodiment, the estimated distance of the head behind the detected position of the display tracking 110 may vary over time, but stabilizes to a fixed position when the tracking device 110 is stable over a long time period. When the tracking device 110 detects in this embodiment that the distance between the smartphone and the head drops below a deactivation radius, the tracking module 402 switches to return to the "stationary" mode.

[0085] The gesture recognition module 406 detects gestures made by the user using the pointing controller 120. Examples of gestures may include, for example, moving the pointing controller 120 in a predetermined motion, or interacting with the slider control interface 232 and / or the inter-digital button in a specific manner (e.g., single tapping, double tapping, maintaining long-term contact, or a combination of interactions in a specific pattern). Here, a pinching gesture may be detected when the user presses the middle finger and index finger (or other finger in contact with the pointing controller 120) together, thereby placing one or more fingers in contact with the inter-digital button 234 on the pointing controller 120 with a threshold amount of pressure for at least a threshold time period. The pinching gesture may be released by separating the fingers or relaxing the applied pressure. In some embodiments, the gesture recognition module 406 may capture the force or time period of the pinching gesture and take different actions depending on these captured parameters. A swiping gesture may be detected when the user performs a swiping motion on the slider controller interface 232. This gesture may typically be performed using the thumb (or other finger) on the hand wearing the pointing controller 120, but alternatively may be performed by a finger on the opposite hand. Here, the swiping gesture may comprise a linear swiping gesture along a line parallel to one or more fingers holding the pointing controller 120 in any direction, or along a line substantially perpendicular to one or more fingers in any direction. Alternatively, the swiping gesture may comprise a radial swiping gesture performed clockwise or counterclockwise about a reference point within the plane of the slider controller interface 232. In some embodiments, the gesture recognition module 408 may capture the force, speed, or distance of the swiping gesture and take different actions depending on these captured parameters.Other types of gestures may also be recognized to perform various tasks.

[0086] The object interaction module 408 determines when a pointing vector or cone intersects an object, where the pointing vector or cone may correspond to the smart device 140, a different real-world object at a known location, or a virtual object within a scene displayed on the display of the tracking device 110. For example, the object interaction module 408 stores coordinates representing the locations occupied by the real-world and virtual objects and detects when the pointing vector or cone intersects the coordinates occupied by one of the objects. If the pointing vector or cone intersects multiple objects, the object interaction module 406 may default to selecting the object closest to the pointing controller 120. In another embodiment, the tracking module 402 may intelligently predict whether the user intends to point at a nearby object (e.g., one that is less than 5 meters away) or a far object (e.g., one that is more than 5 meters away) when the pointing vector intersects multiple objects. For example, the tracking module 402 may detect that the arm is substantially aligned with the user's eye and infer that the user intends to point at a far object when the arm is fully extended. The tracking module 402 may also infer that the user intends to point at a nearby object when the arm is bent and held at a position below eye level.

[0087] In one embodiment, a visual indicator (e.g., a visual outer glow or halo effect, a shadowed contour, a bounding box, or the like) is displayed within the augmented reality display in association with the object being pointed at. Optionally, detailed information about the selected object, such as, for example, an object identifier, the distance from the pointing controller 120 to the selected object, the status of the object, etc., may also be displayed. Further, when the object is pointed at, the object interaction module 406 may cause the tactile motor of the pointing controller 120 to vibrate to provide a physical feedback of the action. Alternatively, other visual or audio feedback may be provided to indicate when the object is selected.

[0088] The object interaction module 408 may determine commands associated with gestures or control interactions that are performed when the user points at a specific object. For example, the object interaction module 408 may detect a confirmation interaction (e.g., a pinching gesture that activates the inter-digit button 234) when the user is pointing at an object. Here, the object interaction module 408 may confirm the selection of the smart device 140 when the smart device 140 or a proxy object associated with the smart device 140 is selected based on a pointing vector or a cone, when the interaction is performed. In response to confirming the selection, the object interaction module 408 may establish a connection to the smart device 140 via the network 130. The object interaction module 408 may then detect the selection of gestures or control elements performed by the user to control various functions of the smart device 140. Here, a set of predetermined gestures may be associated with different control functions of the smart device 140. The mapping of the gestures to those functions may be configured in an intuitive way to enable natural control of the smart device 140. For example, to control a smart thermostat, the user may point or move the hand upward, turn the hand in a clockwise direction, or tilt the hand upward to increase the temperature, or point or move the hand downward, turn the hand in a counterclockwise direction, or tilt the hand downward to decrease the temperature. Alternatively, the slider control interface 232 on the pointing controller 120 may control the temperature based on the direction of the swipe.

[0089] In another exemplary interaction, the user may perform a gesture to control the smart lighting dimmer to increase or decrease the light output intensity or change the color temperature of the light bulb. For example, the tracking device 110 may detect when the pointing controller 120 is pointing at the lighting. The pointing controller 120 triggers a tactile, audible, or visual feedback signal when the pointing vector intersects the position of the lighting and indicates that the lighting is selected. The pointing controller 120 may detect when the user holds a thumb on the slider controller interface 232 and drags the thumb in one direction to increase the brightness of the light and in the opposite direction to decrease the brightness. Releasing the thumb may stop the change in brightness, and moving the pointing vector away from the lighting in a direction away from the lighting may deselect the lighting so that further interaction with the control element 230 no longer generates a control signal for the lighting.

[0090] In some embodiments, a deselection action may be performed to deselect the selected smart device 140. In some embodiments, deselection may be performed by pointing away from the smart device 140 or an associated proxy object. In some embodiments, tactile or other feedback may be output to confirm that the interaction is complete.

[0091] The menu navigation module 410 generates a menu presented on the display device 352 in response to the selection of the smart device 140 or an associated proxy object, or another action or combination of actions, such as, for example, the slider control interface being tapped while an object is selected. The menu may enable the user to view and / or modify advanced configuration settings associated with the smart device 140. In some embodiments, a wheel or slider interface may be displayed to enable the user to quickly modify parameters using a swiping gesture.

[0092] The calibration module 412 performs a calibration process to calibrate the pointing controller 120 in order to initialize the relative position and orientation of the pointing controller 120 with respect to the position and orientation within the virtual environment presented by the content presentation module 362. The roll and pitch of the pointing controller 120 can be detected from the state sensing module 220 using the detected direction of gravity (as sensed by the state sensing module 220) mapped in the downward direction along the vertical axis of the virtual environment. The horizontal direction (yaw) of the pointing controller 120 can be sensed relative to a reference direction during calibration using various techniques. This reference direction may be aligned with the forward direction of the tracking device 110 during the calibration process.

[0093] In one embodiment, a magnetometer within the state sensing module 220 of the pointing controller 120 may operate as a compass to detect magnetic north. A magnetometer within the tracking device 110 may similarly detect magnetic north, and the calibration module 412 may perform a calibration to align these reference directions.

[0094] In another embodiment, the location and orientation of the pointing controller 120 can be detected based on image (visual or depth) analysis performed on one or more images captured by the camera of the tracking device 110 or another external camera. The calibration module 412 may then perform calibration using the detected tracking data and the determined location and position from the image data.

[0095] In another embodiment, the calibration module 412 performs calibration by instructing the user to point straight ahead and then perform a specific gesture (e.g., a double-tap on the slider control interface 232 while also pressing the inter-finger button 234). Unintended actions when the pointing controller 120 is not substantially horizontal, as detected by the state sensing module 220, may be rejected by ignoring the gesture when it is detected. The calibration module 412 may then set the direction as a reference direction mapped to the straight-ahead direction within the virtual environment.

[0096] In another embodiment, calibration may be performed by instructing the user to point at a few real-world objects at locations that are known or can be detected from images captured by the image processing device. Here, the pitch of the pointing controller 120 should approximately align the pitch vector with the target when determining when the user is pointing at the target, and in addition, the pointing controller 120 should be held substantially stationary. The calibration module 412 may then perform calibration using the known positions of these objects within the virtual environment. In one embodiment, this calibration step may be performed as part of a user tutorial to train the user on how to use the pointing controller 120 to interact with the objects.

[0097] In other embodiments that do not rely on the state sensing module 220 of the pointing controller 120 (e.g., embodiments that use IR-based detection of the pointing direction), the calibration module 412 may be omitted.

[0098] In one particular embodiment, the tracking device 110 is configured to display a target object located far away (to minimize the range error), and a prompt is displayed to instruct the user to point at the target object. The calibration module 412 detects when the pointing controller 120 is substantially stationary (e.g., by detecting that the angular rotation rate is below a predetermined threshold), and determines that the current pointing direction is the direction of the target object. In some embodiments, the tracking device 110 may provide a visual indicator to guide the user through calibration. For example, the tracking device 110 may display a visual indicator (e.g., a progress bar animation, a change in the size of the visual indicator, etc.) that begins to "fill" when the pointing controller 120 is stationary over a short period of time and, in addition, the pitch of the pointing controller 120 substantially matches the pitch of the target with respect to the user. During this time, the calibration module 412 records the detected orientation of the pointing controller 120 and determines the difference in the yaw (azimuth) of the pointing controller 120 with respect to the yaw of the tracking device 110. If the user moves the pointing controller 120 during the calibration cycle or the pitch is out of the acceptable range, the progress is reset. Once the calibration process is complete, the target object may be removed from the display and the calibration values are stored. The calibration process described above can be repeated multiple times using target objects at different yaws (azimuths) and / or pitches to improve accuracy. The calibration process can also be performed by instructing the user to use target objects at different depths or to keep the target installed at the periphery of the field of view while facing in one direction.

[0099] In another embodiment, the tracking device 110 may display the outline of the image of the pointing controller 120, place the tracking device 110 on a flat horizontal surface, and then instruct the user to place the pointing controller 120 on the display screen of the tracking device 110 that is aligned with the outline of the image. The calibration module 412 detects when the pitch of the pointing controller 120 is below a threshold angle and when both the tracking device 110 and the pointing controller 120 are held stationary over a threshold time period. When these conditions are detected, the calibration module 412 stores the difference between the detected yaw of the pointing controller 120 and the tracking device 110 as a calibration offset. During operation, this calibration offset is subtracted from the yaw measurement of the pointing controller 120.

[0100] Once calibrated, calibration module 412 may enable the user to verify the calibration by displaying the test target and enabling the user to ensure that the calibration was performed correctly. In another embodiment, calibration module 412 may perform continuous automatic calibration during use. Calibration module 412 may store a set of foci associated with different types of objects. Here, the focus of an object represents a point on an object of a given object type where the user is likely to have a preference for pointing when attempting to point at that object type. For simple shapes, the focus may be calculated by computing the center of mass of the object, assuming a uniform density. For complex shapes, the focus may be calculated by computing the center of mass of the convex hull that "encloses" the shape. For other types of functional objects, the focus may be manually assigned based on the object type or learned for different types of objects using an external tracking system. For these types of objects, the focus may be biased towards the point of interaction. For example, for a computer monitor, the focus may correspond to the center of the screen and may ignore the stand. For a bicycle, the focus may be biased from the center of mass towards a point closer to the handlebars. For a piano, the focus may be biased from the center of mass towards a point closer to the keyboard. For a door, the focus may be biased from the center of mass towards a point closer to the handle / knob.

[0101] In some embodiments, the focus of an object may change with distance. For example, from a far distance, people are likely to point at the center of an object regardless of the object's purpose. Thus, in some embodiments, the center of mass of an object may be used as the focus when the object is far enough away. However, when closer to the object, people tend to direct towards the point of interaction on the functional object, but may continue to point at the center of mass for simpler objects. Thus, in some embodiments, a pre-assigned focus based on the object type may be used when the object is closer than a predetermined distance. Each time an object is selected, the calibration module 412 may determine the difference between the direction of the focus of the object and the actual pointing direction of the pointing controller 120 at the instant the object is selected. If these differences (and especially the yaw component) are consistently biased in one direction, the calibration module 412 may detect an incorrect calibration. In some embodiments, an incorrect calibration is detected only once a sufficient confidence level is reached, such as after the yaw components of several object selections have been consistently biased in one direction. In response to detecting an incorrect calibration, the calibration module 412 can adjust the calibration parameters to correct the incorrect calibration. This recalibration may be performed instantaneously or applied gradually over several seconds (to prevent the user from experiencing any "jumps").

[0102] FIG. 5 is a flowchart illustrating an exemplary embodiment of a process for controlling a smart device 140 using a pointing controller 120. The tracking device 110 obtains tracking data (502) associated with the position and orientation of the pointing controller 120. The tracking data may include motion data from which a position can be derived, and may be in the form of IMU data, image data, RF beacon data, or a combination thereof. The tracking device 110 tracks a pointing vector (504) associated with the pointing direction of a user wearing the pointing controller 120 based on the tracking data. The pointing vector may comprise a line or cone that widens with distance from the pointing controller 120 along the axis of the pointing direction. The tracking device 110 detects a selection of the smart device 140 (506) based on detecting an intersection between the tracked pointing vector and an object location associated with the smart device 140, and places the smart device 140 in the selected state. Here, the object location may be the location of the smart device 140 itself, or the location of a real-world or virtual proxy object associated with the smart device 140. In some embodiments, the tracking device 110 confirms the selection in response to detecting a predetermined interaction (e.g., a gesture or selection of an interface control device on the pointing controller) when the pointing direction intersects the object location. In response to the selection of the smart device 140, the tracking device 110 establishes a connection to the smart device 140 (508), which enables communicating commands to the smart device 140 and / or receiving status information from the smart device 140. The tracking device 140 generates a command (510) for controlling the smart device 140 based on the detected interaction. For example, the tracking device 110 detects one or more interactions (e.g., a predetermined gesture or interaction with an interface control device) performed using the pointing controller and determines a mapping of the interaction to a control command associated with the smart device 140.The tracking device may subsequently detect an interaction associated with the pointing controller 120 to deselect the smart device 140 and return the smart device 140 to an unselected state (512). For example, the tracking device 110 may deselect the smart device 140 in response to detecting that the user points in a direction away from the direction of the object location associated with the smart device 140 or performs a different predetermined gesture associated with deselecting the smart device 140.

[0103] In an alternative embodiment, one or more components of the control processing module 324 may be implemented on the pointing controller 120 instead of on the tracking device 110. For example, in one embodiment, the functions of the tracking module 402 and the gesture recognition module 406 may instead be performed by the pointing controller 120. In this embodiment, the tracking results and the detected gestures may be communicated directly to the tracking device 110 instead of communicating the raw tracking as well as the control element data. Alternatively, in other embodiments, one or more components of the control processing module 324 may be implemented on a separately communicatively coupled device. For example, a mobile device, a personal computer, or a game console may receive the raw tracking and control element data from the pointing controller 120, perform the functions of the control processing module 324 for processing the raw data, transmit the processed control information to the tracking device 110, and cause the tracking device 110 to update the display on the display device 352. In yet another embodiment, one or more components of the control processing module 324 may be implemented on a remote server (e.g., a cloud server) communicatively coupled to the pointing controller 120 and the tracking device 110.

[0104] In another embodiment, the tracking device 110 is omitted and the pointing controller 120 determines the smart device 140 or proxy device that the user intends to interact with by using a modulated infrared (IR) signal. Here, a transmitter located within the pointing controller 120 transmits a modulated signal in a direction substantially aligned with the finger and with a beam angle narrow enough to allow for precise targeting. The signal is received and demodulated by the smart device 140, the proxy device, or a beacon device attached to the smart device 140 or proxy device. The receiving device may then signal back to the pointing controller 120 through the network 130 or by retransmitting a new IR signal. Alternatively, a transmitter located within the smart device 140 (or proxy device or attached beacon device) transmits a signal that is received and demodulated by a direction-sensitive sensor located within the pointing controller 120. Once the target device is identified and selected, the generated command 510 can be transmitted to the smart device 140 through the network 130. In some embodiments, the command may alternatively be communicated directly via the IR channel.

[0105] In the case of an IR-based solution, the transmitted beam or receiver optics will typically be narrow enough to select one object. If multiple objects are illuminated, the tracking device 110 can attempt to resolve the ambiguity by examining the IR illumination intensity or duration of the illumination. If ambiguity resolution is not possible at the sensing level, the tracking device 110 can provide feedback to the user (by indicating through optical, auditory, or tactile feedback that the pointing was ambiguous), thereby allowing the user to clearly point at the intended target object.

[0106] In yet further embodiments, an external camera, depth sensing, or ranging system may be located within a ceiling or wall-mounted module and may be utilized to track the pointing controller 120 or to directly perform pose estimation and hand tracking without the use of the pointing controller 120.

[0107] Additional Considerations Throughout this specification, some embodiments have used the expression "coupled" along with its derivatives. As used herein, the term "coupled" is not necessarily limited to two or more elements being in direct physical or electrical contact. Rather, the term "coupled" may also include two or more elements that do not directly contact each other but still cooperate or interact with each other.

[0108] Similarly, as used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," or any other variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0109] In addition, the use of "the," "a," or "an" is employed to describe elements and components of the embodiments of this specification. This is done merely for convenience and to give a general meaning to the invention. This description should be read to include one or at least one, and the singular form also includes the plural form unless it is obvious that it means otherwise.

[0110] Finally, as used herein, any reference to "one embodiment" or "an embodiment" means that the particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in this specification are not necessarily all referring to the same embodiment.

[0111] Upon a thorough reading of this disclosure, those skilled in the art will appreciate additional alternative structural and functional designs regarding the described embodiments as disclosed from the principles herein. Accordingly, while specific embodiments and applications have been illustrated and described, it should be understood that the disclosed embodiments are not limited to the precise structures and components disclosed herein. Various modifications, changes, and variations will be apparent to those skilled in the art and may be made without departing from the scope in the arrangements, operations, and details of the methods and apparatuses disclosed herein.

Claims

1. A method for controlling interaction with a smart device using a pointing controller, the method comprising: (a) a processor obtaining sensor data from a state sensing device of the pointing controller; (b) in response to the processor determining that a distance between coordinates in a three-dimensional space associated with the smart device and the pointing controller exceeds a threshold distance, the processor inferring that the position of the arm is in an outstretched position, and the processor adjusting parameters of an arm model based on the inferred position of the arm, or (b) in response to the processor determining that a distance between coordinates in the three-dimensional space associated with the smart device and the pointing controller is less than a threshold distance, the processor inferring that the position of the arm is in a relaxed position near the body, and the processor adjusting parameters of the arm model based on the inferred position of the arm; (c) the processor tracking movement of a pointing vector through the three-dimensional space based on the sensor data, the arm model, and the adjusted parameters of the arm model; (d) the processor detecting an intersection between the pointing vector and coordinates in the three-dimensional space associated with the smart device to place the smart device in a selected state; (e) the processor causing a virtual menu associated with the smart device to be displayed on an augmented reality display device; (f) the processor detecting a control interaction between the virtual menu and the pointing controller when the smart device is in the selected state; (g) the processor generating a command for controlling an operation of the smart device based on the control interaction and the method is performed in the order from (a) to (g).

2. The method of claim 1, wherein the coordinates associated with the smart device are coordinates representing a physical location of the smart device.

3. The method according to claim 1, wherein the coordinates associated with the smart device are coordinates representing the location of a proxy object associated with the smart device.

4. Detecting the intersection includes generating a pointing cone having a central axis aligned with the pointing vector, an origin proximate to the location of the pointing controller, and a radius that increases with the distance of the pointing vector from the origin; and detecting the intersection with the pointing vector in response to the pointing cone overlapping the coordinates associated with the smart device. The method according to claim 1, comprising:

5. The method according to claim 1, wherein detecting the control interaction includes detecting activation of an inter-digit button of the pointing controller.

6. Detecting the control interaction includes detecting an interaction with a slider control interface of the pointing controller; navigating between different menu items in the virtual menu in response to the interaction with the slider control interface; and selecting a menu item in response to detecting activation of an inter-digit button of the pointing controller. The method according to claim 1, comprising:

7. The method according to claim 1, wherein detecting the control interaction includes detecting a gesture performed using the pointing controller indicative of a control function of the smart device.

8. The method according to claim 1, wherein the state sensing device comprises a camera integrated with the pointing controller.

9. Tracking the movement of the pointing vector includes detecting whether the pointing controller is indoors or outdoors; and adjusting parameters of the arm model according to whether the pointing controller is indoors or outdoors. The method according to claim 1, comprising:

10. Tracking the movement of the pointing vector includes detecting whether the user of the pointing controller is sitting or standing; and adjusting parameters of the arm model according to whether the user of the pointing controller is sitting or standing. The method according to claim 1, comprising:

11. Tracking the movement of the pointing vector comprises detecting a fatigue level associated with a user of the pointing controller, and adjusting parameters of the arm model according to the detected fatigue level The method according to claim 1.

12. further comprising recognizing the smart device as smart lighting, and detecting the control interaction with the pointing controller includes detecting a swiping gesture on a touch interface of the pointing controller, and generating the command includes controlling the dimming of the smart lighting according to the direction of the swiping gesture The method according to claim 1.

13. A non-transitory computer-readable storage medium storing instructions for controlling an interaction with a smart device using a pointing controller, the instructions, when executed by one or more processors, cause the one or more processors to (a) the processor obtains sensor data from a state sensing device of the pointing controller, and (b) in response to the processor determining that a distance between coordinates in a three-dimensional space associated with the smart device and the pointing controller exceeds a threshold distance, the processor infers that the position of the arm is in an outstretched position, and the processor adjusts parameters of the arm model based on the inferred position of the arm, or (c) in response to the processor determining that a distance between the coordinates in the three-dimensional space associated with the smart device and the pointing controller is less than the threshold distance, the processor infers that the position of the arm is in a relaxed position near the body, and the processor adjusts the parameters of the arm model based on the inferred position of the arm, and (d) the processor tracks the movement of a pointing vector through the three-dimensional space based on the sensor data, the arm model, and the adjusted parameters of the arm model (d) the processor detecting an intersection of the pointing vector with coordinates in the three-dimensional space associated with the smart device to place the smart device in a selected state; (e) the processor causing the augmented reality display device to display a virtual menu associated with the smart device; (f) the processor detecting a control interaction between the pointing controller associated with the virtual menu and the smart device when the smart device is in the selected state; (g) the processor generating a command for controlling the operation of the smart device based on the control interaction (h) causing steps including (a) to (g) to be performed in the order of (a) to (g), a non-transitory computer-readable storage medium. (Claim 14) (13) The non-transitory computer-readable storage medium according to claim 13, wherein the coordinates associated with the smart device are coordinates representing a physical location of the smart device. (Claim 15) (13) The non-transitory computer-readable storage medium according to claim 13, wherein the coordinates associated with the smart device are coordinates representing a location of a proxy object associated with the smart device. (Claim 16) (13) Detecting the intersection includes: (i) generating a pointing cone having a central axis aligned with the pointing vector, an origin proximate to the location of the pointing controller, and a radius increasing with the distance from the origin of the pointing vector; (ii) detecting the intersection with the pointing vector in response to the pointing cone overlapping the coordinates associated with the smart device. (13) The non-transitory computer-readable storage medium according to claim 13. (Claim 17) (13) The non-transitory computer-readable storage medium according to claim 13, wherein detecting the control interaction includes detecting activation of an inter-digit button of the pointing controller. (Claim 18) (13) Detecting the control interaction includes: (i) detecting an interaction with a slider control interface of the pointing controller; (ii) navigating between different menu items in the virtual menu in response to the interaction with the slider control interface; Selecting a menu item in response to detecting activation of the inter-finger button of the pointing controller The non-transitory computer-readable storage medium according to claim 13, comprising:

19. The non-transitory computer-readable storage medium according to claim 13, wherein detecting the control interaction includes detecting a gesture performed using the pointing controller indicating a control function of the smart device.

20. The non-transitory computer-readable storage medium according to claim 13, wherein the state sensing device comprises a camera integrated with the pointing controller.

21. Tracking the movement of the pointing vector includes detecting whether the pointing controller is indoors or outdoors, and adjusting parameters of the arm model according to whether the pointing controller is indoors or outdoors The non-transitory computer-readable storage medium according to claim 13, comprising:

22. Tracking the movement of the pointing vector includes detecting whether the user of the pointing controller is sitting or standing, and adjusting parameters of the arm model according to whether the user of the pointing controller is sitting or standing The non-transitory computer-readable storage medium according to claim 13, comprising:

23. Tracking the movement of the pointing vector includes detecting a fatigue level associated with the user of the pointing controller, and adjusting parameters of the arm model according to the detected fatigue level The non-transitory computer-readable storage medium according to claim 13, comprising:

24. Further comprising recognizing the smart device as smart lighting, wherein detecting the control interaction with the pointing controller includes detecting a swiping gesture on a touch interface of the pointing controller, and generating the command includes controlling dimming of the smart lighting according to a direction of the swiping gesture The non-transitory computer-readable storage medium according to claim 13.

25. A tracking device, comprising one or more processors, A non-transitory computer-readable storage medium storing instructions for controlling interaction with a smart device using a pointing controller, the instructions, when executed by the one or more processors, cause the one or more processors to (a) the processor obtains sensor data from a state sensing device of the pointing controller; (b) in response to the processor determining that the distance between the coordinates in the three-dimensional space associated with the smart device and the pointing controller exceeds a threshold distance, the processor infers that the position of the arm is in the outstretched position, and the processor adjusts the parameters of the arm model based on the inferred position of the arm, or (c) in response to the processor determining that the distance between the coordinates in the three-dimensional space associated with the smart device and the pointing controller is less than a threshold distance, the processor infers that the position of the arm is in a relaxed position near the body, and the processor adjusts the parameters of the arm model based on the inferred position of the arm; (d) the processor tracks the movement of the pointing vector through the three-dimensional space based on the sensor data, the arm model, and the adjusted parameters of the arm model; (e) the processor detects an intersection between the pointing vector and the coordinates in the three-dimensional space associated with the smart device to place the smart device in a selected state; (f) the processor causes a virtual menu associated with the smart device to be displayed on an augmented reality display device; (g) the processor detects a control interaction between the virtual menu and the pointing controller when the smart device is in the selected state; (h) the processor generates a command for controlling the operation of the smart device based on the control interaction including steps to be performed in the order of (a) to (g), a non-transitory computer-readable storage medium and a tracking device comprising.

26. The tracking device according to claim 25, wherein the coordinates associated with the smart device are coordinates representing the physical location of the smart device.

27. The tracking device according to claim 25, wherein the coordinates associated with the smart device are coordinates representing the location of a proxy object associated with the smart device.

28. Detecting the intersection includes generating a pointing cone having a central axis aligned with the pointing vector, an origin proximate to the location of the pointing controller, and a radius that increases with the distance of the pointing vector from the origin; and detecting the intersection with the pointing vector in response to the pointing cone overlapping the coordinates associated with the smart device. The tracking device according to claim 25, comprising:

29. The tracking device according to claim 25, wherein detecting the control interaction includes detecting activation of an inter-finger button of the pointing controller.

30. Detecting the control interaction includes detecting an interaction with a slider control interface of the pointing controller; navigating between different menu items in the virtual menu in response to the interaction with the slider control interface; and selecting a menu item in response to detecting activation of an inter-finger button of the pointing controller. The tracking device according to claim 25, comprising:

31. The tracking device according to claim 25, wherein detecting the control interaction includes detecting a gesture performed using the pointing controller indicating a control function of the smart device.

32. The tracking device according to claim 25, wherein the state sensing device comprises a camera integrated with the pointing controller.

33. Tracking the movement of the pointing vector includes detecting whether the pointing controller is indoors or outdoors; and adjusting parameters of the arm model according to whether the pointing controller is indoors or outdoors. The tracking device according to claim 25, comprising:

34. Tracking the movement of the pointing vector includes detecting whether the user of the pointing controller is sitting or standing; adjusting parameters of the arm model according to whether the user of the pointing controller is sitting or standing; The tracking device according to claim 25, comprising the above.

35. Tracking the movement of the pointing vector includes: detecting a fatigue level associated with the user of the pointing controller; adjusting parameters of the arm model according to the detected fatigue level; The tracking device according to claim 25, comprising the above.

36. further comprising recognizing the smart device as smart lighting; detecting the control interaction with the pointing controller includes detecting a swiping gesture on a touch interface of the pointing controller; generating the command includes controlling dimming of the smart lighting according to the direction of the swiping gesture; The tracking device according to claim 25.

Images