Interacting with smart device using pointing controller

Abstract

To provide favorable interactions with a smart device.SOLUTION: A pointing controller worn by a user enables intuitive control of various connected smart devices. Users may indicate which device is to be controlled by pointing in the direction of the smart device (or a proxy object) and performing a predetermined action to select the device for control. Once selected, the user may interact with the smart device by performing various gestures or interactions with one or more control elements integrated into the pointing controller. The smart device may provide feedback using visual or auditory indicators on the smart device, or by sending a control signal to the pointing controller or other device in proximity to the user that provides haptic, auditory, or visual feedback.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Application No. 62 / 807,094, filed February 18, 2019, which is incorporated herein by reference.

[0002] The present disclosure relates to controlling smart devices, and more particularly to controlling interactions with smart devices using a pointing controller. [Background technology]

[0003] Users traditionally configure smart devices such as thermostats, speakers, and lighting systems through controls on the device itself or through user interfaces accessible on smartphone applications or web portals. Accessing these interfaces is not always convenient for users and can be inefficient. Voice-controlled devices offer an additional level of control, but may be undesirable in quiet situations and undesirable in noisy environments. Summary of the Invention [Means for solving the problem]

[0004] A method, a non-transitory computer-readable storage medium, and a tracking device control interaction with a smart device using a pointing controller. Sensor data is obtained from a state sensing device of the pointing controller. Movement of a pointing vector is tracked through three-dimensional space based on the sensor data and a stored arm model. An intersection of the pointing vector with a coordinate in the three-dimensional space associated with the smart device is detected to place the smart device in a selected state. An augmented reality display device displays a virtual menu associated with the smart device. A control interaction with the pointing controller associated with the virtual menu is detected when the smart device is in the selected state, and a command is generated to control an operation of the smart device based on the control interaction.

[0005] In one embodiment, the coordinates associated with the smart device comprise the physical location of the smart device. In another embodiment, the coordinates associated with the smart device comprise the location of a real or virtual representative device associated with the smart device.

[0006] In one embodiment, detecting an intersection of the pointing vector and the coordinates associated with the smart device includes generating a pointing cone having a center axis aligned with the pointing vector, an origin proximate to the location of the pointing controller, and a radius that increases with distance from the origin of the pointing vector, wherein the intersection is detected in response to the pointing cone overlapping with the coordinates associated with the smart device.

[0007] In one embodiment, detecting the control interaction includes detecting activation of an inter-finger button of the pointing controller. Further, in one embodiment, detecting the control interaction includes detecting interaction with a slider control interface of the pointing controller, navigating among different menu items in the virtual menu in response to the interaction with the slider control interface, and selecting a menu item in response to detecting activation of the inter-finger button of the pointing controller. In another embodiment, detecting the control interaction includes detecting a gesture made with the pointing controller that indicates a control function of the smart device.

[0008] In one embodiment, tracking the movement of the pointing vector includes performing the tracking based on a camera integrated with the pointing controller.

[0009] In one embodiment, tracking the movement of the pointing vector includes detecting whether the pointing controller is indoors or outdoors, and adjusting parameters of the arm model depending on whether the pointing controller is indoors or outdoors. In another embodiment, tracking the movement of the pointing vector includes detecting whether a user of the pointing controller is sitting or standing, and adjusting parameters of the arm model depending on whether the user of the pointing controller is sitting or standing. In another embodiment, tracking the movement of the pointing vector includes detecting a fatigue level associated with a user of the pointing controller, and adjusting parameters of the arm model depending on the detected fatigue level.

[0010] In one embodiment, tracking movement of the pointing vector includes determining that a coordinate in three-dimensional space associated with the smart device is greater than a threshold distance from the pointing controller, generating parameters of an arm model corresponding to the arm in an outstretched position, and tracking movement based on the parameters of the arm model.

[0011] In one embodiment, tracking movement of the pointing vector includes determining that a coordinate in three-dimensional space associated with the smart device is below a threshold distance from the pointing controller, generating parameters of an arm model corresponding to the arm being in a relaxed position near the body, and tracking movement based on the parameters of the arm model.

[0012] In one embodiment, the tracking device recognizes the smart device as a smart light, detects a control interaction with the pointing controller including detecting a swiping gesture on a touch interface of the pointing controller, and generates a command including controlling dimming of the light according to the direction of the swiping gesture.

[0013] Additional embodiments will be apparent to those skilled in the art. The present specification also provides, for example, the following items: (Item 1) 1. A method for controlling interaction with a smart device using a pointing controller, the method comprising: acquiring sensor data from a state sensing device of the pointing controller; tracking movement of a pointing vector through three-dimensional space based on the sensor data and a stored arm model; detecting an intersection of the pointing vector with a coordinate in the three-dimensional space associated with the smart device to place the smart device in a selected state; causing an augmented reality display device to display a virtual menu associated with the smart device; When the smart device is in the selected state, detecting a control interaction with the pointing controller associated with the virtual menu; generating a command to control an operation of the smart device based on the control interaction; A method comprising: (Item 2) Item 10. The method of item 1, wherein the coordinates associated with the smart device comprise a physical location of the smart device. (Item 3) Item 10. The method of item 1, wherein the coordinates associated with the smart device comprise the location of a real or virtual representative device associated with the smart device. (Item 4) Detecting the intersection includes: generating a pointing cone having a central axis aligned with the pointing vector, an origin proximate the location of the pointing controller, and a radius that increases with distance from the origin of the pointing vector; detecting the intersection with the pointing vector in response to the pointing cone overlapping the coordinate associated with the smart device; Item 1. The method according to item 1, comprising: (Item 5) Item 10. The method of item 1, wherein detecting the control interaction includes detecting activation of an inter-finger button of the pointing controller. (Item 6) Detecting the regulatory interaction includes: Detecting an interaction with a slider control interface of the pointing controller; navigating among different menu items within the virtual menu in response to interaction with the slider control interface; selecting a menu item in response to detecting activation of an inter-finger button of the pointing controller; Item 1. The method according to item 1, comprising: (Item 7) Item 10. The method of item 1, wherein detecting the control interaction includes detecting a gesture performed with the pointing controller that indicates a control function of the smart device. (Item 8) Item 10. The method of item 1, wherein tracking the movement of the pointing vector includes performing tracking based on a camera integrated with the pointing controller. (Item 9) Tracking the movement of the Poynting vector includes: Detecting whether the pointing controller is indoors or outdoors; adjusting parameters of the arm model depending on whether the pointing controller is indoors or outdoors; Item 1. The method according to item 1, comprising: (Item 10) Tracking the movement of the Poynting vector includes: Detecting whether a user of the pointing controller is sitting or standing; adjusting parameters of the arm model depending on whether the user of the pointing controller is sitting or standing; Item 1. The method according to item 1, comprising: (Item 11) Tracking the movement of the Poynting vector includes: Detecting a fatigue level associated with a user of the pointing controller; adjusting parameters of the arm model in response to the detected fatigue level; Item 1. The method according to item 1, comprising: (Item 12) Tracking the movement of the Poynting vector includes: determining that the coordinate in the three-dimensional space associated with the smart device is greater than a threshold distance from the pointing controller; generating parameters of the arm model corresponding to the arm being in an outstretched position; tracking the movement based on parameters of the arm model; Item 1. The method according to item 1, comprising: (Item 13) Tracking the movement of the Poynting vector includes: determining that the coordinate in the three-dimensional space associated with the smart device is below a threshold distance from the pointing controller; generating parameters of the arm model corresponding to the arm in a relaxed position adjacent to a body; tracking the movement based on parameters of the arm model; Item 1. The method according to item 1, comprising: (Item 14) further comprising recognizing the smart device as a smart light; detecting the control interaction with the pointing controller includes detecting a swiping gesture on a touch interface of the pointing controller; generating the command includes controlling dimming of the light in response to a direction of the swiping gesture; The method according to item 1. (Item 15) 1. A non-transitory computer-readable storage medium storing instructions for controlling interaction with a smart device using a pointing controller, the instructions, when executed by one or more processors, causing the one or more processors to: acquiring sensor data from a state sensing device of the pointing controller; tracking movement of a pointing vector through three-dimensional space based on the sensor data and a stored arm model; detecting an intersection of the pointing vector with a coordinate in the three-dimensional space associated with the smart device to place the smart device in a selected state; causing an augmented reality display device to display a virtual menu associated with the smart device; When the smart device is in the selected state, detecting a control interaction with the pointing controller associated with the virtual menu; generating a command to control an operation of the smart device based on the control interaction; A non-transitory computer-readable storage medium for causing a computer to perform steps including: (Item 16) Item 16. The non-transitory computer-readable storage medium of item 15, wherein the coordinates associated with the smart device comprise a physical location of the smart device. (Item 17) Item 16. The non-transitory computer-readable storage medium of item 15, wherein the coordinates associated with the smart device comprise a location of a real or virtual representative device associated with the smart device. (Item 18) Detecting the intersection includes: generating a pointing cone having a central axis aligned with the pointing vector, an origin proximate the location of the pointing controller, and a radius that increases with distance from the origin of the pointing vector; detecting the intersection with the pointing vector in response to the pointing cone overlapping the coordinate associated with the smart device; Item 16. The non-transitory computer-readable storage medium of item 15, comprising: (Item 19) Item 16. The non-transitory computer-readable storage medium of item 15, wherein detecting the control interaction includes detecting activation of an inter-finger button of the pointing controller. (Item 20) Detecting the regulatory interaction includes: Detecting an interaction with a slider control interface of the pointing controller; navigating among different menu items within the virtual menu in response to interaction with the slider control interface; selecting a menu item in response to detecting activation of an inter-finger button of the pointing controller; Item 16. The non-transitory computer-readable storage medium of item 15, comprising: (Item 21) Item 16. The non-transitory computer-readable storage medium of item 15, wherein detecting the control interaction includes detecting a gesture performed with the pointing controller that indicates a control function of the smart device. (Item 22) Item 16. The non-transitory computer-readable storage medium of item 15, wherein tracking movement of a pointing vector includes performing tracking based on a camera integrated with the pointing controller. (Item 23) Tracking the movement of the Poynting vector includes: Detecting whether the pointing controller is indoors or outdoors; adjusting parameters of the arm model depending on whether the pointing controller is indoors or outdoors; Item 16. The non-transitory computer-readable storage medium of item 15, comprising: (Item 24) Tracking the movement of the Poynting vector includes: Detecting whether a user of the pointing controller is sitting or standing; adjusting parameters of the arm model depending on whether the user of the pointing controller is sitting or standing; Item 16. The non-transitory computer-readable storage medium of item 15, comprising: (Item 25) Tracking the movement of the Poynting vector includes: Detecting a fatigue level associated with a user of the pointing controller; adjusting parameters of the arm model in response to the detected fatigue level; Item 16. The non-transitory computer-readable storage medium of item 15, comprising: (Item 26) Tracking the movement of the Poynting vector includes: determining that the coordinate in the three-dimensional space associated with the smart device is greater than a threshold distance from the pointing controller; generating parameters of the arm model corresponding to the arm being in an outstretched position; tracking the movement based on parameters of the arm model; Item 16. The non-transitory computer-readable storage medium of item 15, comprising: (Item 27) Tracking the movement of the Poynting vector includes: determining that the coordinate in the three-dimensional space associated with the smart device is below a threshold distance from the pointing controller; generating parameters of the arm model corresponding to the arm in a relaxed position adjacent to a body; tracking the movement based on parameters of the arm model; Item 16. The non-transitory computer-readable storage medium of item 15, comprising: (Item 28) further comprising recognizing the smart device as a smart light; detecting the control interaction with the pointing controller includes detecting a swiping gesture on a touch interface of the pointing controller; generating the command includes controlling dimming of the light in response to a direction of the swiping gesture; Item 16. The non-transitory computer-readable storage medium of item 15. (Item 29) 1. A tracking device comprising: one or more processors; 1. A non-transitory computer-readable storage medium storing instructions for controlling interaction with a smart device using a pointing controller, the instructions, when executed by the one or more processors, causing the one or more processors to: acquiring sensor data from a state sensing device of the pointing controller; tracking movement of a pointing vector through three-dimensional space based on the sensor data and a stored arm model; detecting an intersection of the pointing vector with a coordinate in the three-dimensional space associated with the smart device to place the smart device in a selected state; causing an augmented reality display device to display a virtual menu associated with the smart device; When the smart device is in the selected state, detecting a control interaction with the pointing controller associated with the virtual menu; generating a command to control an operation of the smart device based on the control interaction; a non-transitory computer-readable storage medium for causing a computer to perform steps including: A tracking device comprising: (Item 30) 30. The tracking device of claim 29, wherein the coordinates associated with the smart device comprise a physical location of the smart device. (Item 31) 30. The tracking device of item 29, wherein the coordinates associated with the smart device comprise the location of a real or virtual representative device associated with the smart device. (Item 32) Detecting the intersection includes: generating a pointing cone having a central axis aligned with the pointing vector, an origin proximate the location of the pointing controller, and a radius that increases with distance from the origin of the pointing vector; detecting the intersection with the pointing vector in response to the pointing cone overlapping the coordinate associated with the smart device; Item 30. The tracking device of item 29, comprising: (Item 33) 30. The tracking device of item 29, wherein detecting the control interaction includes detecting activation of an inter-finger button on the pointing controller. (Item 34) Detecting the regulatory interaction includes: Detecting an interaction with a slider control interface of the pointing controller; navigating among different menu items within the virtual menu in response to interaction with the slider control interface; selecting a menu item in response to detecting activation of an inter-finger button of the pointing controller; Item 30. The tracking device of item 29, comprising: (Item 35) 30. The tracking device of item 29, wherein detecting the control interaction includes detecting a gesture made with the pointing controller that indicates a control function of the smart device. (Item 36) 30. The tracking device of item 29, wherein tracking the movement of the pointing vector includes performing tracking based on a camera integrated with the pointing controller. (Item 37) Tracking the movement of the Poynting vector includes: Detecting whether the pointing controller is indoors or outdoors; adjusting parameters of the arm model depending on whether the pointing controller is indoors or outdoors; Item 30. The tracking device of item 29, comprising: (Item 38) Tracking the movement of the Poynting vector includes: Detecting whether a user of the pointing controller is sitting or standing; adjusting parameters of the arm model depending on whether the user of the pointing controller is sitting or standing; Item 30. The tracking device of item 29, comprising: (Item 39) Tracking the movement of the Poynting vector includes: Detecting a fatigue level associated with a user of the pointing controller; adjusting parameters of the arm model in response to the detected fatigue level; Item 30. The tracking device of item 29, comprising: (Item 40) Tracking the movement of the Poynting vector includes: determining that the coordinate in the three-dimensional space associated with the smart device is greater than a threshold distance from the pointing controller; generating parameters of the arm model corresponding to the arm being in an outstretched position; tracking the movement based on parameters of the arm model; Item 30. The tracking device of item 29, comprising: (Item 41) Tracking the movement of the Poynting vector includes: determining that the coordinate in the three-dimensional space associated with the smart device is below a threshold distance from the pointing controller; generating parameters of the arm model corresponding to the arm in a relaxed position adjacent to a body; tracking the movement based on parameters of the arm model; Item 30. The tracking device of item 29, comprising: (Item 42) further comprising recognizing the smart device as a smart light; detecting the control interaction with the pointing controller includes detecting a swiping gesture on a touch interface of the pointing controller; generating the command includes controlling dimming of the light in response to a direction of the swiping gesture; Item 29. The tracking device according to item 29. [Brief explanation of the drawings]

[0014] The disclosed embodiments have other advantages and features that will become more readily apparent from the following detailed description and appended claims when taken in conjunction with the accompanying drawings.

[0015] [Figure 1] Figure (or "FIG.") 1 illustrates an exemplary embodiment of a smart device control system.

[0016] [Figure 2] FIG. 2 illustrates an exemplary embodiment of a pointing controller.

[0017] [Figure 3] FIG. 3 illustrates an exemplary embodiment of a tracking device.

[0018] [Figure 4] FIG. 4 illustrates an exemplary embodiment of a control processing module for processing interactions from a pointing controller.

[0019] [Figure 5]FIG. 5 illustrates an example embodiment of a process for controlling a smart device based on an interaction using a pointing controller.

[0020] [Figure 6] FIG. 6 illustrates an exemplary embodiment of a pointing controller with a form factor for grasping between adjacent fingers. DETAILED DESCRIPTION OF THE INVENTION

[0021] The figures and the following description relate to preferred embodiments by way of example only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of what is claimed.

[0022] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying figures. It should be noted that, wherever practicable, like or similar reference numbers may be used in the figures and may indicate like or similar functionality. The figures depict embodiments of the disclosed systems (or methods) for purposes of illustration only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

[0023] A pointing controller worn by a user enables intuitive control of various connected smart devices. A user may indicate a device to be controlled by pointing toward the smart device (or a surrogate object) and performing a predetermined action to select the device for control. Once selected, the user may interact with the smart device by performing various gestures or interactions with one or more control elements integrated into the pointing controller. The smart device may provide feedback using visual or auditory indicators on the smart device or by sending control signals to a pointing controller or other device in proximity to the user that provides tactile, auditory, or visual feedback.

[0024] 1 is a block diagram of a smart device control system 100, according to one embodiment. Smart device control system 100 includes a smart device 140, a tracking device 110, and a pointing controller 120, connected via a network 130. In alternative configurations, different and / or additional components may be included within smart device control system 100.

[0025] Smart device 140 comprises a connectivity-enabled electronic device capable of performing one or more functions based on various configuration settings. Examples of smart device 140 include, for example, a smart thermostat, a smart lock, a smart refrigerator, a smart speaker, a smart lighting controller, a smart medical device, or other devices. Smart device 140 generally comprises hardware including at least a processor, a storage medium, and a communication interface. In addition, smart device 140 may include one or more sensors for detecting various environmental conditions relevant to its operation. For example, a smart thermostat may include a temperature sensor and a humidity sensor. Smart device 140 generally provides one or more control outputs in response to direct user input, detected environmental conditions, detected events, or a combination thereof. For example, a smart thermostat may control a heating and / or cooling system to control ambient temperature within a desired range. A smart lighting system may control the turning on or off of connected light bulbs, the color of the light output, or the on / off pattern associated with the light bulbs. The smart device 140 may execute software or firmware that enables it to provide a level of automated control to intelligently anticipate a user's desired actions. The smart device 140 may further include an application programming interface (API) that enables other connected devices to provide command input to the smart device 140 or query for status or other information from the smart device 140. The API may be accessible through an application interface or web interface of another network-connected device, such as a smartphone, remote controller, or back-end server.

[0026] The pointing controller 120 comprises a control device that captures user gestures and interactions with control elements integrated into the pointing controller 120. In some embodiments, the pointing controller 120 has a form factor that allows it to be worn on the hand, wrist, or arm. For example, in one embodiment, the pointing controller 120 has a form factor that allows it to be grasped between two adjacent fingers, as illustrated in FIG. 6 . In other embodiments, the pointing controller 120 may have a ring form factor that allows it to be worn on a single finger. In another embodiment, the pointing controller 120 may have a knuckle duster that is worn across multiple fingers. In yet other embodiments, the pointing controller 120 comprises a band that can be worn around the wrist or arm.

[0027] The pointing controller 120 includes various sensors to enable position and orientation sensing of the pointing controller 120, and a control interface to receive direct input from a user wearing the pointing controller 120. For example, the pointing controller 120 may capture human gestures, such as pointing or waving, and may capture interactions with control elements on the pointing controller 120. Different gestures may be utilized to provide different control inputs to the smart device 140. Beneficially, the pointing controller 120 allows a user to interact with the smart device 140 in a natural manner, as will be described in further detail below.

[0028] The tracking device 110 comprises a computing device that operates in conjunction with the pointing controller 120 to process gestures and other interactions detected by the pointing controller 120 and generate control inputs for controlling the smart device 140. In one embodiment, the tracking device 110 interfaces with an API of the smart device 140 to provide the control inputs. For example, the tracking device 110 may receive position tracking data from a sensor of the pointing controller 120 and, based on the tracking data, determine the pointing direction of the pointing controller 120 and / or a particular gesture performed by a user wearing the pointing controller 120. The tracking device 110 may further obtain information about the location of the smart device 140 and / or other objects in the environment of the tracking controller 120. In addition, the tracking device 110 may receive control data indicative of user interactions with control elements on the pointing controller 120. The tracking device 110 then generates control outputs to control aspects of the smart device 140 based on the detected pointing direction, the location of the smart device 140 and / or other objects, and interactions with the control elements of the pointing controller 120.

[0029] In an embodiment, tracking device 110 comprises a smartphone, tablet, head-mounted display device, or other device that interfaces with pointing controller 120 and runs an application for interfacing with smart device 140, including a computing device that does not necessarily include a display. Alternatively, tracking device 110 may be integrated with pointing controller 120 in any of the form factors of pointing controller 120 described above.

[0030] In particular embodiments, tracking device 110 may include a display system that presents digital content, such as audio, images, video, or a combination thereof. Here, tracking device 110 may comprise an augmented reality display device, embodied, for example, as a head-mounted device with an integrated display or a separate display, such as a smartphone or tablet. In an augmented reality application, tracking device 110 enables the presentation of information and / or virtual objects along with the viewer's view of the real world. This overlay may be implemented, for example, through a semi-transparent display that allows the user to view a rendered presentation alongside the real-world view, a projection system that projects virtual objects or information onto the real-world view, or a camera feed that captures the real-world view, combines it with the overlaid presentation, and presents the combined view to the user via a display.

[0031] In an exemplary use case, the pointing controller 120 and tracking device 110 may enable a user to interact with the smart device 140 using natural and intuitive movements. For example, a user may point at the smart device 140 and perform a predetermined interaction with the pointing controller 120 to activate a function of the smart device 140, such as turning on a light, setting a thermostat, changing the speaker volume, or other function. In other embodiments, a user may control the smart device by pointing at a surrogate device, which may comprise a real-world or virtual object associated with the smart device 140. Here, for example, a user may point at a radiator associated with a smart thermostat and have the smart thermostat control the temperature of the radiator. In another example, a user may point at an incandescent light bulb associated with a smart switch and have the smart switch control the operation of the incandescent light bulb. In yet other embodiments, a virtual object in the augmented reality display of the tracking controller 110 may serve as a surrogate device for controlling the smart device 140. For example, a user may point to a virtual menu or virtual icon associated with a smart thermostat to control the functionality of the smart thermostat.

[0032] Network 130 may include any combination of local and / or wide area networks using both wired and / or wireless communication systems. In one embodiment, network 130 uses standard communication technologies and / or protocols and may include one or more of Bluetooth®, Bluetooth Low Energy, WiFi Direct, WiFi, cellular network technologies, or wired communication protocols. Network 130 may encompass different types of connections between different devices. For example, tracking device 110 may communicate with a pointing controller via a Bluetooth® connection and with smart device 140 via a WiFi connection. In some embodiments, all or some of the communication links of network 130 may be encrypted using any suitable technique.

[0033] 2 is a block diagram illustrating an exemplary embodiment of pointing controller 120. In one embodiment, pointing controller 120 includes a control unit 210, a state sensing module 220, a control element 230, a power subsystem 240, a wireless interface 250, and an output device 260. In alternative embodiments, pointing controller 120 includes additional or different components.

[0034] The state sensing module 220 comprises an electronic device for capturing data enabling sensing of the state of the pointing controller, which may include, for example, position, orientation, movement, environmental conditions, or other information about the state of the pointing controller 120. For example, in one embodiment, the state sensing module 220 may comprise a six-degree-of-freedom (6DOF) inertial measurement unit (IMU) having a gyroscope for sensing orientation or angular velocity and an accelerometer for sensing acceleration. In another embodiment, the state sensing module 220 may comprise a nine-degree-of-freedom (9DOF) IMU including a gyroscope and an accelerometer as described above, and further including a magnetometer for detecting magnetic fields (e.g., the Earth's magnetic field). The magnetometer may be utilized as a compass to detect the orientation of the pointing controller 120 relative to a geographical orientation. The IMU may further process the data obtained by direct sensing and convert the measurements into other useful data, such as calculating velocity or position from acceleration data.

[0035] In another embodiment, the state sensing module 220 may include one or more cameras that capture images of the environment suitable for tracking the position and orientation of the pointing controller 120 and correcting for any drift that may have accumulated in the IMU data. Here, the image data may be processed using a scale-invariant feature transform (SIFT) algorithm and an existing map of the space, using simultaneous localization and mapping (SLAM) techniques, using specifically created tracking markers visible by the camera, or using other image-based tracking techniques. Tracking algorithms for deriving the position and orientation of the pointing controller 120 based on captured images may be implemented on the pointing controller 120 itself, or the images may be provided to the tracking device 110 for processing to reduce the power consumption of the pointing controller 120.

[0036] In another embodiment, the condition sensing module 220 may comprise a radio frequency (RF) transceiver that detects beacons from anchor devices at known locations in the environment or from the tracking device 110. The precise location in three-dimensional space can be calculated using triangulation techniques based on the time of flight of the various beacon signals or calculated from received signal strength indications (RSSI) from an array of anchor devices.

[0037] In another embodiment, the state sensing module 220 may include a Bluetooth® direction finding module that obtains the position of the pointing controller 120 relative to the tracking device 110 or other external device (e.g., using an array of antennas in the pointing controller 120, the tracking device 110, or both to determine the direction of radio waves).

[0038] In one embodiment, the state sensing module 220 may include a barometric pressure sensor that measures atmospheric pressure. The height of the pointing controller 120 may be estimated based on the detected pressure, as described in more detail below.

[0039] In one embodiment, the state sensing module 220 may utilize Bluetooth® direction finding to obtain the position of the pointing controller 120 relative to the tracking device 110 (e.g., to determine the direction of radio waves using an array of antennas on the pointing controller 120, the tracking device 110, or both), as described in further detail below.

[0040] In further embodiments, the state sensing module 220 may include an ultrasonic pulse transmitter and / or a microphone that can be used to determine an acoustic time of flight representing the distance between the pointing controller 120 and the tracking device 110 or other reference device, as described in further detail below.

[0041] In another embodiment, the state sensing module 220 may be omitted entirely, and alternative techniques may be used to determine the pointing direction of the pointing controller 120. For example, an infrared (IR) module (not shown) may be included in place of the state sensing module 220, emitting an IR signal that is detectable by a receiver integrated with or mounted on the smart device 140 or a surrogate object (e.g., as a stick-on, low-cost, low-power device).

[0042] The control elements 230 include one or more controls for detecting control inputs from a user. The control elements 230 may include, for example, touch sensors (e.g., capacitive touch sensors), other sensors or transducers, or physical buttons, dials, switches, or other control mechanisms. In particular embodiments, the control elements 230 include a slider control interface 232 and inter-finger buttons 234. In other embodiments, different or additional control elements 230 may be employed.

[0043] The slider control interface 232 comprises a touch-sensitive pad accessible by the user's thumb or other finger. The touch-sensitive pad may comprise an array of sensing elements that detect changes in capacitance or resistance that occur in response to a touch, thereby enabling the touch-sensitive pad to detect the presence or absence of a touch and the location of the touch within the area of ​​the pad. In some embodiments, the touch-sensitive pad may additionally include a touch force sensor to enable sensing of the force applied by the touch. A user may interact with the slider control interface 232 by performing various gestures, such as tapping or swiping, with the thumb or other finger. Swiping may be performed in a forward or backward direction along the axis of the finger (e.g., parallel to the pointing direction), along an axis generally perpendicular to the axis of the finger (e.g., perpendicular to the pointing direction), or in a circular motion in a clockwise or counterclockwise direction. In the form factor of FIG. 6, the slider controller interface 232 may be positioned on the bottom side of the pointing controller 120 on a surface that extends across the bottom of the index and middle fingers.

[0044] The inter-finger button 234 may comprise a touch-sensitive and / or pressure-sensitive pad positioned so that it can be selected by pressing two fingers together. For example, in the form factor of FIG. 6 , the inter-finger button 234 may be on the inside of the curved surface so that it is adjacent to the side of the index or middle finger when the pointing controller 120 is held between the index and middle fingers. In one embodiment, the inter-finger button 234 comprises a force-sensitive resistor that detects force applied to the touch-sensitive pad. Alternatively, the inter-finger button 234 may operate similarly to the touch-sensitive pad of the slider controller interface 232 discussed above. Due to its location, the inter-finger button may be used to detect a “pinching gesture” in which the middle and index fingers (or other pair of adjacent fingers) are pressed toward each other and at least a threshold pressure is applied to the touch-sensitive and / or pressure-sensitive pad of the inter-finger button.

[0045] The power subsystem 240 stores and supplies power to the pointing controller 120. For example, the power subsystem 240 may include a battery, a charging circuit for charging the battery, and one or more voltage regulators for controlling the voltage supplied to other components of the pointing controller 120. In one embodiment, the power subsystem 240 may control the pointing controller 120 to switch between different power modes (e.g., a full power mode, a low power mode, and a sleep mode) to efficiently utilize the battery.

[0046] The wireless interface 250 communicates wirelessly with the tracking device 110 over the network 130. In one embodiment, the wireless interface 250 may comprise, for example, a Bluetooth interface, a Bluetooth low energy interface, a WiFi link, or other wireless interface. The wireless interface 250 may communicate directly with the tracking device 110 over a peer-to-peer connection, or may communicate with the tracking device 110 through one or more intermediate devices via a local area network, a wide area network, or a combination thereof. In one embodiment, the wireless interface 250 may communicate directly with the smart device 140.

[0047] Output device(s) 260 include various devices for providing output from pointing controller 120 in response to control signals from tracking device 110 or directly in response to actions on pointing controller 120. Output device(s) 260 may include, for example, a haptic feedback device (e.g., a linear resonant actuator or an eccentric mass vibration motor), one or more light-emitting diodes (LEDs), or an audio output device.

[0048] The control unit 210 processes inputs from the condition sensing module 220, the control elements 230, the power subsystem 240, and the wireless interface 250 and controls various functions of the pointing controller 120. In one embodiment, the control unit 210 comprises a processor and a non-transitory computer-readable storage medium that stores instructions that, when executed by the processor, cause the processor to perform the functions attributed to the controller 210 described herein. Alternatively, or in addition, the control unit 210 may comprise digital logic embodied as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0049] The control unit 210 may process raw data from the state sensing module 220 and the control elements 230 to detect motion or interaction events, and then transmit the processed events to the tracking device 110 instead of the raw data, thereby reducing the bandwidth over the communication link 130. For example, the control unit 210 may obtain raw accelerometer, gyroscope, and / or magnetometer data from the IMU of the state sensing module 220 and apply a sensor fusion algorithm to determine the detected orientation (e.g., roll, pitch, and yaw values). Furthermore, the control unit 210 may process the raw touch data (e.g., capacitance or resistive sensing), perform processing such as analog-to-digital conversion and filtering, and generate touch detection events indicating the detection of a touch and the position or force of the touch, which are transmitted to the tracking device 110.

[0050] Alternatively, the control unit 210 may send only raw data from the state sensing module 220 and the control element 230 to the tracking device 110, and the processing described above may instead be performed on the tracking device 110. In another embodiment, the control unit 210 may send both raw and processed event data to the tracking device 110.

[0051] In some embodiments, other components of the pointing controller 120 may be coupled to the control unit 210 via a data bus, such as a serial peripheral interface (SPI) bus, a parallel bus, or an I2C bus. Additionally, components of the pointing controller 120 may generate interrupt signals detectable by the control unit, allowing for low latency response to user input.

[0052] 3 is a block diagram illustrating an embodiment of a tracking device 110. In the illustrated embodiment, the tracking device 110 comprises a processor 310, a storage medium 320, a wireless interface 330, sensors 340 including a camera 345 and a state sensing module 342, and an output device 350 including a display 352 and an audio output device 354. Alternative embodiments may include additional or different components. For example, in some embodiments, a tracking device 110 without display capabilities does not necessarily include a display, a camera 345, or a content presentation module 322.

[0053] The wireless interface 330 communicates wirelessly with the pointing controller 120 via the network 130. In some embodiments, the wireless interface 330 may comprise, for example, a Bluetooth interface, a WiFi interface, or both. The wireless interface 330 may communicate directly with the pointing controller 120 via a peer-to-peer connection, or may communicate with the pointing controller 120 through one or more intermediate devices via a local area network, a wide area network, or a combination thereof. The wireless interface 330 may further communicate with the smart device 140 via the network 130.

[0054] In one embodiment, the wireless interface 330 may receive transmissions and commands to the pointing controller 120 and perform actions such as controlling the pointing controller 120 to enter various power modes, requesting detailed information about the status of the pointing controller 120 such as battery status, temperature, or other diagnostic information, updating the firmware of the pointing controller 120, configuring haptic actuators on the pointing controller 120 to respond directly to events detected on the pointing controller 120, such as activating haptic actuators on the pointing controller 120 according to specific vibration patterns, or activating specific buttons or control inputs on the pointing controller 120. The wireless interface 330 may also periodically receive transmissions from the pointing controller 120 containing information such as tracking data from the state sensing module 220 of the pointing controller 120, control data from the control elements 230 of the pointing controller 120, or battery information from the power subsystem 240 of the pointing controller 120.

[0055] Sensors 340 detect various conditions associated with the operating environment of tracking device 110. For example, camera 345 captures real-time video of the real-world environment within the view of tracking device 110. Image data from the camera may be combined with virtual objects or information to present an augmented reality view of the world. Camera 345 may include a traditional image camera, a non-visual camera such as a depth camera or LIDAR camera, or a combination thereof.

[0056] The sensors 340 may also include a state sensing module 342 to sense movement and orientation of the tracking device 110. The state sensing module 342 may include similar components and may operate similarly to the state sensing module 220 of the pointing controller 120 discussed above. For example, the state sensing module 342 may include one or more of an IMU, a radio frequency (RF) transceiver, a Bluetooth® direction finding module, a barometric pressure sensor, an ultrasonic pulse transmitter and / or a microphone, or other sensors.

[0057] Sensors 340 may optionally include other sensors for detecting various conditions, such as, for example, a location sensor (eg, a global positioning system) or a temperature sensor.

[0058] The output device 350 includes various devices for providing output from the tracking device 110 for presenting digital content. In one embodiment, the output device 350 may include at least a display 352 and an audio output device 354. In alternative embodiments, the output device 350 may include additional output devices for providing feedback to the user, such as a haptic feedback device and one or more light-emitting diodes (LEDs). The audio output device 354 may include one or more integrated speakers or a port for connecting one or more external speakers to play audio associated with the presented digital content. The display device 352 comprises an electronic device for presenting image or video content, such as an LED display panel, an LCD display panel, or other type of display. The display device 352 may be configured in a manner to present digital content in an immersive manner, to present a simulation of a virtual or augmented reality environment. For example, the display device 352 may comprise a stereoscopic display that presents different images to the left and right eyes, creating the appearance of a three-dimensional environment. In one embodiment, the display device 352 may present digital content that combines rendered graphics depicting virtual objects and / or environments with content captured from the camera 345, enabling an augmented reality presentation with virtual objects overlaid on a real-world scene.

[0059] The storage medium 320 (e.g., a non-transitory computer-readable storage medium) stores instructions executable by the processor 310 to perform the functions attributed to the tracking device 110 described herein. In one embodiment, the storage medium 320 includes a content presentation module 322 and a control processing module 324. In alternative embodiments, the storage medium 320 may include additional or different modules.

[0060] The content presentation module 322 presents digital content via the display 352 and / or audio output device 354. The displayed content may comprise a virtual reality or augmented reality environment in three-dimensional space. The displayed content may include virtual objects that may be combined with real-world images captured by the camera 345. The content presentation module 322 may adapt its content based on information received from the control processing module 324.

[0061] The control processing module 324 processes input received from the pointing controller 120 via the wireless interface 330 and generates processed input data that may generate control commands for the smart device 140 and / or control the output of the content presentation module 322. For example, the control processing module 324 may track the position of the pointing controller 120 within the virtual environment displayed by the content presentation module 322 based on tracking data received from the state sensing modules 220, 342. Furthermore, the control processing module 324 may process input from the control element 230 and detect gestures performed on the control element 230. Based on the detected tracking of the pointing controller 120 and the detected gestures, the control processing module 324 may determine commands to output to the smart device 140 and / or cause the content presentation module 322 to update the presentation in response to the actions. Examples of the control processing module 324 are described in further detail below.

[0062] 4 illustrates an example embodiment of the control processing module 324. The control processing module 324 includes a tracking module 402, an arm model 404, a gesture recognition module 406, an object interaction module 408, a menu navigation module 410, and a calibration module 412. Alternative embodiments may include different or additional modules.

[0063] The tracking module 402 infers the position and orientation of the pointing controller 120 relative to the user's hand. In embodiments in which the tracking device 110 is integrated into a head-mounted display, the position of the actor's head can be inferred directly from the position of the tracking device 110 because the tracking device 110 is fixed relative to the head position. In particular, the tracking module 402 determines the orientation of the pointing controller 120 based on tracking data from the state sensing module 220 and obtains the position and orientation of the tracking device 110 relative to the environment based on sensor data from the tracking device 110 (e.g., tracking data from the state sensing module 342). The tracking module 402 then estimates the position of the pointing controller 120 relative to the environment based on the orientation of the pointing controller 120, the position and orientation of the tracking device 110, and an arm model 404 that models the posture of the user operating the pointing controller 120.

[0064] Based on the orientation and calculated position of the pointing controller 120, the tracking module 402 generates and continuously updates a pointing vector that originates from the position of the pointing controller 120 and extends in a direction corresponding to the detected orientation. In the case of a pointing controller 120 worn on one or more fingers, the pointing vector may extend along a central axis through the pointing controller 120 aligned with the fingers. The pointing vector may be defined according to three-dimensional coordinates within the virtual environment tracked by the tracking device 110. Thus, the pointing vector provides a pointing direction relative to a scene within the virtual environment. The pointing vector may comprise, for example, a pair of angles including a first angle relative to the ground plane (i.e., pitch angle) and a second angle relative to a vertical plane perpendicular to the ground plane (i.e., yaw angle). In some embodiments, an orientation angle about the axis of the pointing vector (i.e., roll angle) may also be tracked along with the pointing vector.

[0065] In one embodiment, the tracking module 402 may calculate a pointing cone around the pointing vector, where the cone originates at the pointing controller 120, has a central axis aligned with the pointing vector, and has a diameter that increases with distance from the pointing controller 120. The cone angle may be adjustable by a user or developer or may be a hard-coded parameter. Additionally, the cone angle may be automatically updated based on the context of the detected interaction with the object. For example, when interacting with an environment with many objects close together, the cone angle may be automatically reduced relative to an environment with fewer objects that are far apart. The tracking module 402 updates the pointing vector, cone of point, and orientation angle as the user moves the pointing controller 120.

[0066] In one embodiment, the tracking module 402 performs tracking based, at least in part, on IMU data from the state sensing module 220 of the pointing controller 120 .

[0067] In one embodiment, the tracking module 402 may perform tracking based, at least in part, on atmospheric pressure data from the state sensing module 220 and / or the barometric pressure sensor of the tracking device 110. For single-ended sensing, a reference pressure value may be determined corresponding to a baseline height during a calibration process. The tracking module 402 may subsequently obtain barometric pressure readings and calculate a vertical offset from the baseline height based on changes in pressure. In another embodiment, the tracking module 402 uses differential sensing to estimate the vertical position of the pointing controller 120. In this embodiment, a differential pressure is calculated between barometric pressure measurements obtained from a pressure sensor of the pointing controller 120 and a pressure sensor within the external tracking device 110. The differential sensor measurements may be filtered to compensate for natural atmospheric variations due to weather or other factors.

[0068] In another embodiment, the tracking module 402 may track the pointing controller 120 based in part on the relative RSSI of wireless signals received at both the pointing controller 120 and the tracked device 110. The relative RSSI may be used to estimate the distance between the tracked device 110 and the pointing controller 120. The distance estimation may be further improved by modeling the emission and sensitivity patterns of antennas within the pointing controller 120 and the tracked device 110 (or between multiple devices such as the pointing controller 120, an AR headset, and a mobile phone).

[0069] In another embodiment, the tracking module 402 may utilize Bluetooth® direction finding data to obtain the position of the pointing controller 120 relative to the tracking device 110 (e.g., to determine the direction of radio waves using an array of antennas in the pointing controller 120, the tracking device 110, or both). In one embodiment, the roll and pitch components of the pointing direction are obtained from an integrated IMU, and the yaw direction is obtained from Bluetooth® direction finding. In another embodiment, the roll, pitch, and yaw may be obtained from other components of the pointing controller 120, and the Bluetooth® direction finding may be used to perform corrections if discrepancies between the other measurements exist. In another embodiment, a statistical error profile may be determined (e.g., if the error is consistent over several relative orientations), and information about the relative orientation may be determined based on the statistical error profile. In yet another embodiment, Bluetooth® direction finding may be utilized to determine multiple points on a rigid body (e.g., from two or more antenna arrays within the AR viewer), and may additionally estimate the distance between the pointing controller 120 and the tracking device 110 without necessarily relying on RSSI.

[0070] In a further embodiment, the tracking module 402 may perform tracking based on acoustic time-of-flight, which represents the distance between the ultrasonic pulse transmitter and microphone in the pointing controller 120 and the tracking device 110. In one embodiment, the tracking module 402 uses the estimated distance from the acoustic time-of-flight in the tracking calculations only when the detected distance is below a maximum threshold distance (e.g., 1.5 meters). In another embodiment, Doppler shift effects may be detected to estimate the velocity of the pointing controller 120 relative to the tracking device 110. Here, the velocity estimate may be used to compensate for errors in velocity estimates determined from IMU data using dead reckoning. In another embodiment, the distance estimated based on acoustic time-of-flight may be adjusted based on barometric pressure data to compensate for variations in the speed of sound due to pressure differences.

[0071] The parameters of the arm model 404 may be determined during an initialization process and may be updated during tracking, as will be described below. Input parameters for the arm model 404 may include, for example, the user's height, a standardized model of human proportions, a joint angle model, and various operating conditions that may change over time. The user's height may be obtained manually from the user during the initialization process in response to a user prompt requesting the user to enter their height. Alternatively, height may be estimated automatically based on the estimated position of the tracking device 110 relative to the ground. For example, visual analysis may be performed on image data captured by the camera 345 of the tracking device 110 to estimate height. Based on the user's height, the tracking module 402 may perform a lookup in a pre-populated lookup table that maps height to hand, forearm, arm, shoulder, and neck sizes based on a standardized model of human proportions. Then, using the combined dimensions of the human body model and the detected orientation of the pointing controller 120, the tracking module 402 can apply a joint angle model to predict the relative probabilities of various arm poses. The most probable pose may be selected, and the tracking module 402 may estimate the position of the pointing controller 120 relative to the tracking device 110 from the pose.

[0072] In one embodiment, additional information derived by tracking module 402 can be incorporated to more accurately predict the user's pose and eliminate undesirable outcomes. For example, if the most likely predicted pose generated by a joint angle model predicts the user's arm intersecting with the known location of a detected real-world object (an impossible outcome), tracking module 402 may instead select the next most probable prediction that does not predict the arm intersecting with the detected object.

[0073] In another embodiment, the tracking module 402 may utilize information about the user's current location and / or movement history to improve tracking accuracy by applying different parameters of the arm model 404 in different contexts. For example, because people tend to use a wider range of gestures when outdoors than when indoors, the tracking module 402 may adjust the parameters of the arm model 404 depending on whether the user is indoors or outdoors. The tracking module 402 may detect whether the user is indoors or outdoors based on image analysis of captured images or other sensor data. In one technique, the tracking module 402 may determine whether the user is indoors or outdoors based on the presence or absence of a ceiling surface within a certain distance of the user (e.g., within 5 meters above the user), which may be detected based on image analysis from captured images or from other sensors. In another embodiment, the tracking module 402 may measure the number of flat surfaces within a specified distance of the tracking device 110 and determine that the user is indoors if the number exceeds a predetermined threshold, and determine that the user is outdoors if the number does not exceed the threshold. In yet another embodiment, a location sensor (e.g., a global positioning system device) may be used to determine the geographic location of the tracking device 110. Utilizing map data from a map service, the tracking module 402 may then determine that the user is indoors if the location corresponds to a building, or otherwise determine that the user is outdoors. In yet another embodiment, the radio signal strength of a radio signal (e.g., a GPS signal or a cellular data signal) received by the tracking device 110 from a remote source may be used to determine whether the user is indoors or outdoors. For example, when the radio signal strength is above a predetermined threshold, the tracking module 402 determines that the user is outdoors, and when the radio signal strength is below the threshold, the tracking module 402 determines that the user is indoors. In yet another embodiment, the tracking module 402 may perform an analysis of the brightness and / or wavelength of local light sources detected by the camera 345 to detect whether the user is indoors or outdoors.For example, bright ambient light with a daylight color temperature indicates that the user is likely outdoors, while a color temperature consistent with an incandescent light bulb indicates that the user is indoors.

[0074] In another embodiment, the parameters of the arm model 404 may be adapted based on whether the user is sitting or standing, where the tracking module 402 may determine whether the user is sitting or standing by detecting the height of the tracking device 110 relative to the ground and detecting whether the height is significantly below (e.g., above a threshold difference) the user's standing height, as described above.

[0075] In one embodiment, the tracking module 402 may further estimate a user's fatigue level to better predict the position of the pointing controller 120. Here, the tracking module 402 may model the fatigue level by tracking the amount of time the user spends with their wrist at a certain threshold height, where the level of fatigue increases over time. Because a user may prefer to keep their arm lower as fatigue increases, the parameters of the arm model 404 may cause the tracking module 402 to adjust the detected position downward as the predicted fatigue level increases. In one embodiment, the tracking module 402 may apply a machine learning approach to model the fatigue characteristics of a particular user.

[0076] In one embodiment, the tracking module 402 may use image data from the camera 345 to sense the position of the pointing controller 120, hand, forearm, or arm. The tracking module 402 may use the sensed position to recalibrate the orientation and position of the pointing controller 120 relative to the tracking device 110 and account for accumulated drift in the tracking data as described in further detail below. Additionally, the tracking module 402 may apply the sensed position from the image data to improve the accuracy of the arm model 404 by updating estimated parameters such as arm length or predicted joint angles. The arm position may further be estimated from the integration of successive acceleration values ​​from the accelerometer of the state sensing module 220.

[0077] In one embodiment, the tracking module 402 may further utilize position information about virtual objects (or real-world objects at known locations) to infer the position of the pointing controller 120. For example, if the object is close (e.g., below a threshold distance), it may be inferred that the hand is in a relaxed position close to the body. On the other hand, if the object is far away (e.g., above a threshold distance), it may be inferred that the hand is in an outstretched position. The tracking module 402 may adjust parameters of the arm model 404 based on the inferred arm position.

[0078] In cases where the tracking device 110 is not head-mounted (e.g., the tracking device 110 is embodied as a handheld smartphone or tablet), the position of the user's head may be unknown relative to the tracked position of the tracking device 110. In this case, calibration techniques may be applied to estimate the user's head position relative to the position of the tracking device 110. For example, in one embodiment, a user interface on the tracking device 110 prompts the user to touch the tracking device 110 to the user's nose during a calibration phase of the application. Alternatively, the camera of the tracking device 110 may capture an image of the user's face, and a face tracking algorithm may be applied to detect the center point of the face that corresponds to the initial head position. In yet another embodiment, the vertical component of the head position can be obtained manually by prompting the user to enter their height, or the user's height may be obtained from a linked health tracking application or online service accessible by the tracking device 110.

[0079] Once calibrated, the tracking module 402 may estimate the vertical component of the head position to be fixed in three-dimensional space, and vertical movement of the tracking device 110 may be tracked in three-dimensional space relative to this position. Alternatively, the camera 345 of the tracking device 110 may capture images that are processed to detect changes in terrain height. The user's estimated head position may be updated based on detected changes in terrain height so that it is at an approximately fixed vertical position above the ground.

[0080] In the horizontal plane, the tracking module 402 may estimate the head position to be a fixed horizontal offset from the tracked position of the tracking device 110. Thus, as the tracking device 110 moves and rotates in the horizontal plane, the head position is estimated at a fixed horizontal distance from the tracked position of the tracking device 110.

[0081] Recalibration may be performed when the user changes from a sitting to a standing position (or vice versa). This change may be indicated manually by the user or may be detected automatically when an appropriate shift in the vertical position of the tracking device 110 (and / or pointing controller 120) is detected. For example, the camera 345 of the tracking device 110 may capture images that may be processed to detect the height of the tracking device 110 relative to the ground and used to detect when the user sits down or stands up.

[0082] In an alternative embodiment, the user's head position may be assumed to be completely fixed, where instead of estimating the head position in the horizontal plane to track horizontal movement of the tracking device 110 at a fixed offset, the head position may instead be estimated to remain at both a fixed vertical and horizontal position in three-dimensional space without tracking movement of the tracking device 110.

[0083] In yet another embodiment, a hybrid model combining the techniques described above may be used. Here, an initial head location relative to the tracking device 110 is first calibrated using the calibration techniques described above (e.g., by prompting the user to touch the display device to their nose). The tracking module 402 may initially be set to a “stationary” mode, which estimates the head position so that it is maintained at a fixed position in three-dimensional space. The position of the tracking device 110 is tracked using the state sensing module 342 as it moves through three-dimensional space, and the distance between the tracking device 110 and the fixed estimated head position is calculated. When the distance between the estimated head location and the tracking device 110 exceeds a predetermined activation radius (e.g., approximately equal to the estimated length of the user's fully extended arm), the tracking module 402 switches to a “walking” mode. In the “walking” mode, the head position is instead estimated to be a fixed distance behind the detected position of the tracking device 110. When the tracking device 110 detects that the movement has fallen below a threshold speed and remains below the threshold speed for a threshold time period, the tracking module 402 switches back to "steady mode" in which the estimated position of the head becomes fixed and is no longer updated based on the position of the tracking device 110.

[0084] Alternatively, when in "walking mode," the head position relative to the tracking device 110 may instead be estimated using a spring-mass or spring-mass-damper model. In this embodiment, the estimated distance of the head behind the detected position of the display tracking 110 may vary over time, but stabilizes to a fixed position when the tracking device 110 is stable over a long period of time. When the tracking device 110 detects that the distance between the smartphone and the head drops below the deactivation radius, in this embodiment, the tracking module 402 switches back to "stationary" mode.

[0085] The gesture recognition module 406 detects gestures performed by a user using the pointing controller 120. Examples of gestures may include, for example, moving the pointing controller 120 in a predetermined motion or interacting with the slider control interface 232 and / or inter-finger buttons in a particular manner (e.g., single tapping, double tapping, maintaining prolonged contact, or a combination of interactions in a particular pattern). Here, a pinch gesture may be detected when a user presses their middle finger and index finger (or other fingers in contact with the pointing controller 120) together, thereby placing one or more fingers in contact with the inter-finger buttons 234 on the pointing controller 120 with a threshold amount of pressure for at least a threshold time period. The pinch gesture may be released by separating the fingers or by relaxing the applied pressure. In some embodiments, the gesture recognition module 406 may capture the force or time period of the pinch gesture and may take different actions depending on these captured parameters. A swiping gesture may be detected when a user performs a swiping movement on the slider controller interface 232. This gesture may typically be performed with the thumb (or other fingers) on the hand wearing the pointing controller 120, but may alternatively be performed with the fingers on the opposing hand. Here, the swiping gesture may comprise a linear swiping gesture along a line parallel to one or more fingers holding the pointing controller 120 in either direction, or along a line approximately perpendicular to one or more fingers in either direction. Alternatively, the swiping gesture may comprise a radial swiping gesture performed in a clockwise or counterclockwise direction around a reference point in the plane of the slider controller interface 232. In some embodiments, the gesture recognition module 408 may capture the force, speed, or distance of the swiping gesture and take different actions depending on these captured parameters.Other types of gestures may also be recognized to perform various tasks.

[0086] The object interaction module 408 determines when the pointing vector or cone intersects with an object, which may correspond to the smart device 140, a different real-world object at a known location, or a virtual object in a scene being displayed on the display of the tracking device 110. For example, the object interaction module 408 stores coordinates representing locations occupied by real-world and virtual objects and detects when the pointing vector or cone intersects with coordinates occupied by one of the objects. In cases where the pointing vector or cone intersects with multiple objects, the object interaction module 406 may default to selecting the object closest to the pointing controller 120. In another embodiment, the tracking module 402 may intelligently predict whether the user intends to point to a nearby object (e.g., less than 5 meters away) or a distant object (e.g., more than 5 meters away) when the pointing vector intersects with multiple objects. For example, the tracking module 402 may infer that the user intends to point to a distant object when the arm is detected to be substantially aligned with the user's eye and the arm is fully extended. The tracking module 402 may infer that the user intends to point at a nearby object when the arm is bent and held in a position below eye level.

[0087] In some embodiments, a visual indicator (e.g., a visual outer glow or halo effect, a shaded outline, a bounding box, or the like) is displayed within the augmented reality display in association with the object being pointed at. Optionally, detailed information about the selected object, such as, for example, an object identifier, the distance from the pointing controller 120 to the selected object, the object's status, etc., may also be displayed. Additionally, when an object is pointed at, the object interaction module 406 may vibrate a haptic motor in the pointing controller 120 to provide physical feedback of the action. Alternatively, other visual or audio feedback may also be provided to indicate when an object is selected.

[0088] The object interaction module 408 may determine a command associated with a gesture or control interaction performed when a user points at a particular object. For example, the object interaction module 408 may detect a confirmation interaction (e.g., a pinch gesture activating the inter-finger button 234) when a user points at an object. Here, the object interaction module 408 may confirm the selection of the smart device 140 when the interaction is performed when the smart device 140 or a proxy object associated with the smart device 140 is selected based on a pointing vector or cone. In response to confirming the selection, the object interaction module 408 may establish a connection to the smart device 140 via the network 130. The object interaction module 408 may then detect a gesture or selection of a control element performed by the user to control various functions of the smart device 140. Here, a set of predetermined gestures may be associated with different control functions of the smart device 140. The mapping of gestures to those functions may be configured in an intuitive manner to enable natural control of the smart device 140. For example, to control a smart thermostat, a user may point or move their hand upward, rotate their hand clockwise, or tilt their hand upward to increase the temperature, and point or move their hand downward, rotate their hand counterclockwise, or tilt their hand downward to decrease the temperature. Alternatively, a slider control interface 232 on the pointing controller 120 may control the temperature based on the direction of the swipe.

[0089] In another exemplary interaction, a user may perform gestures to control a smart lighting dimmer to increase or decrease the light output intensity or change the color temperature of a light bulb. For example, the tracking device 110 may detect when the pointing controller 120 is pointing at a light. The pointing controller 120 triggers a haptic, audible, or visual feedback signal when the pointing vector intersects with the light's location, indicating that the light is selected. The pointing controller 120 may detect when the user holds their thumb on the slider controller interface 232 and drags their thumb in one direction to increase the brightness of the light and in the opposite direction to decrease the brightness. Releasing the thumb may stop the brightness change, and moving the pointing vector away from the light may cause the light to become deselected, such that further interactions with the control element 230 no longer generate a control signal for the light.

[0090] In some embodiments, a deselect action may be performed to deselect a selected smart device 140. In some embodiments, deselection may be performed by pointing away from the smart device 140 or an associated proxy object. In some embodiments, haptic or other feedback may be output to confirm that the interaction is complete.

[0091] The menu navigation module 410 generates a menu presented on the display device 352 in response to the smart device 140 or an associated proxy object being selected, or another action or combination of actions, such as, for example, tapping a slider control interface while the object is selected. The menu may allow the user to view and / or modify advanced configuration settings associated with the smart device 140. In an embodiment, a wheel or slider interface may be displayed to allow the user to quickly modify parameters using swiping gestures.

[0092] The calibration module 412 performs a calibration process to calibrate the pointing controller 120 to initialize the relative position and orientation of the pointing controller 120 to the position and orientation in the virtual environment presented by the content presentation module 362. The roll and pitch of the pointing controller 120 can be detected from the state sensing module 220 using the detected direction of gravity (as sensed by the state sensing module 220) mapped to a downward direction along the vertical axis of the virtual environment. The horizontal direction (yaw) of the pointing controller 120 can be sensed relative to a reference direction during calibration using various techniques. This reference direction may be aligned with the forward direction of the tracking device 110 during the calibration process.

[0093] In one embodiment, a magnetometer in the state sensing module 220 of the pointing controller 120 may act as a compass to detect magnetic north. A magnetometer in the tracking device 110 may similarly detect magnetic north, and the calibration module 412 may perform calibration to align these reference directions.

[0094] In another embodiment, the location and orientation of the pointing controller 120 can be detected based on image (visual or depth) analysis performed on one or more images captured by a camera of the tracking device 110 or other external camera. The calibration module 412 may then perform calibration using the location and position determined from the detected tracking and image data.

[0095] In another embodiment, the calibration module 412 performs calibration by instructing the user to point straight ahead and then perform a specific gesture (e.g., a double tap on the slider control interface 232 while also pressing the inter-finger button 234). Unintentional actions when the pointing controller 120 is not approximately horizontal, as detected by the state sensing module 220, may be rejected by ignoring the gesture when it is detected. The calibration module 412 may then set the direction as a reference direction that is mapped to the straight-ahead direction in the virtual environment.

[0096] In another embodiment, calibration may be performed by instructing the user to point at a small number of real-world objects in locations that are known or can be detected from images captured by the image processing device. Here, to determine when the user is pointing at a target, the pitch of the pointing controller 120 should approximately match the pitch vector to the target, and in addition, the pointing controller 120 should be held approximately stationary. The calibration module 412 may then perform the calibration using the known locations of these objects in the virtual environment. In one embodiment, this calibration phase may be performed as part of a user tutorial to train the user how to use the pointing controller 120 to interact with objects.

[0097] In yet other embodiments that do not rely on the state sensing module 220 of the pointing controller 120 (eg, embodiments that use IR-based detection of pointing direction), the calibration module 412 may be omitted.

[0098] In one particular embodiment, the tracking device 110 is configured to display a target object located far away (to minimize perspective error), and a prompt is displayed to instruct the user to point to the target object. The calibration module 412 detects when the pointing controller 120 is approximately stationary (e.g., by detecting that the angular rotation rate is below a predetermined threshold) and determines that the current pointing direction is toward the target object. In some embodiments, the tracking device 110 may provide visual indicators to guide the user through calibration. For example, the tracking device 110 may display a visual indicator (e.g., a progress bar animation, a change in the size of the visual indicator, etc.) that begins to “fill in” when the pointing controller 120 is stationary for a short period of time and, in addition, the pitch of the pointing controller 120 approximately matches the pitch of the target relative to the user. During this time, the calibration module 412 records the detected orientation of the pointing controller 120 and determines the difference in the yaw (orientation) of the pointing controller 120 relative to the yaw of the tracking device 110. If the user moves the pointing controller 120 during the calibration period or if the pitch falls outside the allowed range, progress is reset. Once the calibration process is complete, the target object may be removed from the display and the calibration values ​​are stored. The calibration process described above can be repeated multiple times with target objects at different yaw (orientation) and / or pitch to improve accuracy. The calibration process can additionally be performed with target objects at different depths, or by instructing the user to remain pointed in one direction but keep targets located on the periphery of their field of view, to improve calibration.

[0099] In another embodiment, the tracking device 110 may display an outline of an image of the pointing controller 120 and instruct the user to place the tracking device 110 on a flat, horizontal surface and then place the pointing controller 120 on the display screen of the tracking device 110 aligned with the outline of the image. The calibration module 412 detects when the pitch of the pointing controller 120 falls below a threshold angle and when both the tracking device 110 and the pointing controller 120 are held stationary for a threshold time period. When these conditions are detected, the calibration module 412 stores the difference between the detected yaw of the pointing controller 120 and the tracking device 110 as a calibration offset. In operation, this calibration offset is subtracted from the yaw measurement of the pointing controller 120.

[0100] Once calibrated, the calibration module 412 may allow the user to verify the calibration by displaying a test target, allowing the user to ensure that the calibration was performed correctly. In another embodiment, the calibration module 412 may perform continuous automatic calibration during use. The calibration module 412 may store a set of focal points associated with different types of objects. Here, the focal point of an object represents the point on an object of a given object type at which the user is likely to have a pointing preference when attempting to point to that object type. For simple shapes, the focal point may be calculated by calculating the center of mass of the object, assuming uniform density. For complex shapes, the focal point may be calculated by calculating the center of mass of a convex hull that "encloses" the shape. For other types of functional objects, the focal point may be assigned manually based on object type or may be learned for different types of objects using an external tracking system. For these types of objects, the focal point may be biased toward the point of interaction. For example, for a computer monitor, the focal point may correspond to the center of the screen, ignoring the stand. For a bicycle, the focal point may be offset from the center mass towards a point closer to the handlebars, for a piano, the focal point may be offset from the center of mass towards a point closer to the keys, for a door, the focal point may be offset from the center of mass towards a point closer to the handle / press plate.

[0101] In some embodiments, the focus of an object may change with distance. For example, from a long distance, people will likely point to the center of the object, regardless of the object's purpose. Thus, in some embodiments, the center of mass of an object may be used as the focus when the object is farther away than a predetermined distance. However, when closer to the object, people tend to move toward the point of interaction on functional objects, but may continue to point to the center of mass for simpler objects. Thus, in some embodiments, a pre-assigned focus based on object type may be used when the object is closer than a predetermined distance. Each time an object is selected, the calibration module 412 may determine the difference between the direction of the object's focus and the actual pointing direction of the pointing controller 120 at the moment the object is selected. If these differences (and in particular the yaw component) are consistently biased in one direction, the calibration module 412 may detect miscalibration. In some embodiments, miscalibration is detected only once a sufficient confidence level is reached, such as after the yaw components of several object selections are consistently biased in one direction. In response to detecting a miscalibration, the calibration module 412 can adjust the calibration parameters to correct the miscalibration. This recalibration may be performed instantaneously or applied gradually over several seconds (to prevent the user from experiencing any "jumps").

[0102] FIG. 5 is a flowchart illustrating an example embodiment of a process for controlling a smart device 140 using a pointing controller 120. The tracking device 110 acquires (502) tracking data associated with the position and orientation of the pointing controller 120. The tracking data may include motion data from which a position may be derived, and may be in the form of IMU data, image data, RF beacon data, or a combination thereof. The tracking device 110 tracks (504) a pointing vector associated with a pointing direction of a user wearing the pointing controller 120 based on the tracking data. The pointing vector may comprise a line or cone that widens with distance from the pointing controller 120 along the axis of the pointing direction. The tracking device 110 detects (506) a selection of the smart device 140 based on detecting an intersection of the tracked pointing vector and an object location associated with the smart device 140, placing the smart device 140 in a selected state. Here, the object location may be the location of the smart device 140 itself, or the location of a real-world or virtual proxy object associated with the smart device 140. In one embodiment, the tracking device 110 confirms the selection in response to detecting a predetermined interaction (e.g., a gesture or selection of an interface control on the pointing controller) when the pointing direction intersects with the object location. In response to the smart device 140's selection, the tracking device 110 establishes (508) a connection to the smart device 140, which enables it to communicate commands to and / or receive status information from the smart device 140. The tracking device 140 generates (510) commands for controlling the smart device 140 based on the detected interaction. For example, the tracking device 110 detects one or more interactions (e.g., a predetermined gesture or interaction with an interface control) performed using the pointing controller and determines a mapping of the interaction to a control command associated with the smart device 140.The tracking device may subsequently detect an interaction associated with the pointing controller 120 to deselect the smart device 140 and return the smart device 140 to an unselected state (512). For example, the tracking device 110 may deselect the smart device 140 in response to detecting the user pointing away from the direction of the object location associated with the smart device 140 or performing a different predetermined gesture associated with deselecting the smart device 140.

[0103] In alternative embodiments, one or more components of the control processing module 324 may be implemented on the pointing controller 120 instead of on the tracking device 110. For example, in one embodiment, the functions of the tracking module 402 and the gesture recognition module 406 may instead be performed by the pointing controller 120. In this embodiment, tracking results and detected gestures may be communicated directly to the tracking device 110 instead of communicating raw tracking and control element data. Alternatively, in other embodiments, one or more components of the control processing module 324 may be implemented on a separate, communicatively coupled device. For example, a mobile device, a personal computer, or a game console may receive raw tracking and control element data from the pointing controller 120, perform the functions of the control processing module 324 to process the raw data, send the processed control information to the tracking device 110, and cause the tracking device 110 to update the display on the display device 352. In yet another embodiment, one or more components of the control processing module 324 may be implemented on a remote server (e.g., a cloud server) that is communicatively coupled to the pointing controller 120 and the tracking device 110.

[0104] In yet other embodiments, the tracking device 110 is omitted, and the pointing controller 120 determines the smart device 140 or surrogate device with which the user intends to interact by using modulated infrared (IR) signals. Here, a transmitter located within the pointing controller 120 transmits a modulated signal in a direction approximately aligned with the finger and with a sufficiently narrow beam angle to allow precise targeting. The signal is received and demodulated by the smart device 140, the surrogate device, or a beacon device attached to the smart device 140 or surrogate device. The receiving device may then signal back to the pointing controller 120 over the network 130 or by retransmitting a new IR signal. Alternatively, a transmitter located within the smart device 140 (or the surrogate device or attached beacon device) transmits a signal that is received and demodulated by a direction-sensitive sensor located within the pointing controller 120. Once the target device is identified and selected, an issued command 510 can be sent to the smart device 140 over the network 130. In some embodiments, commands may alternatively be communicated directly via an IR channel.

[0105] In the case of IR-based solutions, the transmitted beam or receiver optics will typically be narrow enough to select a single object. In cases where multiple objects are illuminated, the tracking device 110 can attempt to resolve the ambiguity by examining the IR illumination intensity or duration of illumination. If resolution is not possible at the sensory level, the tracking device 110 may provide feedback to the user (by indicating through optical, auditory, or haptic feedback that the pointing was ambiguous), thereby allowing the user to unambiguously point to the intended target object.

[0106] In still further embodiments, an external camera, depth sensing, or ranging system may be located in a ceiling or wall-mounted module and may be utilized to track the pointing controller 120 or to perform pose estimation and hand tracking directly without the use of the pointing controller 120.

[0107] Additional Considerations Throughout this specification, some embodiments have used the phrase "coupled," along with its derivatives. As used herein, the term "coupled" is not necessarily limited to two or more elements being in direct physical or electrical contact. Rather, the term "coupled" can also encompass two or more elements that are not in direct contact with each other, but yet still cooperate or interact with each other.

[0108] Similarly, as used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus comprising a list of elements is not necessarily limited to only those elements and may include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0109] Additionally, the use of "the," "a," or "an" is employed to describe elements and components of embodiments herein. This is done merely for convenience and to give a general sense of the invention. The description should be read to include one or at least one, and the singular also includes the plural unless it is clear that this is not meant otherwise.

[0110] Finally, as used herein, any reference to "one embodiment" or "an embodiment" means that a particular element, feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. Appearances of the phrase "in one embodiment" in various places in the specification do not necessarily all refer to the same embodiment.

[0111] Upon perusal of this disclosure, those skilled in the art will recognize still additional alternative structures and functional designs for the described embodiments as disclosed in accordance with the principles herein. Thus, while particular embodiments and applications have been illustrated and described, it should be understood that the disclosed embodiments are not limited to the precise structure and components disclosed herein. Various modifications, changes, and variations that will be apparent to those skilled in the art may be made in the arrangement, operation, and details of the methods and apparatus disclosed herein without departing from the scope.

Claims

1. A method for controlling interaction with a smart device using a pointing controller, said method comprising: (a) a camera of a tracking device capturing image data of an object in an environment; (b) one or more processors of the tracking device detecting from the image data a location of the object within the environment; and (c) the one or more processors instructing a user, via a user interface of the tracking device, to point at the object using the pointing controller; and (d) the one or more processors acquiring inertial sensor data from a state sensing device of the pointing controller; (e) detecting, by the one or more processors, based on the inertial sensor data of the state sensing device of the pointing controller, when the pitch of the pointing controller matches a pitch vector between the tracking device and the location of the object and when the pointing controller is stationary; (f) in response to detecting from the inertial sensor data that the pitch of the pointing controller matches the pitch vector between the tracking device and the location of the object and that the pointing controller is stationary, the one or more processors perform yaw calibration of the pointing controller to initialize a reference yaw of the pointing controller relative to a yaw of the camera of the tracking device; (g) the one or more processors tracking changes in yaw of the pointing controller relative to the reference yaw based on the inertial sensor data, and tracking changes in roll and pitch relative to the detected direction of gravity; (h) the one or more processors tracking movement of a pointing vector through three-dimensional space based on the changes in yaw, roll, and pitch and the stored arm model; (i) the one or more processors detecting, based on the movement, an intersection of the pointing vector with a coordinate in the three-dimensional space associated with the smart device to place the smart device in a selected state; (j) causing the one or more processors to display, on an augmented reality display device, a virtual menu associated with the smart device; and (k) detecting a control interaction with the pointing controller associated with the virtual menu when the smart device is in the selected state; and (l) generating, by the one or more processors, commands to control operation of the smart device based on the control interactions; and wherein the method is carried out in the order of (a) to (l).

2. The method described in claim 1, wherein the coordinates associated with the smart device are coordinates representing the physical location of the smart device.

3. The method described in claim 1, wherein the coordinates associated with the smart device are coordinates representing the location of a proxy object associated with the smart device.

4. Detecting the intersection generating a pointing cone having a central axis aligned with the pointing vector, an origin proximate the location of the pointing controller, and a radius that increases with distance from the origin of the pointing vector; detecting the intersection with the pointing vector in response to the pointing cone overlapping the coordinate associated with the smart device; The method of claim 1 , comprising:

5. The method of claim 1, wherein detecting the control interaction includes detecting activation of an inter-finger button on the pointing controller.

6. Detecting the regulatory interaction comprises: Detecting an interaction with a slider control interface of the pointing controller; navigating among different menu items within the virtual menu in response to interaction with the slider control interface; selecting a menu item in response to detecting activation of an inter-finger button of the pointing controller; The method of claim 1 , comprising:

7. The method of claim 1, wherein detecting the control interaction includes detecting a gesture performed using the pointing controller that indicates a control function of the smart device.

8. The method of claim 1, wherein the state sensing device comprises a camera integrated with the pointing controller.

9. The method of claim 8, further comprising: the one or more processors recognizing the smart device as a smart light; detecting the control interaction with the pointing controller includes detecting a swiping gesture on a touch interface of the pointing controller; and generating the command includes controlling dimming of the smart lighting in response to a direction of the swiping gesture. The method of claim 1.

10. A non-transitory computer-readable storage medium storing instructions for controlling interaction with a smart device using a pointing controller, the instructions, when executed by one or more processors, causing the one or more processors to: (a) capturing image data of an object in an environment by a camera of a tracking device; (b) detecting a location of the object within the environment from the image data; (c) instructing a user via a user interface of the tracking device to point at the object using the pointing controller; (d) acquiring inertial sensor data from a state sensing device of the pointing controller; (e) detecting, based on the inertial sensor data of the state sensing device of the pointing controller, when the pitch of the pointing controller matches a pitch vector between the tracking device and the location of the object and when the pointing controller is stationary; (f) in response to detecting from the inertial sensor data that the pitch of the pointing controller matches the pitch vector between the tracking device and the location of the object and that the pointing controller is stationary, performing yaw calibration of the pointing controller to initialize a reference yaw of the pointing controller relative to a yaw of the camera of the tracking device; (g) tracking changes in yaw of the pointing controller relative to the reference yaw based on the inertial sensor data, and tracking changes in roll and pitch relative to the detected direction of gravity; (h) tracking movement of a pointing vector through three-dimensional space based on said changes in yaw, roll, and pitch and the stored arm model; (i) detecting an intersection of the pointing vector and a coordinate in the three-dimensional space associated with the smart device based on the movement to place the smart device in a selected state; (j) causing an augmented reality display device to display a virtual menu associated with said smart device; (k) detecting a control interaction with the pointing controller associated with the virtual menu when the smart device is in the selected state; (l) generating a command to control an operation of the smart device based on the control interaction; and wherein the steps are performed in the order of (a) to (l).

11. The non-transitory computer-readable storage medium of claim 10, wherein the coordinates associated with the smart device are coordinates representing the physical location of the smart device.

12. The non-transitory computer-readable storage medium of claim 10, wherein the coordinates associated with the smart device are coordinates representing the location of a proxy object associated with the smart device.

13. Detecting the intersection generating a pointing cone having a central axis aligned with the pointing vector, an origin proximate the location of the pointing controller, and a radius that increases with distance from the origin of the pointing vector; detecting the intersection with the pointing vector in response to the pointing cone overlapping the coordinate associated with the smart device; 11. The non-transitory computer-readable storage medium of claim 10, comprising:

14. The non-transitory computer-readable storage medium of claim 10, wherein detecting the control interaction includes detecting activation of an inter-finger button on the pointing controller.

15. The method of detecting the regulatory interaction, comprising: Detecting an interaction with a slider control interface of the pointing controller; navigating among different menu items within the virtual menu in response to interaction with the slider control interface; selecting a menu item in response to detecting activation of an inter-finger button of the pointing controller; 11. The non-transitory computer-readable storage medium of claim 10, comprising:

16. The non-transitory computer-readable storage medium of claim 10, wherein detecting the control interaction includes detecting a gesture performed using the pointing controller that indicates a control function of the smart device.

17. The non-transitory computer-readable storage medium of claim 10, wherein the state sensing device comprises a camera integrated with the pointing controller.

18. The method of claim 17, further comprising: recognizing the smart device as a smart light; detecting the control interaction with the pointing controller includes detecting a swiping gesture on a touch interface of the pointing controller; and generating the command includes controlling dimming of the smart lighting in response to a direction of the swiping gesture. The non-transitory computer-readable storage medium of claim 10.

19. A tracking device comprising: one or more processors; 1. A non-transitory computer-readable storage medium storing instructions for controlling interaction with a smart device using a pointing controller, the instructions, when executed by the one or more processors, causing the one or more processors to: (a) capturing image data of an object in an environment by a camera of a tracking device; (b) detecting a location of the object within the environment from the image data; (c) instructing a user via a user interface of the tracking device to point at the object using the pointing controller; (d) acquiring inertial sensor data from a state sensing device of the pointing controller; (e) detecting, based on the inertial sensor data of the state sensing device of the pointing controller, when the pitch of the pointing controller matches a pitch vector between the tracking device and the location of the object and when the pointing controller is stationary; (f) in response to detecting from the inertial sensor data that the pitch of the pointing controller matches the pitch vector between the tracking device and the location of the object and that the pointing controller is stationary, performing yaw calibration of the pointing controller to initialize a reference yaw of the pointing controller relative to a yaw of the camera of the tracking device; (g) tracking changes in yaw of the pointing controller relative to the reference yaw based on the inertial sensor data, and tracking changes in roll and pitch relative to the detected direction of gravity; (h) tracking movement of a pointing vector through three-dimensional space based on said changes in yaw, roll, and pitch and the stored arm model; (i) detecting an intersection of the pointing vector and a coordinate in the three-dimensional space associated with the smart device based on the movement to place the smart device in a selected state; (j) causing an augmented reality display device to display a virtual menu associated with said smart device; (k) detecting a control interaction with the pointing controller associated with the virtual menu when the smart device is in the selected state; (l) generating a command to control an operation of the smart device based on the control interaction; and a non-transitory computer-readable storage medium for performing steps including: A tracking device comprising:

20. A tracking device as described in claim 19, wherein the coordinates associated with the smart device are coordinates representing the physical location of the smart device.

Images