Tangible 6-DOF Interface for Augmented Reality

Abstract

Systems and methods are provided that allow physical objects in a surrounding environment to function as user interface implementations within an augmented reality environment. Physical objects detected within the field of view of a camera of a computing device can be designated as user interface implementations in response to user commands. A user interface can be attached to the designated physical object to provide a tangible user interface implementation for user interaction with the augmented reality environment.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of and claims priority to U.S. Non-provisional Patent Application No. 17 / 656,818, entitled "TANGIBLE SIX-DEGREE-OF-FREEDOM INTERFACES FOR AUGMENTED REALITY," filed March 28, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] This relates generally to user interfaces for use in augmented reality, and in particular to user interfaces based on physical objects for use in an augmented reality environment. [Background technology]

[0003] Augmented reality (AR), mixed reality (MR), or virtual reality (VR) systems may provide user interaction with content output by the system in a variety of different ways. In some circumstances, the virtual user interface and / or controls generated by these types of systems may be difficult for users to interact with and / or may lack the ability to receive and detect accurate user input. For example, a user who experiences and interacts with virtual content in an AR / MR / VR environment through a head-mounted device that includes a display, such as smart glasses, may find that the user interface capabilities of such a head-mounted device are somewhat constrained. The use of physical, tangible objects as a medium for the user interface in the AR / MR / VR environment to facilitate user interaction with the virtual content may increase the practicality and functionality of the AR / MR / VR and enhance the user experience. Summary of the Invention

[0004] In one general aspect, a computer-implemented method includes capturing an image of a surrounding environment by a camera of a computing device; identifying in the image a physical object of the surrounding environment; detecting, by the computing device, a command for associating a user interface with the physical object identified in the image; and in response to the command, triggering display of the user interface as a virtual object by a display device of the computing device; mapping at least one feature of the physical object detected in the image to at least one feature of the user interface; detecting user manipulation of the physical object; and in response to the user manipulation of the physical object, triggering a change in a control algorithm of the user interface corresponding to the user manipulation of the physical object.

[0005] The proposed solution therefore relates in particular to a computer-implemented method comprising: capturing an image of a surrounding environment by a camera of a computing device, identifying in the image a physical object of the surrounding environment, detecting by the computing device a command to associate a user interface with the physical object identified in the image, and in response to the command triggering display of the user interface as a virtual object by a display device of the computing device, mapping at least one feature of the physical object detected in the image to at least one feature of the user interface, detecting a user manipulation of the physical object, and in response to the user manipulation of the physical object triggering a change in the user interface corresponding to the user manipulation of the physical object, whereby the proposed solution enables in response to the user manipulation of the physical object to trigger a change in the user interface corresponding to the user manipulation of the physical object, the change in the user interface may relate to a change in at least one control feature of the user interface (i.e. changing how a user can interact with at least one feature of the user interface to trigger at least one movement or action) and / or a change in the function and / or position and / or orientation and / or appearance of at least one displayed feature of the user interface. In some implementations, after mapping at least one feature of the physical object detected in the image to at least one feature of the user interface, user manipulation of the physical object may be enabled to trigger and / or control operation of an electronic device in communication with the computing device, with changes in the user interface reflecting the triggered and / or altered operation of the electronic device.

[0006] In some implementations, triggering a change in a control algorithm of the user interface (or a change in the user interface) includes controlling an operation of an external electronic device in communication with the computing device in response to the user manipulation of the physical object. In some implementations, triggering a change in a control algorithm of the user interface (or a change in the user interface) in response to the user manipulation of the physical object includes altering a functionality of the user interface in response to the user manipulation of the physical object. In some implementations, triggering a change in a control algorithm of the user interface (or a change in controlling the user interface) in response to the user manipulation of the physical object includes tracking hand movements within a field of view of a camera of the computing device with a hand tracking module accessible to the computing device, detecting hand movements corresponding to user input to the user interface based on the tracking, and performing an action in response to the input to the user interface.

[0007] In some implementations, triggering a change in a control algorithm of the user interface (or a change in controlling the user interface) in response to user manipulation of the physical object includes tracking a position and orientation of the physical object by a pose estimation module accessible to the computing device, detecting at least one change in the position or orientation of the physical object based on the tracking, and adjusting at least one of the position or orientation of the user interface in response to the at least one change in the position or orientation of the physical object. In some implementations, detecting a command to associate the user interface with the physical object includes attaching a representation of the user interface to the physical object based on a mapping of at least one feature of the physical object to at least one feature of the user interface, and adjusting the position and orientation of the user interface in response to the detected change in the position and orientation of the physical object such that the representation of the user interface remains attached to the physical object. In some implementations, the computer-implemented method also includes adjusting at least one of an appearance of the user interface or a functionality of the user interface based on a distance between the computing device and the physical object.

[0008] In some implementations, detecting a command to associate the user interface with the physical object includes detecting an audible command by an audio sensor of the computing device, transcribing the audible command into an executable command by a transcription engine accessible to the computing device, and associating the user interface with the physical object in response to the executable command. In some implementations, identifying the physical object includes detecting a visual pattern in the image, comparing the visual pattern detected in the image to visual patterns associated with the image stored in a database accessible to the computing device, and identifying the physical object based on the comparison.

[0009] In some implementations, the computing device is a head-mounted display device, and capturing the image and identifying the physical object includes capturing an image of the surrounding environment with a camera attached to a frame portion of the head-mounted display device, and identifying in the image a physical object held by the user within a field of view of the camera.

[0010] In some implementations, detecting user manipulation of the physics object includes detecting movement of at least one feature of the physics object, detecting movement of at least one feature of a user interface corresponding to the movement of the at least one feature of the physical object, and triggering a change in a control algorithm of the user interface (or a change in the user interface) that includes performing an action in response to the user manipulation of the physics object.

[0011] In another general aspect, a computing system includes a computing device including a camera that captures an image of an environment external to the computing device, a display device, at least one processing device, and a memory that stores instructions that, when executed by the at least one processing device, cause the computing system to perform operations. The operations may include capturing an image of the environment by the camera, identifying a physical object in the environment in the image, detecting a command by the computing device to designate the physical object as a user interface implementation, triggering, by the display device, displaying a user interface as a virtual object attached to the physical object, mapping at least one feature of the physical object detected in the image to at least one feature of the user interface, detecting user manipulation of the physical object, and triggering a change in a control algorithm associated with the user interface (or triggering a change in the user interface) in response to the user manipulation of the physical object.

[0012] In some implementations, triggering a change in a control algorithm associated with the user interface (or a change in the user interface) includes controlling an operation of an external electronic device in communication with the computing device in response to user manipulation of a physical object. In some implementations, performing an action includes altering a functionality of the user interface in response to user manipulation of a physical object. In some implementations, triggering a change in a control algorithm associated with the user interface (or a change in controlling the user interface) includes tracking hand movements within a field of view of a camera of the computing device with a hand tracking module accessible to the computing device, detecting hand movements corresponding to user input to the user interface based on the tracking, and performing an action in response to the user input to the user interface.

[0013] In some implementations, triggering a change in a control algorithm associated with the user interface (or a change in the user interface) includes tracking a position and orientation of the physical object by a pose estimation module accessible to the computing device, detecting at least one change in the position or orientation of the physical object based on the tracking, and adjusting at least one of the position or orientation of the user interface in response to the at least one change in the position or orientation of the physical object. In some implementations, triggering the display of the user interface as a virtual object attached to the physical object includes adjusting at least one of the position or orientation of the user interface in response to the detected change in the position or orientation of the physical object such that the display of the user interface remains attached to the physical object as the physical object moves in response to user manipulation of the physical object.

[0014] In some implementations, triggering a change in a control algorithm associated with the user interface (or a change in the user interface) includes adjusting at least one of an appearance or functionality of the user interface based on a distance between the computing device and the physical object. In some implementations, detecting a command to designate the physical object as a user interface implementation includes detecting an audible command by an audio sensor of the computing device, transcribing the audible command into an executable command by a transcription engine accessible to the computing device, and designating the physical object as the user interface implementation in response to the executable command. In some implementations, detecting user manipulation of the physical object includes detecting movement of at least one feature of the physical object, detecting movement of at least one feature of the user interface corresponding to the movement of the at least one feature of the physical object, and triggering a change in a control algorithm of the user interface (or a change in the user interface) including performing an action in response to the user manipulation of the physical object.

[0015] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0016] [Figure 1A] A third person view of a user within the surrounding environment. [Figure 1B] FIG. 1 is a front view of an exemplary head mounted display device. [Figure 1C] FIG. 2 is a rear view of an exemplary head-mounted display device according to implementations described herein. [Diagram 2] FIG. 1 is a block diagram of an exemplary head-mounted display device according to implementations described herein. [Figure 3A]FIG. 2 illustrates objects in the surrounding environment within a user's field of view. [Figure 3B] FIG. 2 illustrates objects in the surrounding environment within a user's field of view. [Figure 4A] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 4B] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 4C] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 4D] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 4E] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 4F] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 4G] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 4H] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 5A] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 5B] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 5C] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 5D] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 5E] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 5F] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 6A] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 6B] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 6C] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 6D] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 6E] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 7A] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 7B] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 7C] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 7D] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 8A] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 8B] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 8C]FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 8D] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 9A] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 9B] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 9C] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 9D] FIG. 1 illustrates an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to implementations described herein. [Figure 10] 1A-C illustrate an exemplary system for using physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment, according to embodiments described herein. [Figure 11] 1 is a flow chart of a method according to embodiments described herein. [Figure 12] FIG. 1 includes an example of a computing device and a mobile computing device that can be used to implement the techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] A user may experience and interact with content output by various different types of electronic devices having content output capabilities in an AR, MR, or VR environment. In some examples, such electronic devices may include mobile computing devices such as smartphones, tablet computing devices, laptop computing devices, etc. In some examples, such electronic devices may include wearable devices having content output capabilities. Wearable devices may include, for example, head-mounted electronic devices such as smart glasses, ear-mounted electronic devices such as earbuds or headphones, wrist-mounted electronic devices such as smart watches, and other such electronic devices. In some configurations, the user interface capabilities of these types of exemplary wearable devices may be constrained to some extent based on, for example, physical size, available display area, etc. Furthermore, user interaction with content output by these types of wearable devices, for example, through gesture input, audio input, etc., may be uncomfortable for the user when the content is not visible and / or audible to others. In examples where the wearable device includes an interface device and / or control device for interacting with the content, it may still be difficult for the user to interact with the content due to the limitations of the interface / control device.

[0018] Systems and methods according to embodiments described herein may enumerate physical objects in a surrounding environment to facilitate user interaction with content output by the user's computing device. In some examples, the disclosed systems and methods may attach a user interface to a selected physical object in the surrounding environment. In some examples, the disclosed systems and methods may designate or designate a selected physical object in the surrounding environment as an auxiliary input device. In some examples, the disclosed systems and methods may provide tracking of a selected physical object in the surrounding environment to facilitate use of the selected physical object as an auxiliary input device. In some examples, user manipulation of a physical object designated as an auxiliary input device may trigger a change in one or more control algorithms associated with the user interface. In some examples, the one or more control algorithms may include, for example, control algorithms that control the operation, functionality, etc. of the computing device and enable the computing device to perform a particular function in response to input received in response to manipulation of the physical object. This may be reflected in changes to the user interface, which may include a change in at least one control function of the user interface (i.e., changing how a user can interact with at least one feature of the user interface to trigger at least one movement or action) and / or a change in the function and / or position and / or orientation and / or appearance of at least one displayed feature of the user interface. In some examples, a change in one or more control algorithms associated with the user interface may also include a change in the appearance of the user interface. In some examples, a change in one or more control algorithms associated with the user interface may also include a change in user-selectable user interface elements or user interface icons, and / or a change in the number and / or arrangement of user interface elements or icons included in the user interface.In some examples, the change in one or more control algorithms associated with the user interface may also include a change in one or more controllable devices in communication with the computing device. The opportunistic use of physical objects in the surrounding environment to facilitate user interaction in the AR or MR or VR environment may increase user engagement in the AR / MR / VR environment, and the interactive provision may increase the accuracy of user-provided input and improve the overall experience in the AR / MR / VR environment.

[0019] FIG. 1A is a third-party view of a user within an ambient environment 170. Various exemplary computing devices 100, 180, 190 that can output content for consumption by a user are shown in FIG. 1A. The exemplary computing devices include an exemplary head mounted display (HMD) device 100, e.g., in the form of smart glasses, an exemplary wrist-worn computing device 180, e.g., in the form of a smart watch, and an exemplary handheld computing device 190, e.g., in the form of a smartphone. A user may consume content output by one of the exemplary computing devices 100, 180, 190, including, e.g., still image content and / or video content, audio content, etc. A user may utilize the functionality of the computing device(s) 100, 180, 190 to interact with the content output by the computing device(s) 100, 180, 190. A user may interact with content output by the computing device(s) 100, 180, 190, for example, through manipulation of controls of the computing device(s) 100, 180, 190, gesture input detected by the computing device(s) 100, 180, 190, audio input detected by the computing device(s) 100, 180, 190, etc. In some examples, the functionality of the computing device(s) 100, 180, 190 may enable a user to interact with one or more connected or controllable devices in the surrounding environment 170 via the computing device(s) 100, 180, 190. The one or more connected or controllable devices may be external to the computing device(s) 100, 180, 190 and may communicate with the computing device(s) 100, 180, 190. Exemplary connected and / or controllable devices in the ambient environment 170 shown in FIG. 1A include, for purposes of explanation and illustration only, an exemplary smart bulb / smart light 172, an exemplary smart speaker 173, and an exemplary smart TV 174.A user may use the computing device(s) 100, 180, 190 to interact with other connected or controllable devices not explicitly shown in FIG. 1A. In some examples, the computing device(s) 100, 180, 190 may access a network 177 to communicate with external connected or controllable devices. In some examples, the HMD 100 may access information via the network 177 and / or exchange information with external resources 178, receive content via the network 177, etc.

[0020] Hereinafter, HMD 100 is used solely for purposes of explanation and illustration as an exemplary electronic computing device capable of outputting content for consumption by a user, capable of being used by a user to interact with other connected or controllable devices external to HMD 100, etc. The principles described herein may be applied to other types of electronic computing devices capable of outputting content for consumption by a user and capable of interacting with other external devices in communication with the electronic computing device.

[0021] Figure IB is a front view and Figure 1C is a rear view of an exemplary computing device 100 in the form of an HMD 100 worn by a user in Figure 1A. In the illustrations shown in Figures IB and 1C, the HMD 100 is configured as a pair of smart glasses or augmented reality glasses for purposes of explanation and illustration only.

[0022] As shown in FIG. 1B, the exemplary HMD 100 includes a frame 102. The frame 102 includes an edge portion that surrounds the glass portion(s) or lens and an arm portion coupled to the edge portion. In the example shown in FIG. 1B and FIG. 1C, a display device 104 is coupled at the arm portion of the frame 102 and configured to display content to a user within an eye box 105. The HMD 100 may also include an audio output device 106, an illumination device 108, a sensing system 110, a control system 112, at least one processor 114, an outward-facing image sensor 116, or a camera 116. In some implementations, the display device 104 may include a see-through near-eye display. For example, the display device 104 may be configured to project light from a display light source onto a portion of teleprompter glass that acts as a beam splitter mounted at an angle (e.g., at 30-45 degrees). The beam splitter allows for a reflection value and a transmission value that allows light from the display light source to be partially reflected while the remaining light is transmitted. Such an optical design may enable a user to see both physical items in the world next to the digital images (e.g., user interface elements, virtual content, etc.) generated by the display device 104. In some implementations, waveguide optics may be used to render content on the display device 104 of the HMD 100.

[0023] The audio output device 106 may include one or more speakers coupled to the frame 102. The sensing system 110 may include various sensing devices and the control system 112 may include various control system devices to facilitate operation of the HMD 100, including, for example, one or more processors 114 operably coupled to components of the control system 112. In some implementations, the control system 112 may include a communications module that provides for communication and exchange of information between the HMD 100 and other external devices.

[0024] In some implementations, camera 116 may be an outward-facing camera 116, i.e., a world-facing camera 116 capable of capturing still and / or video images of external objects in the surrounding environment within the field of view of camera 116. In some implementations, camera 116 may be a depth camera capable of collecting data related to the distance of external objects from camera 116. In some implementations, lighting device 108 may selectively operate in conjunction with camera 116, for example, for detection of objects within the field of view of camera 116.

[0025] In some implementations, the HMD 100 includes an eye tracking device 120, including, for example, one or more sensors 125, such as one or more image sensors 125. The eye tracking device 120 may detect and track the direction and movement of the gaze of the eyes. Images captured by the sensor(s) 125 may be processed to detect and track the direction and movement of the gaze and to detect gaze fixations. In some implementations, the detected gaze may be processed as user input that is translated into a corresponding interaction with an external object in the surrounding environment 170 visible to the user through the HMD 100 and / or an interaction with an object displayed to the user by the display device 104.

[0026] FIG. 2 is a block diagram of an exemplary system according to embodiments described throughout this disclosure. As shown in FIG. 2, in some embodiments, the system includes one or more computing devices including at least one computing device 200 (e.g., the HMD 100 described above, and / or other electronic computing devices, etc.). The computing device 200 can selectively communicate over a network 206 to access external resources 202, such as, for example, a server computer system, a processor, a database, memory storage, etc. The computing device 200 can operate under the control of a control system 270. The computing device 200 can communicate with one or more external computing devices 204 (another wearable computing device, another mobile computing device, an external controllable device, etc.) either directly (via wired and / or wireless communication) or over the network 206. The computing device 200 can include a communication module 280 to facilitate external communication. In some implementations, computing device 200 includes a sensing system 220 that includes various sensing system components including, for example, one or more eye tracking sensors 222 including, for example, an image sensor, one or more position / orientation sensor(s) 224 including, for example, an inertial measurement unit, accelerometer, gyroscope, magnetometer, etc., and one or more audio sensors 226 capable of detecting audio input. Computing device 200 may include more or fewer sensing devices and / or combinations of sensing devices.

[0027] In some implementations, computing device 200 may include an output system 240 including, for example, one or more display devices capable of displaying still and / or video content and one or more audio output devices capable of outputting audio content. In some implementations, computing device 200 may include one or more camera(s) 260. Camera(s) 260 may be, for example, outward-facing or world-facing cameras capable of capturing still and / or video images of an environment external to computing device 200. Still and / or video images may be displayed by a display device of output system 240 and / or transmitted externally via communication modules 280 and 206 and / or stored in memory 230 of computing device 200. Computing device 200 may include one or more processors 250, which may be formed in a substrate configured to execute one or more machine-executable instructions or portions of software, firmware, or a combination thereof. Processor(s) 250 may be semiconductor-based, including semiconductor materials capable of executing digital logic. Memory 230 may include any type of storage device that stores information in a format that can be read and / or executed by processor(s) 250. Memory 230 may store applications and modules that perform certain operations when executed by processor(s) 250. In some examples, applications and modules may be stored in an external storage device and loaded into memory 230.

[0028] In some implementations, computing device 200 may access additional resources 210 to facilitate detection and tracking of user input. In some implementations, additional resources 210 may be accessed by computing device 200 via network 206 and / or within external resources 202. In some implementations, additional resources may be available within computing device 200. Additional resources 210 may include, for example, one or more databases and / or one or more processing algorithms to facilitate detection and tracking of user input. In some implementations, additional resources include perception modules providing, for example, two-dimensional visual tracking (e.g., of objects detected within the field of view of camera 260), three-dimensional pose estimation, six degrees of freedom pose estimation (e.g., of objects detected within the field of view of camera 260), hand tracking, etc. In some implementations, additional resources 210 include representation databases including, for example, a database of visual patterns associated with objects and movements of objects within the field of view of camera 260, a database of virtual widgets for representation of various functions, a database of relationships between various tracked objects, etc. In some implementations, the additional resources include a transcription engine that provides transcriptions of the detected audio commands for processing by the control system 270 and / or the processor(s) 250.

[0029] FIG. 3A is a third-person view of a user in an ambient environment 170. The user can view the ambient environment 170 and objects within the ambient environment 170 through the lens of the exemplary HMD 100, as shown in FIG. 3B. In the view shown in FIG. 3A, the user moves toward a workspace 300 in the ambient environment 170, such that exemplary objects within the region of the workspace 300 are visible to the user and captured within the field of view of the camera 116 of the HMD 100. In the example shown in FIGS. 3A and 3B, the workspace 300 includes several tangible objects within the ambient environment 170 that provide an opportunistic user interface for interaction within the AR / MR / VR environment. In the example shown in FIGS. 3A and 3B, the exemplary tangible objects include a bottle 320, a disk-shaped object 330 (such as a coin), a book 340, a card 350, and a pin 360, all of which are positioned on a work surface 310. The objects shown in Figure 3A are only some examples of tangible objects that may be enumerated for use in providing a user interface to facilitate user interaction within an AR / MR / VR environment. The principles described herein may be applied to other types of tangible objects that may provide an opportunistic user interface in an AR / MR / VR environment.

[0030] 4A-4H illustrate example systems and methods that may enumerate real-world tangible objects as features of a user interface. The examples shown in FIGS. 4A-4H are described with respect to use of an HMD 100 for output of AR / MR / VR content and user interaction with the content for purposes of explanation and illustration only. The principles described herein may be applied to use of other types of computing devices that provide user interaction with content output by the computing device.

[0031] In some examples, a user may wish to use a surface for viewing and / or interacting with content and / or information that may be output by the HMD 100, for example. In the example shown in FIG. 4A, the user picks up a card 350 from the work surface 310 such that the card 350 is in the user's field of view / visible through the lens(es) of the HMD 100 and in the field of view of the camera of the HMD 100 with a first surface 351 of the card 350 facing the user. The system may detect the card 350 in the field of view of the camera of the HMD 100 and may identify or recognize the card 350 based, for example, on images and / or information previously stored in a database associated with the configuration of the card, information contained on the card, etc. In some examples, this may be performed by a recognition engine operating in conjunction with information stored in a database(s) of additional resources 210 available to the HMD 100. In some examples, the system may detect the surface 351 of the card and identify the surface 351 as a receiving surface for output of content, user interface, etc.

[0032] A user may issue a command to cause content to be output on the surface of the card 350 to facilitate user interaction with the content. In some examples, the command may be a verbal or audible command detected by an audio sensor of the HMD 100 and processed by a transcription engine. In some examples, the command may be a gesture detected by a camera and recognized by a perception module, a manipulation of a control device of the HMD 100, etc. In some examples, in response to detecting the card 350 within the field of view of the camera of the HMD 100 and detecting the command, the HMD 100 may perform a corresponding action. In this example, the action performed may be the designation of the card 350 as a user interface implementation. In the example shown in FIG. 4B, in response to a user command to display today's weather on the card 350, a user interface 410 may be output or displayed on the first surface 351 of the card 350. The user interface 410 may be, for example, a weather widget output on the card 350. In some examples, the system may access or obtain the weather widget from additional resources 210 available to the HMD 100. In some examples, the user interface 410 may be displayed within a first surface 351 of a card 350, as shown in FIG. 4B.

[0033] In some examples, the user interface 410 may remain attached to the first surface 351 of the card 350 as the card 350 moves, as shown in FIG. 4C. In some examples, at least one feature of the card 350 may be mapped to at least one corresponding feature of the user interface 410 to provide coordinated movement of the user interface 410 in response to the movement of the card 350. That is, as the card 350 moves and the orientation of the card 350 changes, the distance of the card 350 from the HMD 100 changes, etc., the user interface 410 may remain attached to the card 350 due to the mapping of the respective features. Changes in the position and / or orientation of the card 350 may be detected and tracked by the perception module, and corresponding changes to the position and / or orientation of the user interface 410 may be implemented accordingly. Changes in the position and / or orientation of the card 350 may facilitate further user interaction with the content output on the card 350.

[0034] In some examples, the user interface 410 may include a user interface element 415 that allows user input and / or user interaction with content displayed on the card 350 via the user interface 410. Upon interaction with the user interface element 415, an action may be executed by the HMD 100. In some examples, user manipulation of the card 350 in the form of user interaction with the user interface element 415 may cause a change in a control algorithm associated with the user interface 410. In the example shown in FIG. 4D, the user interface element 415 is an icon that is selectable by the user. In this example, a user selection causes the weather widget to advance one day to display the weather for the next day. That is, a user input at the user interface element 415, e.g., a tap at the user interface element 415, may cause a change in a control algorithm associated with the user interface 410, causing the user interface 410 to display the weather for the next day (rather than the current day or a previously selected day). Subsequent input at the example user interface element 415 may cause the weather widget to continue to advance to display the weather for subsequent days. In the example shown in Figure 4D, the hand tracking module may detect the presence of a user's finger within the field of view of the camera of the HMD 100 and may detect a tapping motion on a user interface element 415. The user interface element 415 shown in Figure 4D is just one example of a user interface element that a user may manipulate to provide input to a user interface displayed as described with respect to Figures 4A-4D. Various other types of user interface elements may also be displayed to receive other types of input and / or to cause other types of information to be displayed and / or actions to be taken.

[0035] In FIG. 4E, the user has flipped the card 350 such that the second surface 352 of the card 350 is visible within the field of view of the camera of the HMD 100. In this example, user manipulation of the card 350 in the form of flipping the card 350 may trigger a change in a control algorithm associated with the user interface 410 associated with the card 350 as a user interface implementation. In this example, in response to detecting the flip of the card 350, an action may be performed that changes the functionality of the card 350 as a user interface implementation. For example, the user may issue a command to cause the second surface 352 of the card 350 to output content. In some examples, the command may be a verbal or audible command detected by an audio sensor of the HMD 100 and processed by a transcription engine. In some examples, the command may be detected based on a function triggered in response to an element recognized on the second surface 352 of the card 350 (i.e., text, an immediate response code, etc.), a gesture detected by the camera and recognized by a perception module, manipulation of a control device of the HMD 100, etc. In this example, flipping the card 350 may signal a change in a control algorithm associated with the desired functionality of the use of the card 350 as a user interface implementation.

[0036] In the example shown in FIG. 4E, a user command to flip the card and provide the balance on the card 350 triggers the output of a user interface 420 on the second surface 352 of the card 350, as shown in FIG. 4F. The example user interface 420 shown in FIG. 4E provides the requested balance on the card 350. The example user interface also provides a first example user interface element 421 that allows the user to pay the outstanding amount, and a second example user interface element 422 that allows the user to transfer funds to and / or from the card 350. Similar to the example described with respect to FIG. 4D, input (e.g., touch input or tap input) on one of the example user interface elements 421, 422 may depend on the hand tracking module detecting the presence of a user's finger within the field of view of the camera of the HMD 100 and a tapping movement on one of the example user interface elements 421, 422.

[0037] In the example shown in FIG. 4G, the first user interface 431 is output or displayed on the first surface 351 of the card 350. The system may display the first user interface 431 in response to a user command (i.e., a verbal or audible command, or other type of command) to display navigation to a particular location on the card 350. In the arrangement shown in FIG. 4G, the card 350 is held by the user in a first position, e.g., at a first distance from the HMD 100. In FIG. 4H, the position of the card 350 has changed in response to user manipulation of the card 350. That is, in FIG. 4H, the card 350 is held by the user in a second position, e.g., at a second distance from the HMD 100 that is less than the first distance, and thus the card 350 is closer to the HMD 100. Thus, the user moves the card 350 closer to the HMD 100, mimicking a zoom-in to provide a better view of the information provided. At this closer position, user manipulation of the card 350 triggers a change in a control algorithm associated with the user interface such that a second user interface 432 is displayed. The second user interface 432 may display a more detailed view of the navigation provided in the first user interface 431. Thus, in some examples, the appearance of the user interface may change or transform as the position and / or orientation of the card changes. In some examples, the change in appearance may include a change in the information provided to and / or accessible to the user based on the distance between the card 350 and the HMD 100. The change in appearance and the corresponding change in the information provided and / or accessible to the user based on the distance between the card 350 and the HMD 100 may enable the system to provide additional information to the user without requiring specific user input beyond the natural movement of the card 350 (i.e., spoken / audible commands, tap inputs, manipulation of the HMD 100, etc.).

[0038] By displaying the user interface of the card 350 using the first and second surfaces 351, 352, the user may be provided with a way to interact with the content output by the HMD 100 using tangible objects as anchors for the content, for input provided via the user interface displayed on the card 350. The output of content using the surfaces 351, 352 of the card 350 to anchor the display of the user interface is provided merely as an example. Other relatively flat surfaces available to the user in a similarly opportunistic manner may also be used to provide similar functionality.

[0039] 5A-5F show examples of a user interface attached to a disc-shaped object 330, i.e., a coin 330. As shown in FIG. 5A, a user picks up the coin 330 from the work surface 310 such that the coin 330 is in the user's field of view / visible through the lens(es) of the HMD 100 and in the field of view of the camera of the HMD 100. The system may detect the coin 330 in the field of view of the camera of the HMD 100 and may identify or recognize the coin 330, for example, based on detected features of an image of the coin 330 captured by the camera of the HMD 100, as well as images and / or information previously stored in a database. In some examples, this may be performed by a recognition engine operating in conjunction with information stored in the database(s) of the additional resources 210 available to the HMD 100. In some examples, the system may detect and identify features of the coin 330 for use in tracking the movement of the coin 330 and the corresponding movement of a user interface attached to the coin 330.

[0040] A user may issue a command to cause content to be output on the face of the coin 330. In some examples, the command may be a verbal or audible command that is detected by an audio sensor of the HMD 100 and processed by a transcription engine. In some examples, upon detection of the coin 330 within the field of view of the camera of the HMD 100 and detection of the command, the system may perform an action that designates the coin 330 as a user interface implementation. In the example shown in FIG. 5B, in response to a user command to display a volume controller on the coin 330, a user interface 510 is output or displayed on the coin 330 designated as a user interface implementation, as shown in FIG. 5B. The user interface 510 may be manipulated by a user to control, for example, an audio output level of a smart speaker 173 in the ambient environment 170 that is connected to and controllable via the HMD 100. The user interface 510 includes a selection indicator 515 that corresponds to a selected volume level. In some examples, the selection indicator 515 may be mapped to or attached to a particular feature on the coin 330. By mapping points on the selection indicator 515 to corresponding features on the coin 330, a coordinated movement of the selection indicator 515 (virtually overlaid on the coin 330) may be provided as the coin 330 is moved or manipulated by the user. In this manner, user manipulation of the coin 330 (moving or rotating the coin 330) to adjust the volume level causes the selection indicator 515 to move with the coin 330, providing a visual indication of the desired change in the volume level. The rotation of the coin 330, which serves as a volume controller for the smart speaker 173 in this example, in turn triggers a change in the volume level of the smart speaker 173 connected to and / or controllable by the HMD 100. In some examples, the user interface 510 may include a mode indicator 520 to indicate that the system is in a volume control mode, and / or a volume level indicator 530 to indicate the current volume level.5D, 5E, and 5F show the rotation of the coin 330 and the corresponding rotation of the selection indicator 515 of the user interface 510 through increasing volume levels. As the coin 330 rotates, as shown in FIGS. 5D-5F, the selection indicator 515 of the user interface 510, which may be attached to the coin 330 and / or mapped to a selected function of the coin 330, rotates with the coin 330. This allows the coin 330 to function as a volume adjustment knob to control the system volume. With the ready and opportunistic availability of the coin 330 and the ability to have the coin 330 function as a volume adjustment knob in this manner, the user is provided with a tangible user interface for control of the volume of the AR / MR / VR environment.

[0041] 6A-6E show examples where a user interface is attached to an exemplary pin 360. As shown in FIG. 6A, a user picks up the pin 360 from the work surface 310 such that the pin 360 is in the user's field of view / visible through the lens(es) of the HMD 100 and in the field of view of the camera of the HMD 100. The system may detect the pin within the field of view of the camera of the HMD 100 and may identify or recognize the pin 360 based, for example, on the external contour of the pin 360, detected features of an image of the pin 360 captured by the camera of the HMD 100, as well as images and / or information previously stored in a database. In some examples, this may be performed by a recognition engine operating in conjunction with information stored in the database(s) of the additional resources 210 available to the HMD 100. In some examples, the system may detect and identify features of the pin 360 for use in tracking the movement of the pin 360 and the corresponding control of one of the connected / controllable devices in the surrounding environment 170.

[0042] A user may issue a command that causes the pin 360 to function as a user interface implementation. In some examples, the command may be a verbal or audible command that is detected by an audio sensor of the HMD 100 and processed by the transcription engine. In the example shown in FIG. 6B, in response to a user command to use the pin 360 as a light intensity controller, a user interface 610 may be output on or displayed with the pin 360. The pin 360 may then be manipulated by a user to control the level of light intensity output by, for example, a smart light 172 connected to and controllable by the HMD 100. That is, when movement of the pin 360 is detected, the level of light intensity output by the smart light 172 may be adjusted accordingly. The user interface 610 may provide a visual indication of the amount of adjustment of the level of light intensity as the pin 360 is moved. For example, as shown in FIGS. 6B-6E, as the pin 360 is moved in the direction of arrow A, the level of light intensity output by the smart light 172 increases and the visual indication provided by the user interface 610 becomes greater corresponding to the increased level of light intensity. In this manner, as the pin 360 is moved, the light intensity level is adjusted based on the direction and magnitude of the movement of the pin 360. This allows the pin 360 to function as a light adjusting device to adjust the intensity of the light output by the smart light 172. The ready and opportunistic availability of the pin 360 and the ability to cause the pin 360 to function in this manner provides a user with a tangible user interface for control of light intensity in an AR / MR / VR environment.

[0043] 7A-7D show another example of a user interface attached to an exemplary pin 360 for selecting and / or adjusting the color of light output by a smart light 172 connected to and controllable by the HMD 100. As shown in FIG. 7A, a user picks up the pin 360 from the work surface 310 such that the pin is in the user's field of view / visible through the lens(es) of the HMD 100 and in the field of view of the camera of the HMD 100. The system may detect the pin 360 within the field of view of the camera of the HMD 100 and may identify or recognize the pin 360 based, for example, on the external contour of the pin 360, detected features of an image of the pin 360 captured by the camera of the HMD 100, as well as images and / or information previously stored in a database, etc. In some examples, this may be performed by a recognition engine operating in conjunction with information stored in the database(s) of the additional resources 210 available to the HMD 100. In some examples, the system may detect and identify characteristics of the pin 360 for use in tracking the movement of the pin 360 and the corresponding control of one of the connected / controllable devices in the surrounding environment 170.

[0044] A user may issue a command that causes the pin 360 to function as a user interface implementation. In some examples, the command may be a verbal or audible command that is detected by an audio sensor of the HMD 100 and processed by the transcription engine. In response to a user command to use the pin 360 as a light color selector, the user interface 710 may be output on or displayed with the pin 360, as shown in FIG. 7A. The pin 360 may then be manipulated by the user, for example, to select a color of light to be output by a smart light 172 connected to and controllable by the HMD 100, change the color of light output by the smart light 172, etc. That is, when movement of the pin 360 is detected, the color of light output by the smart light 172 may be adjusted accordingly based on a color guide provided by the user interface 710. For example, as shown in FIGS. 7B-7D, the color of light output by the smart light 172 changes as the pin 360 changes relative to a color guide provided in the user interface 710. This allows pins 360 to function as a color selection device for controlling the light output by smart light 172. The ready and opportunistic availability of pins 360 and the ability to function in this manner provides a user with a tangible user interface for control of the light output of an AR / MR / VR environment.

[0045] 8A-8D show another example of a user interface attached to an exemplary pin 360. As shown in FIG. 8A, a user picks up pin 360 from work surface 310 such that pin 360 is in the user's field of view / visible through the lens(es) of HMD 100 and in the field of view of the camera of HMD 100. The system may detect the pin in the field of view of the camera of HMD 100 and may identify or recognize pin 360 as described above. In some examples, the system may detect and identify features of pin 360 for use in tracking movement of pin 360 and translating the detected movement into actions in the AR / MR / VR environment.

[0046] A user may issue a command to cause the pin 360 to function as a user interface implementation. In some examples, the command may be a verbal or audible command that is detected by an audio sensor of the HMD 100 and processed by the transcription engine. In the example shown in FIG. 8B, the user issues a command to use the pin 360 as a marking or drawing implementation. In response to this command, the system generates an annotation corresponding to the detected movement of the pin 360. In the example shown in FIGS. 8B-8D, when the movement of the pin 360 is detected, a corresponding annotation 810 is created in the AR / MR / VR environment. Tracking the pin 360 allows certain features of the pin 360 to track the movement of the pin 360 and display the corresponding annotation 810. This allows the pin 360 to function as a marking or drawing implementation. The ready and opportunistic availability of the pin 360 and the ability to cause the pin 360 to function in this manner provide the user with a tangible user interface for annotation in the AR / MR / VR environment.

[0047] The use of pins 360 as selection and control implementations in an AR / MR / VR environment is shown by way of example in Figures 6A-8D. Other similar tangible objects available to the user in a similar opportunistic manner may also be used to provide similar functionality.

[0048] 9A-9D show examples in which an exemplary bottle 320 is selected for use as an interface implementation in an AR / MR / VR environment. As shown in FIG. 9A, a user picks up a bottle 320 from a work surface 310 such that the bottle 320 is in the user's field of view / visible through the lens(es) of the HMD 100 and in the field of view of the camera of the HMD 100. The system may detect the bottle 320 in the field of view of the camera of the HMD 100 and may identify or recognize the bottle 320 based, for example, on the external contour of the bottle 320, detected features of an image of the bottle 320 captured by the camera of the HMD 100, as well as images and / or information previously stored in a database. In some examples, this may be performed by a recognition engine operating in conjunction with information stored in the database(s) of the additional resources 210 available to the HMD 100. In some examples, the system may detect and identify features of the pin 360 for use in tracking the movement of the pin 360 and the corresponding control of one of the connected / controllable devices in the surrounding environment 170.

[0049] A user may issue commands that cause the bottle 320 to function as a user interface implementation. In some examples, the commands may be verbal or audible commands that are detected by an audio sensor in the HMD 100 and processed by a transcription engine. In the example shown in FIG. 9B-FIG. 9D, movement of the bottle 320 may be tracked in response to a user command to use the bottle 320 as an interface implementation or as a control implementation, such as a joystick in a game. The detected movement of the bottle 320 may be translated into actions in the AR / MR / VR environment, such as command inputs and / or control inputs in a game, such that the bottle 320 functions as a control device for the game. In this manner, as the position and / or orientation of the bottle 320 is changed and the position of the bottle's six degrees of freedom is detected and tracked, the bottle 320 may function as an input implementation / control implementation, and the detected movement may be processed as input in the AR / MR / VR environment. The ready and opportunistic availability of the bottle 320 and the ability to function in this manner provides the user with a tangible control implementation in the AR / MR / VR environment.

[0050] 10A-10C illustrate an example in which a user's hand serves as an interface implementation in an AR / MR / VR environment. As shown in FIG. 10A, the user places his / her hand within the field of view of the camera of the HMD 100. The system may detect the presence of the hand within the field of view of the camera of the HMD 100 and may identify or recognize the hand, for example, based on images and / or information previously stored in a database. In some examples, this may be performed by a recognition engine operating in conjunction with information stored in a database(s) of additional resources 210 available to the HMD 100. In some examples, the system may detect and identify hand features for use in tracking hand movements to facilitate user interface placement, hand tracking, etc.

[0051] A user may issue commands that cause the hand to act as a user interface implementation. In some examples, the commands may be verbal or audible commands that are detected by an audio sensor in the HMD 100 and processed by the transcription engine. In the example shown in FIG. 10A, a first user interface 1010 is output or displayed on the palm of the user's hand in response to a first command to display the text of the transcription on the user interface device (i.e., the user's hand). In the example shown in FIG. 10B, a rotation, translation, or change in posture triggers the generation of a second user interface 1020 that is output or displayed on the user's hand. In this example, a change in posture of the user's hand triggers a transformation of the transcription shown in the first user interface 1010. In FIG. 10C, a tap or touch input triggers the output or display of a third user interface 1030 on the user's hand, allowing for remapping of the functionality of the user's hand for user interfaces and user interactions. The opportunistic availability of a user's hand for use as an interface surface, and the ability to act in this manner, provides the user with a tangible interface and control implementation of the AR / MR / VR environment.

[0052] FIG. 11 is a flow chart of a method 1100 according to an embodiment described herein. The method 1100 provides for the use of tangible physical objects in a surrounding environment as six degrees of freedom interfaces and control devices in an AR / MR / VR environment. An object implemented as an interface and / or control device may be captured within the field of view of a camera of a computing device (block 1110). A visual pattern identifying a feature, etc. may be identified within an image of the object captured by the camera of the computing device (block 1120). The detected visual pattern and / or identifying feature may be compared or matched with visual patterns and / or identifying features stored in a database accessible to the computing device. By comparing the detected visual pattern and identifying features detected in the image and previously stored information in the database, usage data, etc., the object may be identified and a three-dimensional pose of the object may be estimated (block 1130). Relationship information may be retrieved, for example, from a database accessible to the computing device (block 1140). The relationship information may be used to determine relevant applications related to the detected object, relevant user interfaces associated with the detected object, etc. In response to detecting a command related to an interaction with the user interface (block 1150), an updated user interface, updated display information, updated functionality, etc. may be generated and output (block 1160). The user command may be, for example, a spoken command detected and transcribed by a transcription engine to process a gesture input detected based on six degrees of freedom tracking of an object, a tap input detected with respect to a displayed user interface, six degrees of freedom tracking of a user's hand, etc. The process may continue until the object is no longer detected within the field of view of the camera of the computing device (block 1170).

[0053] 12 illustrates an example of a computing device 1200 and a mobile computing device 1250 that may use the techniques described herein. The computing device 1200 includes a processor 1202, a memory 1204, a storage device 1206, a high-speed interface 1208 that connects to the memory 1204 and a high-speed expansion port 1210, and a low-speed interface 1212 that connects to a low-speed bus 1214 and the storage device 1206. Each of the components 1202, 1204, 1206, 1208, 1210, and 1212 are interconnected using various buses and may be mounted on a common motherboard or in other manners as needed. The processor 1202 can process instructions for execution within the computing device 1200, including instructions stored in the memory 1204 or the storage device 1206, to display graphical information of a GUI on an external input / output device, such as a display 1216, coupled to the high-speed interface 1208. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory, as appropriate. Also, multiple computing devices 1200 may be connected together, with each device providing a portion of the required operations (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system).

[0054] The memory 1204 stores information within the computing device 1200. In one implementation, the memory 1204 is a volatile memory unit(s). In another implementation, the memory 1204 is also a non-volatile memory unit(s). The memory 1204 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0055] The storage device 1206 can provide mass storage for the computing device 1200. In one embodiment, the storage device 1206 can be or include a computer-readable medium such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including a storage area network or other configuration of devices. The computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as, for example, the memory 1204, the storage device 1206, or a memory on the processor 1202.

[0056] The high-speed controller 1208 manages bandwidth-intensive operations of the computing device 1200, and the low-speed controller 1212 manages low-bandwidth intensive operations. Such an allocation of functions is merely an example. In one embodiment, the high-speed controller 1208 is coupled to the memory 1204, the display 1216 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 1210 that may accept various expansion cards 1210 (not shown). In an embodiment, the low-speed controller 1212 is coupled to the storage device 1206 and the low-speed expansion port 1214. The low-speed expansion ports, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled, for example, via a network adapter, to one or more input / output devices such as a keyboard, pointing device, scanner, etc., or may be coupled to a network device such as a switch or router.

[0057] The computing device 1200 may be implemented in a number of different forms, as shown in the figure. For example, the computing device 1200 may be implemented as a standard server 1220, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 1224. Additionally, the computing device 1200 may be implemented in a personal computer, such as a laptop computer 1222. Alternatively, the components of the computing device 1200 may be combined with other components in a mobile device (not shown), such as device 1250. Each such device may include one or more of the computing devices 1200, 1250, and the entire system may consist of multiple computing devices 1200, 1250 in communication with each other.

[0058] Computing device 1250 includes, among other components, a processor 1252, memory 1264, input / output devices such as a display 1254, a communications interface 1266, and a transceiver 1268. Device 1250 may also include a storage device, such as a microdrive or other device, to provide additional storage. Each of components 1250, 1252, 1264, 1254, 1266, and 1268 are interconnected using various buses, and some of the components may be mounted on a common motherboard or in other manners as desired.

[0059] The processor 1252 can execute instructions in the computing device 1250, including instructions stored in the memory 1264. The processor can be implemented as a chipset of chips including separate analog and digital processors. The processor can provide coordination of other components of the device 1250, such as control of a user interface, applications run by the device 1250, and wireless communications by the device 1250.

[0060] The processor 1252 may communicate with a user via a control interface 1258 and a display interface 1256 coupled to a display 1254. The display 1254 may be, for example, a TFT LCD (thin film transistor liquid crystal display), LED (light emitting diode), or OLED (organic light emitting diode) display, or other suitable display technology. The display interface 1256 may include suitable circuitry for driving the display 1254 to present graphical and other information to the user. The control interface 1258 may receive commands from a user and translate the commands for submission to the processor 1252. Additionally, an external interface 1262 may be provided in communication with the processor 1252 to enable short-range communication between the device 1250 and other devices. The external interface 1262 may provide, for example, wired communication in some implementations or wireless communication in other implementations, and multiple interfaces may be used.

[0061] The memory 1264 stores information within the computing device 1250. The memory 1264 can be implemented as one or more of a computer-readable medium, a volatile memory unit(s), and / or a non-volatile memory unit(s). An expansion memory 1274 can also be provided and connected to the device 1250 via an expansion interface 1272, which can include, for example, a SIMM (single in-line memory module) card interface. Such expansion memory 1274 can provide additional storage space for the device 1250 or can store applications or other information for the device 1250. In particular, the expansion memory 1274 can include instructions for performing or supplementing the aforementioned processes and can include secure information. Thus, for example, the expansion memory 1274 can be provided as a security module for the device 1250 and can be programmed with instructions that enable secure use of the device 1250. Additionally, secure applications can be provided via a SIMM card along with additional information, such as placing identifying information on the SIMM card in an unhackable manner.

[0062] The memory may include, for example, flash memory and / or NVRAM memory (non-volatile random access memory), as described below. In one embodiment, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 1264, expansion memory 1274, or memory on processor 1252, which may be received, for example, via transceiver 1268 or external interface 1262.

[0063] Device 1250 may communicate wirelessly via communication interface 1266, which may include digital signal processing circuitry if necessary. Communication interface 1266 may provide communication in various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, among others. Such communication may occur, for example, via radio frequency transceiver 1268. Additionally, short-range communication may occur, such as using Bluetooth, WiFi, or other such transceivers (not shown). Additionally, GPS (Global Positioning System) receiver module 1270 may provide additional wireless data related to navigation and position to device 1250, which may be used as needed by applications executing on device 1250.

[0064] Device 1250 may also communicate audibly using audio codec 1260, which may receive speech information from a user and convert it into usable digital information. Audio codec 1260 may also generate sounds audible to the user, such as through a speaker in a handset of device 1250. Such sounds may include sounds from voice telephone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on device 1250.

[0065] The computing device 1250 may be implemented in many different forms, as shown in the figure. For example, the computing device 1250 may be implemented as a mobile phone 1280. The computing device 1250 may also be implemented as part of a smartphone 1282, a personal digital assistant, or other similar mobile device.

[0066] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations of one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from the storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0067] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0068] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer that has a display device (such as an LED (light-emitting diode), or OLED (organic LED), or LCD (liquid crystal display) monitor / screen) for displaying information to a user, a keyboard and a pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0069] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or that includes middleware components (e.g., an application server), or that includes a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an embodiment of the systems and techniques described herein), or a combination of such back-ware, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0070] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0071] In some implementations, the computing device shown in the figure may include sensors that interface with the AR headset / HMD device 1290 to generate an augmented environment for viewing the inserted content in the physical space. For example, one or more sensors included on the computing device 1250 or other computing devices shown in the figure may provide input to the AR headset 1290 or may provide input to the AR space in general. The sensors may include, but are not limited to, a touch screen, an accelerometer, a gyroscope, a pressure sensor, a biometric sensor, a temperature sensor, a humidity sensor, and an ambient light sensor. The computing device 1250 may use the sensors to determine an absolute position and / or a detected rotation of the computing device in the AR space that can then be used as input to the AR space. For example, the computing device 1250 may be embedded in the AR space as a virtual object such as a controller, a laser pointer, a keyboard, a weapon, etc. The user's positioning of the computing device / virtual object when embedded in the AR space allows the user to position the computing device to view the virtual object in a particular way in the AR space. For example, if the virtual object represents a laser pointer, the user may manipulate the computing device as if it were an actual laser pointer. A user can move a computing device left and right, up and down, in a circle, etc., to use the device in a manner similar to using a laser pointer. In some implementations, a user can aim a virtual laser pointer at a target location.

[0072] In some implementations, one or more input devices included on or connected to the computing device 1250 can be used as input to the AR space. The input devices can include, but are not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, an earphone or earphones with input capabilities, a game controller, or other connectable input devices. A user interacting with an input device included on the computing device 1250 when the computing device is integrated into the AR space can cause certain actions to occur within the AR space.

[0073] In some implementations, the touch screen of the computing device 1250 can be rendered as a touch pad in the AR space. A user can interact with the touch screen of the computing device 1250. The interaction is rendered, for example, in the AR headset 1290, as movements on the rendered touch pad in the AR space. The rendered movements can control virtual objects in the AR space.

[0074] In some implementations, one or more output devices included in the computing device 1250 can provide output and / or feedback to a user of the AR headset 1290 in the AR space. The output and feedback can be visual, tactical, or audio. The output and / or feedback can include, but is not limited to, vibration, turning one or more lights or strobes on and off or flashing and / or blinking, sounding an alarm, chiming, playing a song, and playing an audio file. The output devices can include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), strobes, and speakers.

[0075] In some implementations, the computing device 1250 may appear as another object in the computer-generated 3D environment. A user's interaction with the computing device 1250 (e.g., rotating, shaking, touching, swiping a finger across the touch screen) can be interpreted as an interaction with an object in the AR space. In the example of a laser pointer in the AR space, the computing device 1250 appears as a virtual laser pointer in the computer-generated 3D environment. As the user manipulates the computing device 1250, the user in the AR space sees the movement of the laser pointer. The user receives feedback from the interaction with the computing device 1250 in the AR environment on the computing device 1250 or on the AR headset 1290. The user's interaction with the computing device can be translated into an interaction with a user interface generated in the AR environment of the controllable device.

[0076] In some implementations, the computing device 1250 may include a touch screen. For example, a user may interact with the touch screen to interact with the user interface of the controllable device. For example, the touch screen may include user interface elements, such as sliders, that may control characteristics of the controllable device.

[0077] Computing device 1200 is intended to represent various forms of digital computers and devices, including, but not limited to, laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 1250 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smart phones, and other similar computing devices. The components illustrated herein, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the implementation of the invention described and / or claimed herein.

[0078] Having described several embodiments, it will of course be understood that various modifications can be made without departing from the spirit and scope of the invention.

[0079] Moreover, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desired results. Moreover, other steps may be provided in or removed from the described flows, and other components may be added to or removed from the described systems. Accordingly, other embodiments are within the scope of the following claims.

[0080] In addition to the above, the system, program, or functionality described herein may provide controls to the user that allow the user to choose both whether and when the system, program, or functionality described herein may enable collection of user information (e.g., information about the user's social network, social actions, or activities, occupation, user preferences, or the user's current location) and whether to send content or communications to the user from the server. Additionally, certain data may be processed in one or more ways such that personally identifiable information is removed before it is stored or used. For example, the user's identity may be processed such that personally identifiable information of the user cannot be identified, or if location information is obtained (such as to the city, zip code, or state level), the user's geographic location may be generalized such that the user's specific location cannot be identified. Thus, the user may control what information is collected about the user, how that information is used, and what information is provided to the user.

[0081] As described herein, while certain features of the described embodiments have been illustrated, numerous modifications, substitutions, changes, and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the embodiments. It is to be understood that these are presented by way of example only, and not by way of limitation, and that various changes in form and details may be made. Any part of the apparatus and / or method described herein may be combined in any combination, except in mutually exclusive combinations. The embodiments described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different embodiments described.

Claims

1. 1. A computer-implemented method comprising: Capturing an image of a surrounding environment by a camera of a computing device; identifying, within the image, physical objects within the surrounding environment; Detecting, by the computing device, a command to associate a user interface with the physical object identified in the image; in response to the command, triggering a display of the user interface as a virtual object by a display device of the computing device; mapping at least one feature of the physical object detected in the image to at least one feature of the user interface; Detecting a user manipulation of the physical object; in response to the user manipulation of the physics object, triggering a change in the user interface corresponding to the user manipulation of the physics object; The method includes:

2. 2. The computer-implemented method of claim 1, wherein triggering the change in the user interface includes controlling operation of an external electronic device in communication with the computing device in response to the user manipulation of the physical object.

3. 3. The computer-implemented method of claim 1 or 2, wherein triggering the change in the user interface in response to the user manipulation of the physical object comprises changing a functionality of the user interface in response to the user manipulation of the physical object.

4. Triggering the change in the user interface in response to the user manipulation of the physical object comprises: tracking hand movements within a field of view of the camera of the computing device with a hand tracking module accessible to the computing device; detecting hand movements corresponding to user input to the user interface based on the tracking; performing an action in response to the input to the user interface; 4. A computer implemented method according to claim 1, comprising:

5. Triggering the change in the user interface in response to the user manipulation of the physical object comprises: tracking a position and orientation of the physical object by a pose estimation module accessible to the computing device; detecting a change in at least one of the position or the orientation of the physical object based on the tracking; adjusting at least one of a position or an orientation of the user interface in response to the change in the at least one of the position or the orientation of the physical object; 2. A computer implemented method according to any one of the preceding claims, comprising:

6. Detecting the command to associate the user interface with the physical object includes: attaching a representation of the user interface to the physical object based on the mapping of the at least one feature of the physical object to the at least one feature of the user interface; adjusting the position and orientation of the user interface in response to detected changes in the position and orientation of the physical object such that the representation of the user interface remains attached to the physical object; 6. The computer implemented method of claim 5, comprising:

7. 7. The computer-implemented method of claim 5 or 6, further comprising adjusting at least one of an appearance of the user interface or a functionality of the user interface based on a distance between the computing device and the physical object.

8. Detecting the command to associate the user interface with the physical object includes: Detecting an audible command with an audio sensor of the computing device; transcribing the audible commands into executable commands by a transcription engine accessible to the computing device; associating the user interface with the physical object in response to the executable command; 2. A computer implemented method according to any one of the preceding claims, comprising:

9. Identifying the physical object comprises: Detecting a visual pattern within the image; comparing the visual pattern detected in the image with visual patterns associated with images stored in a database accessible to the computing device; identifying the physical object based on the comparison; and 2. A computer implemented method according to any one of the preceding claims, comprising:

10. and wherein the computing device is a head mounted display device, capturing the image and identifying the physical object comprises: capturing the image of the surrounding environment with the camera attached to a frame portion of the head mounted display device; identifying, within the image, the physical object held by a user within a field of view of the camera; 2. A computer implemented method according to any one of the preceding claims, comprising:

11. Detecting the user manipulation of the physical object includes: Detecting a movement of the at least one feature of the physical object; detecting a movement of the at least one feature of the user interface corresponding to the movement of the at least one feature of the physical object; triggering the change in the control algorithm of the user interface, the change including performing an action in response to the user manipulation of the physical object; 2. A computer implemented method according to any one of the preceding claims, comprising:

12. 1. A computing system comprising: a computing device, the computing device comprising: a camera for capturing images of an external environment of the computing device; A display device; at least one processing device; and a memory storing instructions that, when executed by the at least one processing device, cause the computing system to: capturing an image of the surrounding environment with the camera; identifying, within the image, physical objects within the surrounding environment; Detecting, by the computing device, a command to designate the physical object as a user interface implementation; in response to the command, triggering, by the display device, a display of a user interface as a virtual object attached to the physical object; mapping at least one feature of the physical object detected in the image to at least one feature of the user interface; Detecting a user manipulation of the physical object; triggering a change in the user interface in response to the user manipulation of the physical object; A computing system that performs operations including:

13. 13. The computing system of claim 12, wherein triggering the change in the user interface comprises controlling operation of an external electronic device in communication with the computing device in response to the user manipulation of the physical object.

14. 14. The computing system of claim 12 or 13, wherein triggering the change in the user interface comprises modifying a functionality of the user interface in response to user manipulation of the physical object.

15. Triggering the change in the user interface includes: tracking hand movements within a field of view of the camera of the computing device with a hand tracking module accessible to the computing device; detecting hand movements corresponding to user input to the user interface based on the tracking; performing an action in response to the user input into the user interface; A computing system according to any one of claims 12 to 14, comprising:

16. Triggering the change in the user interface includes: tracking a position and orientation of the physical object by a pose estimation module accessible to the computing device; detecting a change in at least one of the position or the orientation of the physical object based on the tracking; adjusting at least one of a position or an orientation of the user interface in response to the change in the at least one of the position or the orientation of the physical object; A computing system according to any one of claims 12 to 15, comprising:

17. Triggering the display of the user interface as the virtual object attached to the physical object includes: adjusting at least one of a position or orientation of the user interface in response to a detected change in a position or orientation of the physical object such that the representation of the user interface remains attached to the physical object as the physical object moves in response to user manipulation of the physical object; A computing system according to any one of claims 12 to 16, comprising:

18. Triggering the change in the user interface includes: The computing system of any one of claims 12 to 17, further comprising adjusting at least one of an appearance or functionality of the user interface based on a distance between the computing device and the physical object.

19. Detecting the command to designate the physical object as the user interface implementation includes: Detecting an audible command with an audio sensor of the computing device; transcribing the audible commands into executable commands by a transcription engine accessible to the computing device; In response to the executable command, designating the physical object as the user interface implementation; A computing system according to any one of claims 12 to 18, comprising:

20. Detecting the user manipulation of the physical object includes: Detecting a movement of the at least one feature of the physical object; detecting a movement of the at least one feature of the user interface corresponding to the movement of the at least one feature of the physical object; triggering the change in the control algorithm of the user interface, the change including performing an action in response to the user manipulation of the physical object; A computing system according to any one of claims 12 to 19, comprising:

Images