Tangible six-degree-of-freedom interface for augmented reality

By identifying and responding to physical objects in the surrounding environment, the interaction difficulties of virtual user interfaces in the AR/MR/VR system are solved, and a more accurate and rich user experience is achieved.

CN118946869APending Publication Date: 2024-11-12GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280094251.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-28
Filing Date
2022-11-30
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Virtual user interfaces in existing augmented reality, mixed reality and virtual reality systems are difficult to interact with users effectively, and lack precise user input detection capabilities, limiting user experience and interaction functions.

Method used

Through a computer implementation method, the camera captures the surrounding environment images, recognizes physical objects, detects commands associated with these objects, and responds to user manipulation, triggers changes in the user interface, including adjustments to control algorithms and display features, so that the physical objects become users The carrier of the interface.

Benefits of technology

It enhances the interaction ability between users and virtual content, improves interaction accuracy and experience, and expands users' participation in AR/MR/VR environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118946869A_ABST
    Figure CN118946869A_ABST
Patent Text Reader

Abstract

Systems and methods are provided in which a physical object in an ambient environment may act as a user interface tool in an augmented reality environment. A physical object detected within a field of view of a camera of a computing device may be designated as a user interface tool in response to a user command. A user interface may be attached to the specified physical object to provide a tangible user interface tool for user interaction with the augmented reality environment.
Need to check novelty before this filing date? Find Prior Art

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation of and claims priority to U.S. non-provisional patent application No. 17 / 656,818, filed on March 28, 2022, entitled “TANGIBLE SIX-DEGREE-OF-FREEDOM INTERFACES FOR AUGMENTED REALITY,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] This generally relates to user interfaces for use in augmented reality, and in particular to user interfaces based on physical objects for use in augmented reality environments. Background Art

[0003] Augmented reality (AR) systems or mixed reality (MR) systems or virtual reality (VR) systems can provide user interaction with content output by the system in a variety of different ways. In some cases, the virtual user interface and / or controls generated by these types of systems may be difficult for users to interact with, and / or may lack the ability to receive and detect precise user input. For example, a user who experiences and interacts with virtual content in an AR / MR / VR environment through a head-mounted device including a display such as smart glasses may find that the user interface functionality of such a head-mounted device is somewhat constrained. Using physical tangible objects as carriers of user interfaces in AR / MR / VR environments to facilitate user interaction with virtual content can enhance the practicality and functionality of AR / MR / VR devices and can enhance the user experience. Summary of the invention

[0004] In one general aspect, a computer-implemented method includes: capturing, by a camera of a computing device, an image of a surrounding environment; identifying in the image a physical object in the surrounding environment; detecting, by the computing device, a command to associate a user interface with the physical object identified in the image; triggering, in response to the command, display of the user interface as a virtual object by a display device of the computing device; mapping at least one feature of the physical object detected in the image to at least one feature of the user interface; detecting a user manipulation of the physical object; and, in response to the user manipulation of the physical object, triggering a change in a control algorithm of the user interface corresponding to the user manipulation of the physical object.

[0005] Thus, the proposed solution specifically relates to a computer-implemented method comprising: capturing an image of a surrounding environment by a camera of a computing device; identifying a physical object in the surrounding environment in the image; detecting, by the computing device, a command to associate a user interface with the physical object identified in the image; triggering, in response to the command, display of the user interface as a virtual object by a display device of the computing device; mapping at least one feature of the physical object detected in the image to at least one feature of the user interface; detecting a user manipulation of the physical object; and in response to the user manipulation of the physical object, triggering a change of the user interface corresponding to the user manipulation of the physical object. Thus, the proposed solution allows triggering a change of the user interface corresponding to the user manipulation of the physical object in response to the user manipulation of the physical object, wherein the change of the user interface may involve a change of at least one control function of the user interface (i.e., a change in how a user can interact with at least one feature of the user interface for triggering at least one operation or action) and / or a change of functionality and / or position and / or orientation and / or appearance of at least one displayed feature of the user interface. In some implementations, after at least one feature of a physical object detected in an image has been mapped to at least one feature of a user interface, user manipulation of the physical object can allow for triggering and / or controlling operations of an electronic device that communicates with a computing device, where changes in the user interface reflect the triggered and / or changed operations of the electronic device.

[0006] In some implementations, triggering the change in the control algorithm of the user interface (or the change in the user interface) includes: controlling the operation of an external electronic device in communication with the computing device in response to the user manipulation of the physical object. In some implementations, triggering the change in the control algorithm of the user interface (or the change in the user interface) in response to the user manipulation of the physical object includes: changing the functionality of the user interface in response to the user manipulation of the physical object. In some implementations, triggering the change in the control algorithm of the user interface (or controlling the change in the user interface) in response to the user manipulation of the physical object includes: tracking hand movements in the field of view of the camera of the computing device by a hand tracking module accessible to the computing device; detecting hand movements corresponding to user input into the user interface based on the tracking; and performing an action in response to the input into the user interface.

[0007] In some implementations, triggering the change of the control algorithm of the user interface (or controlling the change of the user interface) in response to the user manipulation of the physical object includes: tracking the position and orientation of the physical object by a posture estimation module accessible to the computing device; detecting a change in at least one of the position or the orientation of the physical object based on the tracking; and adjusting at least one of the position or the orientation of the user interface in response to the change in the position or the orientation of the physical object. In some implementations, detecting the command to associate the user interface with the physical object includes: attaching the display of the user interface to the physical object based on the mapping of the at least one feature of the physical object to the at least one feature of the user interface; and adjusting the position and the orientation of the user interface in response to the detected change in the position and the orientation of the physical object so that the display of the user interface remains attached to the physical object. In some implementations, the computer-implemented method also includes: adjusting at least one of the appearance of the user interface or the functionality of the user interface based on the distance between the computing device and the physical object.

[0008] In some implementations, detecting the command to associate the user interface with the physical object includes: detecting an audible command by an audio sensor of the computing device; transcribing the audible command into an executable command by a transcription engine accessible by the computing device; and associating the user interface with the physical object in response to the executable command. In some implementations, identifying the physical object includes: detecting a visual pattern in the image; comparing the visual pattern detected in the image to visual patterns associated with images stored in a database accessible by the computing device; and identifying the physical object based on the comparison.

[0009] In some implementations, the computing device is a head-mounted display device, and wherein capturing the image and identifying the physical object includes: capturing the image of the surrounding environment by the camera mounted on a frame portion of the head-mounted display device; and identifying in the image the physical object held by a user within a field of view of the camera.

[0010] In some implementations, detecting the user manipulation of the physical object includes: detecting movement of at least one feature of the physical object; detecting movement of at least one feature of the user interface corresponding to the movement of the at least one feature of the physical object; and triggering the change of the control algorithm of the user interface (or the change of the user interface), including performing an action in response to the user manipulation of the physical object.

[0011] In another general aspect, a computing system includes: a computing device including: a camera that captures an image of an environment external to the computing device; a display device; at least one processing device; and a memory storing instructions that, when executed by the at least one processing device, cause the computing system to perform operations. The operations may include: capturing an image of the environment by the camera; identifying a physical object in the environment in the image; detecting a command to designate the physical object as a user interface tool by the computing device; triggering display of a user interface by the display device as a virtual object attached to the physical object; mapping at least one feature of the physical object detected in the image to at least one feature of the user interface; detecting a user manipulation of the physical object; and triggering a change in a control algorithm associated with the user interface (or triggering a change in the user interface) in response to the user manipulation of the physical object.

[0012] In some implementations, triggering the change in the control algorithm associated with the user interface (or the change in the user interface) includes: controlling the operation of an external electronic device in communication with the computing device in response to the user manipulation of the physical object. In some implementations, performing the action includes: changing the functionality of the user interface in response to the user manipulation of the physical object. In some implementations, triggering the change in the control algorithm associated with the user interface (or controlling the change in the user interface) includes: tracking hand movements in the field of view of the camera of the computing device by a hand tracking module accessible to the computing device; detecting hand movements corresponding to user input into the user interface based on the tracking; and performing an action in response to the user input into the user interface.

[0013] In some implementations, triggering the change of the control algorithm associated with the user interface (or the change of the user interface) includes: tracking the position and orientation of the physical object by a posture estimation module accessible to the computing device; detecting a change in at least one of the position or the orientation of the physical object based on the tracking; and adjusting at least one of the position or orientation of the user interface in response to the change in the at least one of the position or the orientation of the physical object. In some implementations, triggering the display of the user interface as the virtual object attached to the physical object includes: adjusting at least one of the position or orientation of the user interface in response to the detected change in the position or orientation of the physical object so that the display of the user interface remains attached to the physical object when the physical object moves in response to user manipulation of the physical object.

[0014] In some implementations, triggering the change in the control algorithm associated with the user interface (or the change in the user interface) includes: adjusting at least one of the appearance or functionality of the user interface based on the distance between the computing device and the physical object. In some implementations, detecting the command to designate the physical object as the user interface tool includes: detecting an audible command by an audio sensor of the computing device; transcribing the audible command into an executable command by a transcription engine accessible by the computing device; and designating the physical object as the user interface tool in response to the executable command. In some implementations, detecting the user manipulation of the physical object includes: detecting movement of the at least one feature of the physical object; detecting movement of the at least one feature of the user interface corresponding to the movement of the at least one feature of the physical object; and triggering the change in the control algorithm of the user interface (or the change in the user interface) includes performing an action in response to the user manipulation of the physical object.

[0015] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A It is a third-person view of the user's surroundings.

[0017] Figure 1B is a front view of an example head mounted display device according to an implementation as described herein, and Figure 1C is a rear view of an example head mounted display device according to an implementation as described herein.

[0018] Figure 2 is a block diagram of an example head mounted display device according to an implementation as described herein.

[0019] Figure 3A and Figure 3B Objects in the surrounding environment that are within the user's field of view are shown.

[0020] 4A to 4H An example system for utilizing physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment in accordance with implementations described herein is shown.

[0021] FIG. 5A to FIG. 5F An example system for utilizing physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment in accordance with implementations described herein is shown.

[0022] FIG. 6A to FIG. 6EAn example system for utilizing physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment in accordance with implementations described herein is shown.

[0023] 7A to 7D An example system for utilizing physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment in accordance with implementations described herein is shown.

[0024] FIG. 8A to FIG. 8D An example system for utilizing physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment in accordance with implementations described herein is shown.

[0025] 9A to 9D An example system for utilizing physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment in accordance with implementations described herein is shown.

[0026] FIG. 10A to FIG. 10C An example system for utilizing physical objects in a surrounding environment as a mechanism for a user interface in an AR / MR / VR environment in accordance with implementations described herein is shown.

[0027] Fig.11 is a flow chart of a method according to an implementation described herein.

[0028] Fig.12 Examples of computing devices and mobile computing devices are shown that can be used to implement the techniques described herein. DETAILED DESCRIPTION

[0029] Users can experience content output by various different types of electronic devices with content output capabilities and interact with the content in an AR environment, an MR environment, or a VR environment. In some examples, these electronic devices may include mobile computing devices such as smart phones, tablet computing devices, laptop computing devices, etc. In some examples, these electronic devices may include wearable devices with content output capabilities. Wearable devices may include, for example, head-mounted electronic devices such as smart glasses, ear-mounted electronic devices such as earplugs or headphones, wrist-mounted electronic devices such as smart watches, and other such electronic devices. In some configurations, the user interface capabilities of these types of example wearable devices may be subject to certain constraints based on, for example, physical size, available display area, etc. Further, when the content output by these types of wearable devices through, for example, gesture input, audio input, etc. is not visible and / or audible to others, the user interaction with the content may be uncomfortable for the user. In examples where the wearable device includes an interface device and / or control device for interacting with the content, due to the limitations of the interface / control device, this may still make it difficult for the user to interact with the content.

[0030] According to the implementation described herein, the system and method can requisition physical objects in the surrounding environment to facilitate user interaction with the content output by the user's computing device. In some examples, the disclosed system and method can attach a user interface to a physical object selected in the surrounding environment. In some examples, the disclosed system and method can assign or specify a physical object selected in the surrounding environment as an auxiliary input device. In some examples, the disclosed system and method can provide tracking of physical objects selected in the surrounding environment to facilitate the use of the selected physical object as an auxiliary input device. In some examples, user manipulation of a physical object that has been designated as an auxiliary input device can trigger changes in one or more control algorithms associated with the user interface. In some examples, one or more control algorithms may include, for example, a control algorithm that controls the operation, functionality, etc. of a computing device to allow the computing device to perform certain functions in response to an input, which is received in response to manipulation of a physical object. This can be reflected in a change in the user interface, which may include a change in at least one control function of the user interface (i.e., a change in the way a user can interact with at least one feature of the user interface to trigger at least one operation or action) and / or a change in the functionality and / or position and / or orientation and / or appearance of at least one displayed feature of the user interface. In some examples, changes to one or more control algorithms associated with the user interface may also include changes in the appearance of the user interface. In some examples, changes to one or more control algorithms associated with the user interface may also include changes in user interface elements or icons that are selectable by the user of the user interface and / or changes in the number and / or arrangement of user interface elements or icons included in the user interface. In some examples, changes to one or more control algorithms associated with the user interface may also include changes in one or more controllable devices that communicate with the computing device. Opportunistic use of physical objects in the surrounding environment to facilitate user interaction in an AR or MR or VR environment can expand user engagement in an AR / MR / VR environment by providing interaction, can improve the accuracy of input provided by the user, and can enhance the overall experience in an AR / MR / VR environment.

[0031] Figure 1A is a third-person view of the user in the surrounding environment 170 . Figure 1A100, 180, 190 that can output content for consumption by a user. The example computing devices include an example head mounted display (HMD) device 100 in the form of, for example, smart glasses, an example wrist-worn computing device 180 in the form of, for example, a smart watch, and an example handheld computing device 190 in the form of, for example, a smart phone. A user can consume content output by one of the example computing devices 100, 180, 190, including, for example, still and / or moving image content, audio content, and the like. A user can utilize the functionality of the computing device 100, 180, 190 to interact with the content output by the computing device 100, 180, 190. A user can interact with the content output by the computing device 100, 180, 190 through, for example, manipulation of controls of the computing device 100, 180, 190, gesture input detected by the computing device 100, 180, 190, audio input detected by the computing device 100, 180, 190, and the like. In some examples, the functionality of computing devices 100, 180, 190 may allow a user to interact with one or more connected or controllable devices in surrounding environment 170 via computing device 100, 180, 190. One or more connected or controllable devices may be located external to computing device 100, 180, 190 and in communication with computing device 100, 180, 190. For purposes of discussion and illustration only, Figure 1A Example connected and / or controllable devices in the surrounding environment 170 shown in FIG. 1 include example smart light bulbs / smart lamps 172, example smart speakers 173, and example smart TVs 174. A user may use computing devices 100, 180, 190 to communicate with Figure 1A In some examples, computing devices 100, 180, 190 can access network 177 to communicate with externally connected or controllable devices. In some examples, HMD 100 can access external resources 178 via network 177 and / or exchange information with the external resources, receive content via network 177, etc.

[0032] Hereinafter, for purposes of discussion and illustration only, HMD 100 will be used as an example electronic computing device that can output content to be consumed by a user, can be used by a user to interact with other connected or controllable devices external to HMD 100, etc. The principles to be described herein can be applied to other types of electronic computing devices that can output content for consumption by a user and can interact with other external devices that communicate with the electronic computing device.

[0033] Figure 1B It is presented by Figure 1Aa front view of an example computing device 100 in the form of an HMD 100 worn by a user, and Figure 1C is a rear view of the example computing device. Figure 1B and Figure 1C In the example shown in FIG. 1 , for purposes of discussion and illustration only, the HMD 100 is configured as a pair of smart glasses or augmented reality glasses.

[0034] like Figure 1B As shown in FIG, an example HMD 100 includes a frame 102. The frame 102 includes an edge portion surrounding a glass portion or lens and an arm portion coupled to the edge portion. Figure 1B and Figure 1C In the example shown in , the display device 104 is coupled in an arm portion of the frame 102, which is configured to display content to the user within the eyebox 105. The HMD 100 may also include an audio output device 106, an illumination device 108, a sensing system 110, a control system 112, at least one processor 114, and an externally facing image sensor 116 or camera 116. In some implementations, the display device 104 may include a see-through near-eye display. For example, the display device 104 may be configured to project light from a display source onto a portion of a teleprompter glass that acts as a beam splitter that sits at an angle (e.g., 30 to 45 degrees). The beam splitter may implement a reflection value and a transmission value that allow light from the display source to be partially reflected and the remaining light to be transmitted through. This optical design may allow a user to see physical objects in the world in two ways alongside a digital image (e.g., a user interface element, virtual content, etc.) generated by the display device 104. In some implementations, waveguide optics may be used to depict content on the display device 104 of the HMD 100 .

[0035] The audio output device 106 may include one or more speakers coupled to the frame 102. The sensing system 110 may include various sensing devices, and the control system 112 may include various control system devices for facilitating the operation of the HMD 100, including, for example, one or more processors 114 operably coupled to components of the control system 112. In some implementations, the control system 112 may include a communication module that provides for communication and information exchange between the HMD 100 and other external devices.

[0036] In some implementations, the camera 116 can be an external-facing camera 116, or a world-facing camera 116, capable of capturing still and / or moving images of external objects in the surrounding environment within the field of view of the camera 116. In some implementations, the camera 116 can be a depth camera that can collect data related to the distance of external objects from the camera 116. In some implementations, the lighting device 108 can selectively operate, for example, with the camera 116 to detect objects in the field of view of the camera 116.

[0037] In some implementations, the HMD 100 includes a gaze tracking device 120, which includes, for example, one or more sensors 125, such as one or more image sensors 125. The gaze tracking device 120 can detect and track eye gaze direction and movement. The images captured by the sensor 125 can be processed to detect and track gaze direction and movement and detect gaze fixation. In some implementations, the detected gaze can be processed as user input to be converted into corresponding interactions with external objects in the surrounding environment 170 visible to the user through the HMD 100 and / or with objects displayed to the user by the display device 104.

[0038] Figure 2 is a block diagram of an example system according to implementations described throughout this disclosure. Figure 2 As shown in , in some implementations, the system includes one or more computing devices, including at least one computing device 200 (such as, for example, the HMD 100 described above, and / or other electronic computing devices). The computing device 200 can selectively communicate via a network 206 to access external resources 202, such as, for example, a server computer system, a processor, a database, a memory storage device, etc. The computing device 200 can operate under the control of a control system 270. The computing device 200 can communicate with one or more external computing devices 204 (another wearable computing device, another mobile computing device, an externally controllable device, etc.) directly (via wired and / or wireless communication) or via a network 206. The computing device 200 may include a communication module 280 to facilitate external communications. In some implementations, computing device 200 includes a sensing system 220 that includes various sensing system components, including, for example: one or more gaze tracking sensors 222, including, for example, an image sensor; one or more position / orientation sensors 224, including, for example, an inertial measurement unit, an accelerometer, a gyroscope, a magnetometer, etc.; and one or more audio sensors 226 that can detect audio input. Computing device 200 can include more or fewer sensing devices and / or combinations of sensing devices.

[0039] In some implementations, the computing device 200 may include an output system 240, which includes, for example, one or more display devices that can display still and / or moving image content and one or more audio output devices that can output audio content. In some implementations, the computing device 200 may include one or more cameras 260. The camera 260 may be, for example, an externally facing or world-facing camera that can capture still and / or moving images of the environment outside the computing device 200. The still and / or moving images may be displayed by the display device of the output system 240 and / or sent externally via the communication modules 280 and 206, and / or stored in the memory 230 of the computing device 200. The computing device 200 may include one or more processors 250 that may be formed in a substrate, and the one or more processors 250 are configured to execute one or more machine executable instructions or multiple pieces of software, firmware, or a combination thereof. The processor 250 may be semiconductor-based, including semiconductor materials that can execute digital logic. The memory 230 may include any type of storage device that stores information in a format that can be read and / or executed by the processor 250. Memory 230 may store applications and modules that perform certain operations when executed by processor 250. In some examples, the applications and modules may be stored in an external storage device and loaded into memory 230.

[0040] In some implementations, the computing device 200 may access additional resources 210 to facilitate detection and tracking of user input. In some implementations, the computing device 200 may be able to access additional resources 210 via the network 206 and / or within the external resource 202. In some implementations, the additional resources may be available within the computing device 200. The additional resources 210 may include, for example, one or more databases and / or and one or more processing algorithms to facilitate detection and tracking of user input. In some implementations, the additional resources include, for example, a perception module that provides two-dimensional visual tracking (e.g., of objects detected within the field of view of the camera 260), three-dimensional pose estimation, six-degree-of-freedom pose estimation (e.g., of objects detected within the field of view of the camera 260), hand tracking, etc. In some implementations, the additional resources 210 include a representation database, which includes, for example, a database of visual patterns associated with objects within the field of view of the camera 260 and the movement of the objects, a database of virtual widgets for representations of various functionalities, a database of relationships between various tracked objects, etc. In some implementations, the additional resources include a transcription engine that provides transcription of detected audio commands for processing by the control system 270 and / or the processor 250 .

[0041] Figure 3A170 is a third-person view of the user in the surrounding environment 170. The user can view the surrounding environment 170 and objects in the surrounding environment 170 through the lenses of the example HMD 100, such as Figure 3B As shown in Figure 3A In the view shown in , the user has moved toward the workspace 300 in the surrounding environment 170, so that example objects within the area of ​​the workspace 300 are visible to the user and captured within the field of view of the camera 116 of the HMD 100. Figure 3A and Figure 3B In the example shown in , the workspace 300 includes a plurality of tangible objects in the surrounding environment 170 that provide an opportunistic user interface for interaction in the AR / MR / VR environment. Figure 3A and Figure 3B In the example shown in , example tangible objects include a bottle 320 , a disc-shaped object 330 (such as a coin), a book 340 , a card 350 , and a pin 360 positioned on a work surface 310 . Figure 3A The objects shown in are only some examples of tangible objects that can be requisitioned for use in providing a user interface to facilitate user interaction in an AR / MR / VR environment. The principles described herein can be applied to other types of tangible objects that can provide an opportunistic user interface in an AR / MR / VR environment.

[0042] 4A to 4H Example systems and methods are shown in which real-world tangible objects can be enlisted as mechanisms for user interfaces. For purposes of discussion and illustration only, 4A to 4H The examples shown in the description are about using HMD 100 for output of AR / MR / VR content and user interaction with the content. The principles to be described herein can be applied to the use of other types of computing devices that provide user interaction with content output by the computing device.

[0043] In some examples, a user may wish to use, for example, a surface to view and / or interact with content and / or information that may be output by HMD 100. Figure 4AIn the example shown in , the user has picked up a card 350 from the work surface 310 so that the card 350 is in the user's field of view / visible through the lenses of the HMD 100 and within the field of view of the camera of the HMD 100, with a first surface 351 of the card 350 facing the user. The system can detect the card 350 within the field of view of the camera of the HMD 100, and can identify or recognize the card 350 based on, for example, previously stored images and / or information in a database associated with the configuration of the card, information contained on the card, etc. In some examples, this can be performed by a recognition engine working in conjunction with information stored in a database of additional resources 210 available to the HMD 100. In some examples, the system can detect the surface 351 of the card and recognize the surface 351 as a receiving surface for output of content, a user interface, etc.

[0044] The user may issue a command that causes content to be output on the surface of the card 350 to facilitate user interaction with the content. In some examples, the command may be a verbal or audible command detected by an audio sensor of the HMD 100 and processed by a transcription engine. In some examples, the command may be a gesture detected by a camera and recognized by a perception module, manipulation of a control device of the HMD 100, and the like. In some examples, in response to detection of the card 350 within the field of view of the camera of the HMD 100 and detection of the command, the HMD 100 may perform a corresponding action. In this example, the action to be performed may be to designate the card 350 as a user interface tool. Figure 4B In the example shown in , in response to a user command to display today's weather on card 350, user interface 410 may be output or displayed on first surface 351 of card 350. User interface 410 may be, for example, a weather widget output on card 350. In some examples, the system may access or retrieve the weather widget from additional resources 210 available to HMD 100. In some examples, user interface 410 may be displayed within the scope of first surface 351 of card 350, such as Figure 4B as shown in .

[0045] In some examples, user interface 410 may remain attached to first surface 351 of card 350 while card 350 is moved, such as Figure 4C. In some examples, at least one feature of card 350 may be mapped to at least one corresponding feature of user interface 410 to provide coordinated movement of user interface 410 in response to movement of card 350. That is, as card 350 moves and the orientation of card 350 changes and the distance of card 350 from HMD 100 changes, etc., user interface 410 may remain attached to card 350 due to the mapping of corresponding features. Changes in the position and / or orientation of card 350 may be detected and tracked by the perception module, and corresponding changes in the position and / or orientation of user interface 410 may be implemented accordingly. Changes in the position and / or orientation of card 350 may facilitate further user interaction with content output on card 350.

[0046] In some examples, user interface 410 may include user interface elements 415 that allow user input into and / or user interaction with content displayed on card 350 via user interface 410. Interaction with user interface elements 415 may result in actions being performed by HMD 100. In some examples, user manipulation of card 350 in the form of user interaction with user interface elements 415 may result in changes to a control algorithm associated with user interface 410. Figure 4D In the example shown in , user interface element 415 is an icon that can be selected by the user. In this example, selection by the user causes the weather widget to advance one day and display the weather for the following day. That is, user input at user interface element 415 (e.g., a tap at user interface element 415) can cause a change in the control algorithm associated with user interface 410, causing user interface 410 to display the weather for the following day (rather than the current day or previously selected day). Subsequent input at the example user interface element 415 can cause the weather widget to continue to advance to display the weather for subsequent days. Figure 4D In the example shown in , the hand tracking module can detect the presence of a user's finger within the field of view of the camera of HMD 100 , and can detect a tap action at user interface element 415 . Figure 4D The user interface element 415 shown in FIG. 4 is only one example of a user interface element that can be manipulated by a user to provide information such as about 4A to 4D The user interface displayed provides input. Various other types of user interface elements may also be displayed to receive other types of input and / or cause other types of information to be displayed and / or take other types of actions.

[0047] exist Figure 4E, the user has flipped the card 350 so that the second surface 352 of the card 350 is visible within the field of view of the camera of the HMD 100. In this example, the user manipulation of the card 350 in the form of flipping the card 350 can trigger a change in the control algorithm associated with the user interface 410, which is associated with the card 350 as a user interface tool. In this example, in response to the detected flipping of the card 350, an action that changes the functionality of the card 350 as a user interface tool can be performed. For example, the user can issue a command that causes content to be output on the second surface 352 of the card 350. In some examples, the command can be a verbal or audible command that is detected by the audio sensor of the HMD 100 and processed by the transcription engine. In some examples, the command can be detected based on a function triggered in response to an element recognized on the second surface 352 of the card 350 (i.e., text, a quick response code, etc.), a gesture detected by the camera and recognized by the perception module, a manipulation of the control device of the HMD 100, etc. In this example, the flipping of the card 350 may signal a change in a control algorithm associated with the desired functionality of using the card 350 as a user interface tool.

[0048] exist Figure 4E In the example shown in , the flipping of the card and the user command to provide the balance on the card 350 triggers the output of the user interface 420 on the second surface 352 of the card 350, such as Figure 4F as shown in . Figure 4E The example user interface 420 shown in FIG. 4 provides a requested balance on the card 350. The example user interface also provides: a first example user interface element 421 that allows the user to pay the outstanding balance; and a second example user interface element 422 that allows the user to transfer funds to and / or from the card 350. Figure 4D As with the described examples, input (e.g., touch or tap input) at one of the example user interface elements 421, 422 can rely on detection by the hand tracking module of the presence of a user's finger within the field of view of the camera of the HMD 100 and a tap action at one of the example user interface elements 421, 422.

[0049] exist Figure 4G In the example shown in , the first user interface 431 has been output or displayed on the first surface 351 of the card 350. The system may display the first user interface 431 in response to a user command (i.e., a spoken or audible command or other type of command) to display navigation to a particular location on the card 350. Figure 4G In the arrangement shown in , the card 350 is held by the user at a first position, for example at a first distance from the HMD 100. Figure 4H, the position of card 350 has changed in response to user manipulation of card 350. That is, in Figure 4H , card 350 is held by the user at a second position, e.g., at a second distance from HMD 100 that is less than the first distance, such that card 350 is closer to HMD 100. Thus, the user has moved card 350 closer to HMD 100, emulating zooming in to provide a closer look at the information provided. At this closer position, user manipulation of card 350 has triggered a change in a control algorithm associated with the user interface, such that second user interface 432 is displayed. Second user interface 432 can provide a more detailed view of the navigation provided in first user interface 431. Thus, in some examples, the appearance of the user interface can change or deform as the position and / or orientation of the card changes. In some examples, the change in appearance can include a change in the information provided to the user and / or accessible to the user. The change in appearance based on the distance between the card 350 and the HMD 100, and the corresponding change in information provided to and / or accessible to the user, can allow the system to provide additional information to the user without requiring specific user input beyond the natural movement of the card 350 (i.e., spoken / audible commands, tap inputs, manipulation of the HMD 100, etc.).

[0050] Using the first surface 351 and the second surface 352 of the card 350 for displaying a user interface can provide a way for the user to interact with the content output by the HMD 100 and the input provided via the user interface displayed on the card 350 using a tangible object as an anchor point. The use of the surfaces 351, 352 of the card 350 to output content to anchor the display of the user interface is provided by way of example only. Other relatively flat surfaces that are available to the user in a similarly opportunistic manner can also be employed to provide similar functionality.

[0051] FIG. 5A to FIG. 5F An example is shown in which the user interface is attached to a disc-shaped object 330 or a coin 330. Figure 5A, a user has picked up a coin 330 from a work surface 310 such that the coin 330 is in the user's field of view / visible through the lenses of the HMD 100 and within the field of view of the camera of the HMD 100. The system may detect the coin 330 within the field of view of the camera of the HMD 100 and may identify or recognize the coin 330 based on, for example, detected features in an image of the coin 330 captured by the camera of the HMD 100 and previously stored images and / or information in a database. In some examples, this may be performed by a recognition engine working in conjunction with information stored in a database of additional resources 210 available to the HMD 100. In some examples, the system may detect and recognize features of the coin 330 for use in tracking the movement of the coin 330 and corresponding movement of a user interface attached to the coin 330.

[0052] The user may issue a command that causes content to be output on the surface of the coin 330. In some examples, the command may be a verbal or audible command that is detected by an audio sensor of the HMD 100 and processed by the transcription engine. In some examples, detection of the coin 330 in the field of view of the camera of the HMD 100 and detection of the command may cause the system to perform an action that designates the coin 330 as a user interface tool. Figure 5B In the example shown in , in response to a user command to display a volume controller on the coin 330, the user interface 510 is output or displayed on the coin 330 designated as the user interface tool, such as Figure 5B . The user interface 510 can be manipulated by the user to, for example, control the audio output level of a smart speaker 173 connected to the HMD 100 and controllable via the HMD 100 in the surrounding environment 170. The user interface 510 includes a selection indicator 515 corresponding to the selected volume level. In some examples, the selection indicator 515 can be mapped to or attached to a specific feature on the coin 330. The mapping of the point on the selection indicator 515 to the corresponding feature on the coin 330 can provide a selection indicator 515 (virtually covered on the coin 330) to move in coordination with the coin 330 or manipulated by the user. In this way, the user manipulation of the coin 330 (movement or rotation of the coin 330) for adjusting the volume level causes the selection indicator 515 to move with the coin 330, thereby providing a visual indication of the desired change in the volume level. In this example, the rotation of the coin 330, which acts as a volume controller for the smart speaker 173, in turn triggers a change in the volume level of the smart speaker 173 connected to the HMD 100 and / or controllable by the HMD 100. In some examples, user interface 510 may include a mode indicator 520 indicating that the system is in a volume control mode and / or a volume level indicator 530 indicating a current volume level. Figure 5D , Figure 5E and Fig. 5F Rotating the coin 330 through increasing volume levels and correspondingly rotating the selection indicator 515 of the user interface 510 is shown. FIG. 5D to FIG. 5F As shown in FIG, as the coin 330 is rotated, a selection indicator 515 of a user interface 510, which may be attached to the coin 330 and / or mapped to a selected feature of the coin 330, rotates with the coin 330. This allows the coin 330 to act as a volume adjustment knob for controlling the volume of the system. The ready availability of the coin 330 and the ability to make the coin 330 act as a volume adjustment knob in this manner provide the user with a tangible user interface for controlling the volume in an AR / MR / VR environment.

[0053] FIG. 6A to FIG. 6E An example is shown in which a user interface is attached to an example pin 360. Fig. 6A , a user has picked up a pin 360 from a work surface 310 such that the pin 360 is in the user's field of view / visible through the lenses of the HMD 100 and is within the field of view of the camera of the HMD 100. The system may detect the pin within the field of view of the camera of the HMD 100 and may identify or recognize the pin 360 based on, for example, the outer contour of the pin 360, detected features in an image of the pin 360 captured by the camera of the HMD 100, and previously stored images and / or information in a database, etc. In some examples, this may be performed by a recognition engine working in conjunction with information stored in a database of additional resources 210 available to the HMD 100. In some examples, the system may detect and recognize features of the pin 360 for use in tracking the movement of the pin 360 and corresponding control of one of the connected / controllable devices in the surrounding environment 170.

[0054] The user may issue a command that causes the pin 360 to function as a user interface tool. In some examples, the command may be a verbal or audible command that is detected by an audio sensor of the HMD 100 and processed by a transcription engine. Figure 6B In the example shown in , in response to a user command to use the pin 360 as a light intensity controller, the user interface 610 may be output on or displayed with the pin 360. The pin 360 may then be manipulated by the user to, for example, control the level of light intensity output by the smart light 172 connected to and controllable by the HMD 100. That is, when movement of the pin 360 is detected, the level of light intensity output by the smart light 172 may be adjusted accordingly. When the pin 360 is moved, the user interface 610 may provide a visual indication of the amount of adjustment to the level of light intensity. For example, as shown in FIG. FIG. 6B to FIG. 6EAs shown in , as pin 360 moves in the direction of arrow A, the level of light intensity output by smart light 172 increases, and the visual indication provided by user interface 610 becomes larger, corresponding to the increased level of light intensity. In this manner, as pin 360 is moved, the light intensity level is adjusted based on the direction and magnitude of the movement of pin 360. This allows pin 360 to act as a light adjustment device for controlling the intensity of light output by smart light 172. The ready availability of pin 360 and the ability to function with pin 360 in this manner provide the user with a tangible user interface for controlling light intensity in an AR / MR / VR environment.

[0055] 7A to 7D Another example is shown in which a user interface is attached to the example pin 360 to select and / or adjust the color of light output by the smart light 172 connected to and controllable by the HMD 100. Fig. 7A , a user has picked up a pin 360 from a work surface 310 such that the pin is in the user's field of view / visible through the lenses of the HMD 100 and within the field of view of the camera of the HMD 100. The system may detect the pin 360 within the field of view of the camera of the HMD 100 and may identify or recognize the pin 360 based on, for example, an outer contour of the pin 360, detected features in an image of the pin 360 captured by the camera of the HMD 100, and previously stored images and / or information in a database, etc. In some examples, this may be performed by a recognition engine working in conjunction with information stored in a database of additional resources 210 available to the HMD 100. In some examples, the system may detect and recognize features of the pin 360 for use in tracking the movement of the pin 360 and corresponding control of one of the connected / controllable devices in the surrounding environment 170.

[0056] The user may issue a command that causes pin 360 to function as a user interface tool. In some examples, the command may be a verbal or audible command that is detected by an audio sensor of HMD 100 and processed by a transcription engine. In response to a user command to use pin 360 as a light color selector, user interface 710 may be output on or displayed with pin 360, such as Fig. 7A The pin 360 can then be manipulated by the user to, for example, select the color of light output by the smart light 172 connected to and controllable by the HMD 100, change the color of light output by the smart light 172, etc. That is, when the movement of the pin 360 is detected, the color of light output by the smart light 172 can be adjusted accordingly based on the color guide provided by the user interface 710. For example, FIG. 7B to FIG. 7D, the color of the light output by the smart light 172 changes as the pin 360 is positioned relative to the color guide provided in the user interface 710. This allows the pin 360 to act as a color selection device for controlling the light output by the smart light 172. The ready availability of the pin 360 and the ability to function with the pin 360 in this manner provides the user with a tangible user interface for controlling light output in an AR / MR / VR environment.

[0057] FIG. 8A to FIG. 8D Another example is shown in which a user interface is attached to an example pin 360. Fig. 8A , the user has picked up pin 360 from work surface 310 so that pin 360 is in the user's field of view / visible through the lenses of HMD 100 and within the field of view of the camera of HMD 100. The system can detect the pin within the field of view of the camera of HMD 100 and can identify or recognize pin 360 as described above. In some examples, the system can detect and recognize features of pin 360 for use in tracking the movement of pin 360 and translating the detected movement into actions in the AR / MR / VR environment.

[0058] The user may issue a command that causes the pin 360 to function as a user interface tool. In some examples, the command may be a verbal or audible command that is detected by an audio sensor of the HMD 100 and processed by a transcription engine. Figure 8B In the example shown in , the user has issued a command to use the pin 360 as a marking or drawing tool. In response to this command, the system generates an annotation corresponding to the detected movement of the pin 360. FIG. 8B to FIG. 8D In the example shown in , when the movement of the pin 360 is detected, a corresponding annotation 810 is made in the AR / MR / VR environment. The tracking of the pin 360 and specifically the features of the pin 360 allows the movement of the pin 360 to be tracked and the corresponding annotation 810 to be displayed. This allows the pin 360 to act as a marking or drawing tool. The ready availability of the pin 360 and the ability to make the pin 360 functional in this way provide users with a tangible user interface for annotating in the AR / MR / VR environment.

[0059] FIG. 6A to FIG. 8D The use of a pin 360 as a selection and control tool in an AR / MR / VR environment is provided by way of example in FIG. Other similar tangible objects that are available to the user in a similarly opportunistic manner may also be employed to provide similar functionality.

[0060] 9A to 9D An example is shown in which an example bottle 320 is selected for use as an interface tool in an AR / MR / VR environment. Fig. 9A, a user has picked up a bottle 320 from a work surface 310 such that the bottle 320 is in the user's field of view / visible through the lenses of the HMD 100 and within the field of view of the camera of the HMD 100. The system can detect the bottle 320 within the field of view of the camera of the HMD 100 and can identify or recognize the bottle 320 based on, for example, the outer contour of the bottle 320, detected features in an image of the bottle 320 captured by the camera of the HMD 100, and previously stored images and / or information in a database, etc. In some examples, this can be performed by a recognition engine working in conjunction with information stored in a database of additional resources 210 available to the HMD 100. In some examples, the system can detect and recognize features of the pin 360 for use in tracking the movement of the pin 360 and corresponding control of one of the connected / controllable devices in the surrounding environment 170.

[0061] The user may issue a command that causes bottle 320 to function as a user interface tool. In some examples, the command may be a verbal or audible command that is detected by an audio sensor of HMD 100 and processed by a transcription engine. FIG. 9B to FIG. 9D In the example shown in , the movement of bottle 320 can be tracked in response to a user command to use bottle 320 as an interface tool or a control tool such as a joystick in a game. The detected movement of bottle 320 can be translated into actions in the AR / MR / VR environment such as commands and / or control inputs in the game, so that bottle 320 acts as a control device for the game. In this way, when the position and / or orientation of bottle 320 changes and the six-degree-of-freedom position of the bottle is detected and tracked, bottle 320 can act as an input / control tool, and the detected movement can be processed as input in the AR / MR / VR environment. The ready availability of bottle 320 and the ability to make the bottle function in this way provide users with tangible control tools in AR / MR / VR environments.

[0062] FIG. 10A to FIG. 10C An example is shown in which the user's hands act as an interface tool in an AR / MR / VR environment. Fig. 10A , the user has positioned his / her hand in the field of view of the camera of the HMD 100. The system can detect the hand within the field of view of the camera of the HMD 100 and can identify or recognize the hand based on, for example, previously stored images and / or information in a database, etc. In some examples, this can be performed by a recognition engine working in conjunction with information stored in a database of additional resources 210 available to the HMD 100. In some examples, the system can detect and recognize features of the hand for use in tracking the movement of the hand to facilitate placement of a user interface, tracking of the hand, etc.

[0063] The user may issue a command that causes the hand to function as a user interface tool. In some examples, the command may be a verbal or audible command that is detected by an audio sensor of the HMD 100 and processed by a transcription engine. Fig. 10A In the example shown in , in response to a first command to display the transcribed text on the user interface device (ie, the user's hand), the first user interface 1010 has been output or displayed on the palm of the user's hand. Fig. 10B In the example shown in , the rotation, or movement or change of posture has triggered the generation of the second user interface 1020 to be output or displayed on the user's hand. In this example, the change of the posture of the user's hand has triggered the translation of the transcription shown in the first user interface 1010. Fig. 10C In the example of FIG. 1 , the tap or touch input has triggered the output or display of the third user interface 1030 on the user's hand, thereby allowing the functionality of the user's hand to be remapped for the purpose of user interface and interaction. The opportunistic availability of the user's hand for use as an interface surface and the ability to function the hand in this way provide the user with a tangible interface and control tool in the AR / MR / VR environment.

[0064] Fig.111 is a flow chart of a method 1100 according to an implementation described herein. The method 1100 provides for the use of tangible physical objects in a surrounding environment as six-degree-of-freedom interfaces and control devices in an AR / MR / VR environment. An object to be implemented as an interface and / or control device can be captured within the field of view of a camera of a computing device (block 1110). Visual patterns, identification features, etc. can be identified in an image of an object captured by a camera of a computing device (block 1120). The detected visual patterns and / or identification features can be compared or matched with visual patterns and / or identification features stored in a database accessible to the computing device. By comparing the detected visual patterns and identification features detected in the image with information previously stored in the database, usage data, etc., the object can be identified and the three-dimensional pose of the object can be estimated (block 1130). Relationship information can be retrieved, for example, from a database accessible to the computing device (block 1140). The relationship information can be used to determine related applications related to the detected object, related user interfaces associated with the detected object, etc. In response to detection of a command related to interacting with the user interface (block 1150), an updated user interface, updated display information, updated functionality, etc. may be generated and output (block 1160). The user command may be, for example, a spoken command detected and transcribed by a transcription engine for processing, a gesture input detected based on six-degree-of-freedom tracking of an object, a tap input detected with respect to the displayed user interface and six-degree-of-freedom tracking of a user's hand, etc. The process may continue until the object is no longer detected in the field of view of a camera of the computing device (block 1170).

[0065] Fig.12An example of a computer device 1200 and a mobile computer device 1250 that can be used with the techniques described herein is shown. The computing device 1200 includes a processor 1202, a memory 1204, a storage device 1206, a high-speed interface 1208 connected to the memory 1204 and a high-speed expansion port 1210, and a low-speed interface 1212 connected to a low-speed bus 1214 and the storage device 1206. Each of the components 1202, 1204, 1206, 1208, 1210, and 1212 are interconnected using various buses and can be mounted on a common motherboard or otherwise installed as appropriate. The processor 1202 can process instructions for execution within the computing device 1200, including instructions stored in the memory 1204 or on the storage device 1206 for displaying graphical information of a GUI on an external input / output device such as a display 1216 coupled to the high-speed interface 1208. In other implementations, multiple processors and / or multiple buses may be used as appropriate, along with multiple memories and multiple types of memory. In addition, multiple computing devices 1200 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

[0066] The memory 1204 stores information within the computing device 1200. In one implementation, the memory 1204 is one or more volatile memory units. In another implementation, the memory 1204 is one or more non-volatile memory units. The memory 1204 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0067] The storage device 1206 can provide mass storage for the computing device 1200. In one implementation, the storage device 1206 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device, a flash memory or other similar solid-state memory device or an array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. A computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 1204, the storage device 1206, or a memory on the processor 1202.

[0068] The high-speed controller 1208 manages bandwidth-intensive operations of the computing device 1200, while the low-speed controller 1212 manages less bandwidth-intensive operations. This allocation of functions is merely an example. In one implementation, the high-speed controller 1208 is coupled to the memory 1204, the display 1216 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 1210 that can accept various expansion cards (not shown). In an implementation, the low-speed controller 1212 is coupled to the storage device 1206 and the low-speed expansion port 1214. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device, such as a switch or a router, for example, through a network adapter.

[0069] As shown in the figure, computing device 1200 can be implemented in many different forms. For example, it can be implemented as a standard server 1220 or multiple times in a group of these servers. It can also be implemented as part of a rack server system 1224. In addition, it can be implemented in a personal computer such as a laptop computer 1222. Alternatively, components from computing device 1200 can be combined with other components in a mobile device (not shown) such as device 1250. Each of these devices can contain one or more of computing devices 1200, 1250, and the entire system can be composed of multiple computing devices 1200, 1250 that communicate with each other.

[0070] The computing device 1250 includes a processor 1252, a memory 1264, an input / output device such as a display 1254, a communication interface 1266, and a transceiver 1268, among other components. The device 1250 may also be provided with a storage device such as a micro drive or other device to provide additional storage. Each of the components 1250, 1252, 1264, 1254, 1266, and 1268 are interconnected using various buses, and several of the components may be mounted on a common motherboard or otherwise as appropriate.

[0071] Processor 1252 can execute instructions within computing device 1250, including instructions stored in memory 1264. The processor can be implemented as a chipset including separate multiple analog and digital processor chips. The processor can provide, for example, coordination for other components of device 1250, such as control of a user interface, applications run by device 1250, and wireless communications performed by device 1250.

[0072] The processor 1252 can communicate with the user through a control interface 1258 and a display interface 1256 coupled to the display 1254. The display 1254 can be, for example, a TFT LCD (thin film transistor liquid crystal display) and an LED (light emitting diode) or OLED (organic light emitting diode) display or other appropriate display technology. The display interface 1256 may include appropriate circuit systems for driving the display 1254 to present graphical information and other information to the user. The control interface 1258 can receive commands from the user and convert them for submission to the processor 1252. In addition, an external interface 1262 in communication with the processor 1252 can be provided to enable near-field communication of the device 1250 with other devices. The external interface 1262 can, for example, provide wired communication in some implementations, or provide wireless communication in other implementations, and multiple interfaces can also be used.

[0073] The memory 1264 stores information within the computing device 1250. The memory 1264 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 1274 may also be provided and connected to the device 1250 via an expansion interface 1272, which may include, for example, a SIMM (single in-line memory module) card interface. Such an expansion memory 1274 may provide additional storage space for the device 1250, or may also store applications or other information of the device 1250. Specifically, the expansion memory 1274 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Thus, for example, the expansion memory 1274 may be provided as a security module for the device 1250, and may be programmed with instructions that permit secure use of the device 1250. In addition, a security application may be provided via a SIMM card together with additional information, such as placing identification information on the SIMM card in an unbreakable manner.

[0074] The memory may include, for example, flash memory and / or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or machine-readable medium, such as memory 1264, expansion memory 1274, or memory on processor 1252, which may be received, for example, via transceiver 1268 or external interface 1262.

[0075] The device 1250 can communicate wirelessly via a communication interface 1266, which may include digital signal processing circuitry, if necessary. The communication interface 1266 may provide for communication in various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, as well as other modes or protocols. Such communication may occur, for example, via a radio frequency transceiver 1268. In addition, short-range communication may occur, such as using Bluetooth, Wi-Fi, or another such transceiver (not shown). In addition, a GPS (global positioning system) receiver module 1270 may provide additional navigation-related and location-related wireless data to the device 1250, which may be used as appropriate by applications running on the device 1250.

[0076] Device 1250 may also communicate audibly using audio codec 1260, which may receive verbal information from a user and convert the verbal information into usable digital information. Audio codec 1260 may also generate audible sounds for the user, such as through a speaker (e.g., in an earpiece of device 1250). Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 1250.

[0077] As shown in the figure, computing device 1250 can be implemented in many different forms. For example, it can be implemented as a cellular phone 1280. It can also be implemented as part of a smart phone 1282, a personal digital assistant, or other similar mobile device.

[0078] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be either special purpose or general purpose and can be coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0079] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium," "computer-readable medium," refer to any computer program product, apparatus, and / or device (e.g., a disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0080] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (LED (light emitting diode), or OLED (organic LED), or LCD (liquid crystal display) monitor / screen) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including sound, voice, or tactile input.

[0081] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a client can interact with an implementation of the systems and techniques described herein), or any combination of these back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0082] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship with each other.

[0083] In some implementations, the computing device depicted in the figure may include sensors that interface with the AR head-mounted device / HMD device 1290 to generate an augmented environment for viewing inserted content in the physical space. For example, one or more sensors included on the computing device 1250 depicted in the figure or another computing device may provide input to the AR head-mounted device 1290, or generally, to the AR space. Sensors may include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 1250 may use sensors to determine the absolute position and / or detected rotation of the computing device in the AR space, and then the absolute position and / or the detected rotation may be used as input to the AR space. For example, the computing device 1250 may be incorporated into the AR space as a virtual object such as a controller, a laser pointer, a keyboard, a weapon, etc. The positioning of the computing device / virtual object by the user when incorporated into the AR space may allow the user to position the computing device so as to view the virtual object in some way in the AR space. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device as if it were an actual laser pointer. The user can move the computing device left to right, up and down, in circles, etc., and use the device in a manner similar to using a laser pointer. In some implementations, the user can use the virtual laser pointer to aim at a target location.

[0084] In some implementations, one or more input devices included on or connected to the computing device 1250 can be used as input to the AR space. The input device can include, but is not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, headphones or earbuds with input functionality, a game controller, or other connectable input device. When the computing device 1250 is incorporated into the AR space, a user interacting with the input device included on the computing device can cause specific actions to occur in the AR space.

[0085] In some implementations, the touch screen of the computing device 1250 can be rendered as a touch pad in the AR space. The user can interact with the touch screen of the computing device 1250. For example, in the AR head mounted device 1290, the interaction is rendered as movement on the rendered touch pad in the AR space. The rendered movement can control the virtual object in the AR space.

[0086] In some implementations, one or more output devices included on the computing device 1250 can provide output and / or feedback to a user of the AR headset 1290 in the AR space. The output and feedback can be visual, tactile, or auditory. The output and / or feedback can include, but are not limited to, vibration, turning one or more lights or flashes on and off or blinking and / or flashing, sounding an alarm, playing a chime, playing a song, and playing an audio file. The output device can include, but is not limited to, a vibration motor, a vibration coil, a piezoelectric device, an electrostatic device, a light emitting diode (LED), a flash, and a speaker.

[0087] In some implementations, computing device 1250 may appear as another object in a computer-generated 3D environment. Interaction by a user with computing device 1250 (e.g., rotating, shaking, touching a touch screen, swiping a finger across a touch screen) may be interpreted as interaction with an object in the AR space. In the example of a laser pointer in the AR space, computing device 1250 appears as a virtual laser pointer in the computer-generated 3D environment. As the user manipulates computing device 1250, the user in the AR space sees the movement of the laser pointer. The user receives feedback on computing device 1250 or on AR head mounted device 1290 from the interaction with computing device 1250 in the AR environment. User interaction with the computing device may be translated into interaction with a user interface generated for a controllable device in the AR environment.

[0088] In some implementations, the computing device 1250 may include a touch screen. For example, a user may interact with the touch screen to interact with a user interface of a controllable device. For example, the touch screen may include a user interface element such as a slider that may control a property of the controllable device.

[0089] Computing device 1200 is intended to represent various forms of digital computers and devices, including but not limited to laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 1250 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are meant to be examples only, and are not meant to limit implementations of the inventions described and / or claimed in this document.

[0090] A number of embodiments have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the specification.

[0091] In addition, the logic flows depicted in the figures do not require the particular order or sequential order shown to achieve the desired results. In addition, other steps can be provided, or steps can be deleted from the described processes, and other components can be added to or removed from the described systems. Therefore, other embodiments are within the scope of the appended claims.

[0092] In addition to the above, controls may be provided to allow the user to make a choice about whether and when the systems, programs, or features described herein may be able to collect user information (e.g., information about the user's social network, social behavior, or activities, occupation, user preferences, or the user's current location) and whether to send content or communications from a server to the user. In addition, before certain data is stored or used, it may be processed in one or more ways to remove personally identifiable information. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location for which location information is obtained may be generalized (such as to a city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.

[0093] Although certain features of the described implementations have been shown as described herein, those skilled in the art will now expect many modifications, substitutions, variations and equivalents. Therefore, it should be understood that the appended claims are intended to cover all these modifications and variations that fall within the scope of the implementations. It should be understood that they are presented only in an exemplary and non-restrictive manner, and various changes in form and detail may be made. Except for mutually exclusive combinations, any part of the devices and / or methods described herein may be combined in any combination. The implementations described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different implementations described.

Claims

1. A computer-implemented method comprising: capturing an image of the surrounding environment by a camera of the computing device; identifying, in the image, physical objects in the surrounding environment; detecting, by the computing device, a command to associate a user interface with the physical object identified in the image; triggering display of the user interface as a virtual object by a display device of the computing device in response to the command; mapping at least one feature of the physical object detected in the image to at least one feature of the user interface; detecting user manipulation of the physical object; as well as In response to the user manipulation of the physical object, a change of the user interface corresponding to the user manipulation of the physical object is triggered. 2 . The computer-implemented method of claim 1 , wherein triggering the change in the user interface comprises controlling operation of an external electronic device in communication with the computing device in response to the user manipulation of the physical object.

3. A computer-implemented method as described in claim 1 or 2, wherein triggering the change of the user interface in response to the user manipulation of the physical object includes changing the functionality of the user interface in response to the user manipulation of the physical object.

4. The computer-implemented method of any one of claims 1 to 3, wherein triggering the change in the user interface in response to the user manipulation of the physical object comprises: tracking, by a hand tracking module accessible by the computing device, hand movements in a field of view of the camera of the computing device; detecting hand movement corresponding to user input into the user interface based on the tracking; as well as An action is performed in response to the input into the user interface.

5. The computer-implemented method of any of the preceding claims, wherein triggering the change in the user interface in response to the user manipulation of the physical object comprises: tracking a position and orientation of the physical object by a pose estimation module accessible by the computing device; detecting a change in at least one of the position or the orientation of the physical object based on the tracking; as well as At least one of the position or orientation of the user interface is adjusted in response to the change in the at least one of the position or the orientation of the physical object.

6. The computer-implemented method of claim 5, wherein detecting the command to associate the user interface with the physical object comprises: attaching a display of the user interface to the physical object based on the mapping of the at least one feature of the physical object to the at least one feature of the user interface; as well as The position and the orientation of the user interface are adjusted in response to the detected changes in the position and the orientation of the physical object such that the display of the user interface remains attached to the physical object.

7. The computer-implemented method of claim 5 or 6, further comprising: At least one of an appearance of the user interface or functionality of the user interface is adjusted based on a distance between the computing device and the physical object.

8. The computer-implemented method of any one of the preceding claims, wherein detecting the command to associate the user interface with the physical object comprises: detecting an audible command by an audio sensor of the computing device; transcribing the audible commands into executable commands by a transcription engine accessible by the computing device; as well as The user interface is associated with the physical object in response to the executable command.

9. The computer-implemented method of any one of the preceding claims, wherein identifying the physical object comprises: detecting a visual pattern in the image; comparing the visual pattern detected in the image to visual patterns associated with images stored in a database accessible to the computing device; as well as The physical object is identified based on the comparison.

10. The computer-implemented method of any of the preceding claims, wherein the computing device is a head mounted display device, and wherein capturing the image and identifying the physical object comprises: capturing the image of the surrounding environment by the camera mounted on a frame portion of the head mounted display device; as well as The physical object held by a user within a field of view of the camera is identified in the image.

11. The computer-implemented method of any of the preceding claims, wherein detecting the user manipulation of the physical object comprises: detecting movement of the at least one feature of the physical object; detecting movement of the at least one feature of the user interface corresponding to the movement of the at least one feature of the physical object; as well as Triggering the change of the control algorithm of the user interface comprises performing an action in response to the user manipulation of the physical object.

12. A computing system comprising: A computing device comprising: a camera that captures images of a surrounding environment external to the computing device; Display device; at least one processing device; and a memory storing instructions that, when executed by the at least one processing device, cause the computing system to perform operations comprising: capturing an image of the surrounding environment by the camera; identifying, in the image, physical objects in the surrounding environment; detecting, by the computing device, a command to designate the physical object as a user interface tool; triggering, in response to the command, display of a user interface by the display device as a virtual object attached to the physical object; mapping at least one feature of the physical object detected in the image to at least one feature of the user interface; detecting user manipulation of the physical object; and A change of the user interface is triggered in response to the user manipulation of the physical object.

13. The computing system of claim 12, wherein triggering the change in the user interface comprises controlling an operation of an external electronic device in communication with the computing device in response to the user manipulation of the physical object.

14. The computing system of claim 12 or 13, wherein triggering the change in the user interface comprises changing functionality of the user interface in response to user manipulation of the physical object.

15. The computing system of any one of claims 12 to 14, wherein triggering the change in the user interface comprises: tracking, by a hand tracking module accessible by the computing device, hand movements in a field of view of the camera of the computing device; detecting hand movement corresponding to user input into the user interface based on the tracking; as well as An action is performed in response to the user input into the user interface.

16. The computing system of any one of claims 12 to 15, wherein triggering the change in the user interface comprises: tracking a position and orientation of the physical object by a pose estimation module accessible by the computing device; detecting a change in at least one of the position or the orientation of the physical object based on the tracking; as well as At least one of the position or orientation of the user interface is adjusted in response to the change in the at least one of the position or the orientation of the physical object.

17. The computing system of any one of claims 12 to 16, wherein triggering the user interface as the display of the virtual object attached to the physical object comprises: At least one of the position or orientation of the user interface is adjusted in response to a detected change in the position or orientation of the physical object such that the display of the user interface remains attached to the physical object as the physical object moves in response to user manipulation of the physical object.

18. The computing system of any one of claims 12 to 17, wherein triggering the change in the user interface comprises: At least one of an appearance or functionality of the user interface is adjusted based on a distance between the computing device and the physical object.

19. The computing system of any one of claims 12 to 18, wherein detecting the command to designate the physical object as the user interface tool comprises: detecting an audible command by an audio sensor of the computing device; transcribing the audible commands into executable commands by a transcription engine accessible by the computing device; as well as The physical object is designated as the user interface tool in response to the executable command.

20. The computing system of any one of claims 12 to 19, wherein detecting the user manipulation of the physical object comprises: detecting movement of the at least one feature of the physical object; detecting movement of the at least one feature of the user interface corresponding to the movement of the at least one feature of the physical object; as well as Triggering the change of the control algorithm of the user interface comprises performing an action in response to the user manipulation of the physical object.