Mobile phone shell for tracking and positioning

Abstract

A housing for a portable device, such as a smartphone, includes a light source, such as an LED, which can be detected and tracked by a head-mounted augmented or virtual reality device when illuminated. The light source may be located at a corner of the housing and may emit infrared light. A relative pose between the smartphone and the head-mounted device may be determined based on computer vision techniques performed on an image captured by the head-mounted device that includes light from the light source. Relative movement between the smartphone and the head-mounted device may be used to provide user input to the head-mounted device as if touch input on a portable device. In some cases, the housing is inductively powered from the portable device.

Description

[0001] Priority Claim

[0002] This application claims the benefit of priority to U.S. Patent Application No. 17 / 970,274, filed Oct. 20, 2022, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to interactions between mobile devices and, more particularly, to using a portable device as a controller for an augmented or virtual reality device. Background Art

[0004] One of the disadvantages of a head-mounted augmented or virtual reality device is that the space on the device for providing controls is relatively small, and the use of these controls may interfere with the potential for the freedom of movement inherent in such a head-mounted device. In some cases, dedicated hand-held controls are provided, but this increases cost and inconvenience because such devices need to be carried with the head-mounted device, charged, etc. Brief Description of the Drawings

[0005] To facilitate the identification of discussion of any particular element or act, one or more of the most significant digits in the reference numerals refer to the figure number in which that element is first introduced.

[0006] Figure 1 is a perspective view of a head-mounted device according to some examples.

[0007] Figure 2 shows a Figure 1 another view of the head-mounted device according to some examples.

[0008] Figure 3 is a block diagram of a networking system showing details of a head-mounted device including Figure 1 according to some examples.

[0009] Figure 4A and Figure 4B show the front and back of a smartphone case 400 according to some examples, respectively.

[0010] Figure 5A is a view of an environment 502 through a user's glasses, where the portable device and the case are visible, according to some examples.

[0011] Figure 5B is a view of the environment through a user's glasses, where the case serves as an input device for the glasses, according to some examples.

[0012] Figure 6 is a flowchart 600 showing a method of providing input to a user interface on a head-mounted device, according to some examples.

[0013] Figure 7 is a graphical representation of a networked environment in which the present disclosure may be deployed, according to some examples.

[0014] Figure 8 is a block diagram showing a software architecture in which the present disclosure may be implemented, according to some examples.

[0015] Figure 9 is a graphical representation of a machine in the form of a computer system within which a set of instructions may be executed to cause the machine to perform any one or more of the methods discussed herein. DETAILED DESCRIPTION

[0016] Typical smart glasses platforms enable a user to interact with various types of virtual content. Such platforms are configured to display virtual content in the smart glasses' lenses over a real-world environment seen through the smart glasses' lenses. To interact with the virtual content, smart glasses typically include embedded sensors. The smart glasses can detect touch or swipe inputs based on information detected by the embedded sensors and can then update the display of the virtual content. Interacting with the embedded sensors to perform various modifications to the virtual content is not very intuitive and has a very steep learning curve. As a result, users cannot accurately perform various desired interactions with the virtual content, which degrades the overall user experience. Additionally, due to the steep learning curve, users often have to re-perform certain actions multiple times until they learn how to use the sensors, which wastes the smart glasses' resources.

[0017] Some smart glasses platforms use hand or gesture recognition to enable a user to interact with virtual content. Specifically, the smart glasses can detect gestures in images captured by the smart glasses and can perform corresponding modifications to the virtual content. Learning how to correctly make such gestures also involves a steep learning curve and can also be non-intuitive. Additionally, performing image processing to detect gestures involves multiple machine learning models, which consume a significant amount of the smart glasses' hardware resources, which can be wasteful and drain the smart glasses' battery. This can also lead to abandonment of using the smart glasses and degrade the overall enjoyment of using the smart glasses.

[0018] The disclosed examples improve the efficiency of using an electronic device by providing an AR device that determines relative movement between a smart phone and a head-mounted device to provide user input to the head-mounted device. This improves the overall efficiency of operating the device and provides a more intuitive system for user use.

[0019] In some examples, the housing of a portable device such as a smartphone includes a light source such as an LED, which can be detected and tracked by a head-mounted augmented or virtual reality device when lit. The light source can be located at a corner of the housing and can emit infrared light. The relative pose between the smartphone and the head-mounted device can be determined by computer vision techniques performed on an image captured by the head-mounted device that includes light from the light source. The smartphone and the head-mounted device are performing visual-inertial odometry (VIO) to independently track their poses. The relative movement between the smartphone and the head-mounted device and the movement of the smartphone can be used to provide user input to the head-mounted device, just like touch input on the smartphone. In some cases, the housing is inductively powered from the portable device.

[0020] In some examples, a computer-implemented method is provided for providing gesture user input to a user interface displayed by a head-mounted device using a portable device in a device housing. The method can include illuminating a light source located on the device housing, capturing an image by the head-mounted device that includes a representation of the light source located in the device housing, determining an initial relative pose between the head-mounted device and the portable device based on the image, determining a change in the initial relative pose due to relative movement of the head-mounted device and the portable device, and updating the user interface based on the change in the initial relative pose.

[0021] Determining the change in the initial relative pose can include capturing additional images that include a representation of the light source located in the device housing, and determining an updated relative pose between the head-mounted device and the portable device based on the additional images.

[0022] Determining the change in the initial relative pose can include obtaining first VIO-based pose information of the head-mounted device, obtaining second VIO-based pose information of the portable device, and determining the change in the initial relative pose between the head-mounted device and the portable device based on the first VIO-based pose information and the second VIO-based pose information.

[0023] The light source in the device housing can be inductively powered from the portable device and can be located at a corner of the device housing.

[0024] The computer-implemented method can further include determining a change in the absolute pose of the portable device, and updating the user interface based on the change in the absolute pose of the portable device.

[0025] In some examples, after a specified interval, additional images that include a representation of the light source located in the device housing are captured, and an updated relative pose between the head-mounted device and the portable device is determined based on the additional images. The initial relative pose is then set to the updated relative pose.

[0026] In some examples, a non-transitory computer-readable storage medium is provided. The computer-readable storage medium includes instructions that, when executed by a processor, cause the processor to perform operations for providing gesture user input to a user interface displayed by a head-mounted device using a portable device in a device housing according to any one of the above methods and limitations. The operations include, but are not limited to: capturing, by the head-mounted device, an image including a representation of a light source located in the device housing, determining an initial relative pose between the head-mounted device and the portable device based on the image, determining a change in the initial relative pose due to relative movement of the head-mounted device and the portable device, and updating the user interface based on the change in the initial relative pose.

[0027] In some examples, a computing system is provided, including: a head-mounted device, a processor and a memory storing instructions, and a device housing for accommodating a portable device. The device housing includes a light source located therein. The instructions stored by the memory, when executed by the processor, configure the system to perform operations for providing gesture user input to a user interface displayed by the head-mounted device when lighting the light source on the device housing according to any one of the above methods and limitations. The operations include, but are not limited to: capturing, by the head-mounted device, an image including a representation of a light source located in the device housing, determining an initial relative pose between the head-mounted device and the portable device based on the image, determining a change in the initial relative pose due to relative movement of the head-mounted device and the portable device, and updating the user interface based on the change in the initial relative pose.

[0028] According to the accompanying drawings, the description, and the claims, other technical features may be readily apparent to those skilled in the art.

[0029] Figure 1 is a perspective view of a head-mounted AR device (e.g., glasses 100) according to some examples. The glasses 100 may include a frame 102 made of any suitable material such as plastic or metal, and the any suitable material includes any suitable shape memory alloy. In one or more examples, the frame 102 includes a first optical element holder or left optical element holder 104 (e.g., a display or lens holder) and a second optical element holder or right optical element holder 106 connected by a bridge portion 112. A first optical element or left optical element 108 and a second optical element or right optical element 110 may be disposed within the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 100.

[0030] The frame 102 additionally includes a left arm piece or left temple piece 122 and a right arm piece or right temple piece 124. In some examples, the frame 102 can be formed from a single piece of material to have an integrated or monolithic construction.

[0031] The glasses 100 can include a computing device such as a computer 120, which can be of any suitable type for being carried by the frame 102, and in one or more examples, the computing device can have a suitable size and shape to be partially disposed within one of the temple pieces 122 or temple piece 124. The computer 120 can include one or more processors as well as a memory, wireless communication circuitry, and a power source. As discussed below, the computer 120 includes low-power circuitry, high-speed circuitry, and a display processor. Various other examples can include these elements configured differently or integrated together in different ways. Additional details of aspects of the computer 120 can be implemented as shown for the data processor 302 discussed below.

[0032] The computer 120 additionally includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed within the left temple piece 122 and is electrically coupled to the computer 120 disposed within the right temple piece 124. The glasses 100 can include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, a transmitter, or a transceiver (not shown), or a combination of such devices.

[0033] The glasses 100 include a first imaging device or left imaging device 114 and a second imaging device or right imaging device 116. Although two imaging devices are depicted, other examples contemplate the use of a single or additional (i.e., more than two) imaging devices. In one or more examples, in addition to the left imaging device 114 and the right imaging device 116, the glasses 100 further include any number of input sensors or other input / output devices. Such sensors or input / output devices can additionally include biometric sensors, location sensors, motion sensors, and the like.

[0034] In some examples, the left imaging device 114 and the right imaging device 116 provide video frame data for the glasses 100 to use to extract 3D information from real-world scenes.

[0035] The glasses 100 may further include a touchpad 126, which is mounted to or integrated with one or both of the left temple piece 122 and the right temple piece 124. The touchpad 126 is generally vertically arranged, and in some examples, the touchpad 126 is approximately parallel to the user's temple. As used herein, generally vertically aligned means that the touchpad is more vertical than horizontal, although potentially more vertical than this. Additional user input may be provided by one or more buttons 128, which in the illustrated example are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. The one or more touchpads 126 and buttons 128 provide a means by which the glasses 100 can receive input from a user of the glasses 100.

[0036] Figure 2 The glasses 100 are shown from the user's perspective. For clarity, Figure 1 several of the elements shown have been omitted. As Figure 1 described, Figure 2 the glasses 100 shown include a left optical element 108 and a right optical element 110 that are respectively fixed within a left optical element holder 104 and a right optical element holder 106.

[0037] The glasses 100 include a forward optical assembly 202 that includes a right projector 204 and a right near-eye display 206; and a forward optical assembly 210 that includes a left projector 212 and a left near-eye display 216.

[0038] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or optical elements such as mirrors, lenses, or prisms). The light 208 emitted by the projector 204 encounters the diffractive structure of the waveguide of the near-eye display 206, which directs the light towards the user's right eye to provide an image that overlays the view of the real world seen by the user on or within the right optical element 110. Similarly, the light 214 emitted by the projector 212 encounters the diffractive structure of the waveguide of the near-eye display 216, which directs the light towards the user's left eye to provide an image that overlays the view of the real world seen by the user on or within the left optical element 108. The combination of the GPU, the forward optical assembly 202, the left optical element 108, and the right optical element 110 provides the optical engine of the glasses 100. The glasses 100 use the optical engine to generate an overlay of the user's real-world view, including displaying a 3D user interface to the user of the glasses 100.

[0039] However, it should be understood that other display technologies or configurations may be used within the optical engine to display images to the user within the user's field of view. For example, instead of the projector 204 and waveguide, an LCD, LED, or other display panel or surface may be provided.

[0040] In use, a user of the glasses 100 will be presented with information, content, and various 3D user interfaces on the near-eye display. As described in more detail herein, the user can then interact with the glasses 100 using the touchpad 126 and / or buttons 128, voice input or touch input on an associated device (e.g., Figure 3 the portable device 328 shown in ) and / or hand movements, positioning, and location detected by the glasses 100.

[0041] Figure 3 is a block diagram of a networked system 300 showing details of the glasses 100 according to some examples. The networked system 300 includes the glasses 100, a portable device 328, and a server system 332. The portable device 328 can be a smart phone, tablet computer, phablet, laptop computer, access point, or any other such device capable of connecting to the glasses 100 using a low-power wireless connection 336 and / or a high-speed wireless connection 334. The portable device 328 is connected to the server system 332 via a network 330. The network 330 can include any combination of wired and wireless connections. The server system 332 can be one or more computing devices that are part of a service or network computing system. The details of the software architecture 804 or the machine 900 described in Figure 8 and Figure 9 can be used to implement the portable device 328 and any elements of the server system 332 and the network 330, respectively.

[0042] The glasses 100 include a data processor 302, a display 310, one or more imaging devices 308, and additional input / output elements 316. The input / output elements 316 can include a microphone, an audio speaker, a biometric sensor, additional sensors, or additional display elements integrated with the data processor 302. Examples of the input / output elements 316 are further discussed in Figure 8 and Figure 9 For example, the input / output elements 316 can include any I / O component 906, and the I / O component 906 includes an output component 928, an inertial measurement unit 936, etc. Examples of the display 310 are discussed in Figure 2 In the specific examples described herein, the display 310 includes displays for the user's left and right eyes.

[0043] The data processor 302 includes an image processor 306 (e.g., a video processor), a GPU, and a display driver 338, a tracking module 340, an interface 312, low-power circuitry 304, and high-speed circuitry 320. The components of the data processor 302 are interconnected by a bus 342.

[0044] The interface 312 refers to any source of user commands provided to the data processor 302. In one or more examples, the interface 312 is a physical button that, when pressed, sends a user input signal from the interface 312 to the low-power processor 314. The low-power processor 314 may process the pressing and then immediately releasing of such a button as a request to capture a single image, and vice versa. The low-power processor 314 may process the pressing of such a button for a first time period as a request to capture video data while the button is pressed and to stop video capture when the button is released, where the video captured while the button is pressed is stored as a single video file. Alternatively, pressing the button for a long time period may capture a still image. In some examples, the interface 312 may be any mechanical switch or physical interface capable of accepting user input associated with requesting data from the imaging device 308. In other examples, the interface 312 may have software components or may be associated with commands received wirelessly from another source such as from the portable device 328.

[0045] The image processor 306 includes circuitry for receiving signals from the imaging device 308 and processing those signals from the imaging device 308 into a format suitable for storage in the memory 324 or suitable for transmission to the portable device 328. In one or more examples, the image processor 306 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from the imaging device 308, as well as volatile memory used by the microprocessor in operation.

[0046] The low-power circuitry 304 includes a low-power processor 314 and low-power wireless circuitry 318. These elements of the low-power circuitry 304 may be implemented as separate elements or may be implemented as part of a single system-on-a-chip on a single IC. The low-power processor 314 includes logic for managing other elements of the glasses 100. As described above, for example, the low-power processor 314 may receive user input signals from the interface 312. The low-power processor 314 may also be configured to receive input signals or command communications from the portable device 328 via a low-power wireless connection 336. The low-power wireless circuitry 318 includes circuit elements for implementing a low-power wireless communication system. Bluetooth TM Smart (also known as Bluetooth TMLow power consumption) is a standard implementation of a low-power wireless communication system that can be used to implement the low-power radio circuit system 318. In other examples, other low-power communication systems can be used.

[0047] The high-speed circuit system 320 includes a high-speed processor 322, a memory 324, and a high-speed radio circuit system 326. The high-speed processor 322 can be any processor capable of managing the high-speed communication and operation of any general-purpose computing system used by the data processor 302. The high-speed processor 322 includes processing resources used to manage high-speed data transmission on the high-speed wireless connection 334 using the high-speed radio circuit system 326. In some examples, the high-speed processor 322 executes an operating system such as the LINUX operating system or other such operating systems such as Figure 8 the operating system 812. In addition to any other duties, the high-speed processor 322 that executes the software architecture of the data processor 302 manages data transmission with the high-speed radio circuit system 326. In some examples, the high-speed radio circuit system 326 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, the high-speed radio circuit system 326 can implement other high-speed communication standards.

[0048] The memory 324 includes any storage device capable of storing the camera device data generated by the camera device 308 and the image processor 306. Although the memory 324 is shown integrated with the high-speed circuit system 320, in other examples, the memory 324 can be a separate stand-alone element of the data processor 302. In some such examples, electrical wiring can provide a connection from the image processor 306 or the low-power processor 314 to the memory 324 through the chip including the high-speed processor 322. In other examples, the high-speed processor 322 can manage the addressing of the memory 324 such that the low-power processor 314 will direct the high-speed processor 322 whenever a read or write operation involving the memory 324 is desired.

[0049] The tracking module 340 estimates the position and orientation ("pose") of the glasses 100. For example, the tracking module 340 uses image data and corresponding inertial data and GPS data from the camera device 308 and the position component 940 to track the location and determine the pose of the glasses 100 relative to a reference frame (e.g., the real-world environment). The tracking module 340 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine the updated three-dimensional pose of the glasses 100, which indicates changes in the relative position and orientation with respect to physical objects in the real-world environment. The tracking module 340 enables the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 310.

[0050] The GPU and display driver 338 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 310 when the glasses 100 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 338 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the user's position and orientation relative to physical objects in the user's real-world environment.

[0051] One or more of the functions or operations described herein may also be performed in an application residing on the glasses 100 or the portable device 328 or a remote server. For example, one or more of the functions or operations described herein may be performed by one of the applications 806 such as the messaging application 846.

[0052] Figure 4A and Figure 4B show the front and back of a smartphone case 400 according to some examples. The case 400 includes a body 402 having side walls 404 that define a recess in the front side into which a smartphone is received. The case 400 includes a camera aperture 406 through which one or more cameras on the smartphone can capture images and videos. The case 400 also includes buttons 408 that have positions corresponding to the buttons on the smartphone.

[0053] The case 400 also has an induction pad and control electronics 410 that may protrude from the back of the case 400 in some examples. The induction pad and control electronics 410 receive power from a corresponding induction pad in the smartphone, and the power can directly power the control electronics or charge a battery in the case 400.

[0054] The case 400 also includes four LEDs 412 in protrusions 414 located at the four corners of the case. The LEDs 412, which emit infrared light in some examples, provide an optical output that is fixed relative to the smartphone, and this optical output can be used by the glasses 100 to determine the relative pose between the smartphone and the glasses. The LEDs 412 can be provided in different shapes, but in some examples provide light dots or light spots whose position or centroid is easier to determine than other LED shapes. There can also be a different number of LEDs 412 in different positions that may not be symmetric, but a symmetric arrangement of an even number of LEDs 412 may be more visually pleasing to the user of the case.

[0055] The LED 412 is illuminated by the induction pad and control electronics 410 based on user input or a signal from an application running on the portable device 328 or glasses 100. In some examples, the LED 412 is illuminated based on direct user input (such as pressing an "on" button on the housing 400 or in the application user interface), or can be indirect (such as by selecting a function within the application that would benefit from the use of the LED).

[0056] Instructions and other data for turning the LED on or off can be transmitted from the portable device 328 to the housing 400 via NFC or other short-range communication protocols. If the housing 400 is inductively powered, the induction coil in the portable device 328 will be activated prior to any instructions transmitted from the portable device 328 to the housing 400. In some examples, the LED will simply be turned on and off by activating and deactivating the induction coil in the portable device 328. In additional examples, activation of the induction coil in the portable device 328 can occur automatically based on detecting that the housing 400 has been placed on the portable device, such as as a result of detecting a suitable change in the capacitance or resonance of the transmission induction coil in the portable device 328. In such a case, all the user needs to do is place the housing 400 on the portable device 328, and the LED 412 will be illuminated and can be tracked by the glasses 100.

[0057] Since the LED 412 emits infrared light in some examples, the user of the housing 400 may not know when they are illuminated or not. To provide visual feedback, the rear of the housing 400 can be provided with a logo 416 that is illuminated from behind by an LED that emits visible light. Such an LED is connected to the induction pad and control electronics 410 and will illuminate the logo 416 when the LED 412 is illuminated.

[0058] Figure 5A is a view of the environment 502 through the user's glasses 100 according to some examples, where the portable device 328 and the housing 400 are visible. In this case, the LED 412 is illuminated to allow relative pose determination as described below. Figure 5AAlso shown is text prompt 504 displayed by glasses 100 to the user, which indicates to the user to hold the phone in front of glasses 100 so that the relative pose can be determined. In most cases, it will not be necessary to prompt the user to hold the portable device 328 in front of them, as its use as a gesture input device should naturally involve holding it in front of the user. Once LED 412 is lit, tracking of portable device 328 by glasses 100 can occur without indicating to the user to do anything, as long as portable device 328 is within the field of view of glasses 100. However, if portable device 328 is not within the field of view of glasses 100, or when providing a tutorial on gesture input, an appropriate message can be provided.

[0059] Both the portable device 328 and the glasses 100 are performing visual inertial odometry (VIO) to determine and maintain tracking of their respective positions and orientations ("poses") within the environment 502. The relative pose between the glasses 100 and the portable device 328 can be determined by having the glasses 100 extract the position of the LED 412 in an image captured by one or more of the imaging devices 308 in the glasses 100. Since the relative position of the LED 412 on the housing 400 is known, the glasses 100 can determine the pose transformation between the housing 400 (and thus the portable device 328) and the glasses 100. Subsequent images captured by one of the glasses 100 can be similarly used to update any changes in the relative pose between the portable device 328 and the glasses 100, or to provide a periodic recalibration of the relative pose determined by the VIO running on both devices, as will be described in more detail below.

[0060] In some cases, the coordinate systems of the body 402 / portable device 328 and the glasses 100 are adjusted or aligned such that the poses of the two devices are represented in the same coordinate system, e.g., a "world" coordinate system associated with the local physical environment.

[0061] The glasses 100 may have less processing power and battery capacity than the portable device 328. To avoid the glasses 100 having to continuously determine the relative pose between the glasses 100 and the portable device 328 by extracting the position of the LED 412 in the images captured by the glasses 100, the change in the relative pose may be determined based on the changes in the respective poses of the glasses 100 and the portable device 328 determined by VIO. To avoid drift that may occur from such inertial tracking, an optical determination of the relative pose using the position of the LED 412 in the images captured by the glasses 100 may be performed periodically to recalibrate the relative pose between the glasses 100 and the portable device 328. The periodic recalibration occurs at a frequency that balances sufficient accuracy between the relative poses and power savings of the glasses 100. In some examples, the relative pose is recalibrated once per second.

[0062] In some examples, the determination of the VIO-based relative pose update is performed by the portable device 328. The VIO-based pose of the glasses 100 is transmitted to the portable device 328 via a short-range transmission method such as Bluetooth over a previously established link between the glasses 100 and the portable device 328. The portable device 328 then determines the relative pose between the two devices and transmits it back to the glasses 100.

[0063] Also transmitted from the portable device 328 to the glasses 100 is the actual pose (or pose update) of the portable device 328 itself, and any button or touchscreen input received at the portable device 328. As will be discussed in more detail below, the relative pose between the glasses 100 and the portable device 328 and the pose of the portable device 328 itself are used as inputs to the user interface of the glasses 100.

[0064] Figure 5B is a view of the environment 502 through the user's glasses 100 according to some examples, where the movement of the housing 400 serves as an input device for the glasses 100. In Figure 5B a Bluetooth communication link has been established between the portable device 328 and the glasses 100, and the coordinate systems of the two devices are aligned as described above. The portable device 328 reports its pose and any user touch input received to the glasses 100 via the short-range data transmission link. In some examples, the portable device 328 also determines and reports an update to the relative pose between the portable device 328 and the glasses 100 based on the VIO-based pose update received from the glasses 100.

[0065] The glasses 100 periodically capture images including the LED 412 located in the housing 400, and determine the relative pose between the glasses 100 and the portable device 328 based on the position of the LED 412 in the images.

[0066] As shown, portable device 328 is displaying user interface elements such as a directional touchpad 514, which can be used, for example, to receive touch inputs that will move cursor 512 displayed by glasses 100. Movement of cursor 512 can also be controlled by movement of portable device 328. Also shown are various AR elements 510 displayed by glasses 100 that are anchored to portable device 328. AR elements 510 can include, for example, virtual reality indicators or beams that can provide a visual indication of the location to which portable device 328 is pointed. Also shown is a user interface window 508 displayed by glasses 100. Included in window 508 are user interface elements 506 such as buttons or other UI elements that can be selected to perform various functions.

[0067] Using a combination of gestures and touch inputs leveraging portable device 328, a user can interact with augmented reality elements (such as AR elements 510 and window 508) displayed by the glasses. For example, by pointing portable device 328 (and thus housing 400) in different directions, a user can move cursor 512 within the field of view of display 310, for example to place it above one of the user interface elements 506 (such as a button) in window 508. The button can then be activated by a single touch on the touchscreen of portable device 328. For example, window 508 can also be moved within the field of view of glasses 100 by moving the cursor over window 508, pressing and holding on the touchscreen, and then pointing portable device 328 to drag the window to its intended position in the environment or in display 310 of glasses 100.

[0068] Portable device 328 thus serves as a six-degree-of-freedom controller with touch input for glasses 100. This alleviates some of the challenges of providing a user interface on glasses 100.

[0069] Which pose or poses are used to affect UI elements will vary. The positioning of AR elements 510 fixed to portable device 328 depends on the relative pose of portable device 328 and glasses 100, as it will be necessary to update the perceived position of AR elements 510 displayed by glasses 100 when the two devices move relative to each other. In cases where movement of cursor 512 is affected by movement of portable device 328, the position of the cursor within the field of view of the glasses can depend only on the pose of portable device 328 to allow the user to turn their head without causing cursor 512 to move out of the field of view of glasses 100.

[0070] Finally, if the position of window 508 is fixed relative to glasses 100, then neither a change in the pose of glasses 100 nor a change in the pose of portable device 328 will affect the position of window 508 in the field of view of glasses 100. On the other hand, if the position of window 508 is fixed relative to an item in the surrounding environment (e.g., above window 516), then the position of window 508 in the field of view of glasses 100 will change with the pose of glasses 100 rather than with the pose of portable device 328.

[0071] In some examples, the user interface element will have an associated parameter that specifies which coordinate system (the coordinate system of glasses 100, portable device 328, or the environment) the particular user interface element is located in, thereby specifying which device movement will (or will not) affect the movement of the corresponding user interface element. Additional user input can also be included or available, such as voice commands.

[0072] Figure 6 FIG. 600 is a flowchart showing a method of providing input to a user interface on a head-mounted device. For illustrative purposes, the operations of flowchart 600 are described herein as occurring serially or linearly. However, multiple operations of flowchart 600 can occur in parallel. Additionally, the operations of flowchart 600 need not be performed in the order shown, and / or one or more blocks of flowchart 600 need not be performed and / or can be replaced by other operations. The operations of flowchart 600 can be performed by glasses 100, portable device 328, or some combination thereof.

[0073] The method begins at operation 602, where portable device 328 or glasses 100 receive user input that initiates, specifies, or requests the use of portable device 328 or housing 400 to provide gesture user input. Such user input can be, for example, selecting a function within an application that uses gesture input running on glasses 100 or portable device 328, or by the user selecting an application that uses gesture input (such as a game).

[0074] In operation 604, portable device 328 enables its induction coil to provide power to housing 400 when needed. In operation 606, in response to an instruction received from portable device 328 or glasses 100 or activation of the on / off button on housing 400, the LEDs 412 are illuminated by the induction pad and control electronics 410. Instructions can then be provided by glasses 100 or portable device 328 to hold the phone in front of the wearer of glasses 100, as Figure 5A shown.

[0075] Then, in operation 608, the glasses 100 capture an image of the housing with the LED illuminated. In operation 610, the relative pose between the glasses 100 and the portable device 328 is determined, and in operation 612, the glasses 100 receive the pose of the portable device 328 from the portable device 328 via short-range data transmission.

[0076] In operation 614, the glasses 100 update the user interface displayed by the glasses 100 based on the updated relative pose and pose update, such as as discussed above with reference to Figure 6 In some examples, the update in operation 614 sets an initial value of the user interface assuming that the current poses of the two devices represent an initial or neutral pose of the user of the portable device 328, e.g., to position the cursor 512 at the center of the field of view of the glasses 100.

[0077] In operation 616, a VIO-based pose update is obtained from / by the portable device 328 and the glasses 100. Then in operation 618, a relative pose update between the portable device 328 and the glasses 100 is determined. In some examples, the VIO-based pose update of the glasses 100 is transmitted to the portable device 328, and the portable device 328 determines the relative pose update and transmits it back to the glasses 100. In other examples, the VIO-based pose update of the portable device 328 is transmitted to the glasses 100, and the glasses 100 determines the relative pose update.

[0078] In operation 620, the glasses 100 update the user interface displayed by the glasses 100 based on the updated relative pose and pose update, such as as discussed above with reference to Figure 6 In operation 622, the glasses 100 determine whether it is time to recalibrate the relative pose between the glasses 100 and the portable device 328 to address any potential drift in the relative pose determined by VIO running on the two devices. If it is not yet time, the method returns to operation 616, where a VIO pose update is obtained, and the method continues from operation 616. If it is time, the method returns to operation 608, where the glasses 100 capture another image including the LED 412, and the method continues from operation 608.

[0079] The method in flow chart 600 continues until a user input is received to terminate the use of the portable device 328 as a gesture input device. For example, this can be exiting a relevant function within the application or exiting the application.

[0080]

[0081] ​A variety of examples are envisioned. Example 1 is a computer-implemented method for providing gesture user input to a user interface displayed by a head-mounted device using the movement of a portable device within a device housing, the method comprising: capturing, by the head-mounted device, an image including a representation of a light source located within the device housing and illuminated; determining an initial relative pose between the head-mounted device and the portable device based on the image; determining a change in the initial relative pose resulting from relative movement between the head-mounted device and the portable device; and updating the user interface based on the change in the initial relative pose.

[0082] In Example 2, the subject matter according to Example 1 includes, wherein determining the change in the initial relative pose includes: capturing an additional image including a representation of a light source located within the device housing; and determining an updated relative pose between the head-mounted device and the portable device based on the additional image.

[0083] In Example 3, the subject matter according to Examples 1 to 2 includes, wherein determining the change in the initial relative pose includes: obtaining first pose information of the head-mounted device; obtaining second pose information of the portable device; and determining the change in the initial relative pose between the head-mounted device and the portable device based on the first pose information and the second pose information.

[0084] In Example 4, the subject matter according to Example 3 includes: after a specified interval, capturing an additional image including a representation of a light source located within the device housing; determining an updated relative pose between the head-mounted device and the portable device based on the additional image; and setting the initial relative pose to the updated relative pose.

[0085] In Example 5, the subject matter according to Examples 1 to 4 includes powering the light source in the device housing inductively from the portable device.

[0086] In Example 6, the subject matter according to Examples 1 to 5 includes: determining a change in the absolute pose of the portable device; and updating the user interface based on the change in the absolute pose of the portable device.

[0087] In Example 7, the subject matter according to Examples 1 to 6 includes, wherein the light source is located at a corner of the device housing.

[0088] Example 8 is a non-transitory computer-readable storage medium that includes instructions that, when executed by a processor, cause the processor to perform operations for providing gesture user input to a user interface displayed by a head-mounted device using movement of a portable device within a device housing. The operations include: capturing, by the head-mounted device, an image that includes a representation of an illuminated light source located within the device housing; determining an initial relative pose between the head-mounted device and the portable device based on the image; determining a change in the initial relative pose due to relative movement between the head-mounted device and the portable device; and updating the user interface based on the change in the initial relative pose.

[0089] In example 9, the subject matter according to example 8 includes, wherein determining the change in the initial relative pose includes: capturing an additional image that includes a representation of a light source located within the device housing; and determining an updated relative pose between the head-mounted device and the portable device based on the additional image.

[0090] In example 10, the subject matter according to examples 8 to 9 includes, wherein the operations further include: obtaining first pose information of the head-mounted device; obtaining second pose information of the portable device; and determining a change in the initial relative pose between the head-mounted device and the portable device based on the first pose information and the second pose information.

[0091] In example 11, the subject matter according to example 10 includes, wherein the operations further include: after a specified interval, capturing an additional image that includes a representation of a light source located within the device housing; determining an updated relative pose between the head-mounted device and the portable device based on the additional image; and setting the initial relative pose to the updated relative pose.

[0092] In example 12, the subject matter according to examples 8 to 11 includes, wherein the operations further include: inductively powering the light source within the device housing from the portable device.

[0093] In example 13, the subject matter according to examples 8 to 12 includes, wherein the operations further include: determining a change in the absolute pose of the portable device; and updating the user interface based on the change in the absolute pose of the portable device.

[0094] Example 14 is a computing system, including: a head-mounted device; a processor and a memory storing instructions; and a device housing for accommodating a portable device, the device housing including a light source located therein, wherein the instructions stored by the memory, when executed by the processor, configure the system to perform an operation for providing a gesture user input to a user interface displayed by the head-mounted device when the light source on the device housing is lit, the operation including: capturing, by the head-mounted device, an image including a representation of the light source located in the device housing; determining an initial relative pose between the head-mounted device and the portable device based on the image; determining a change in the initial relative pose due to relative movement between the head-mounted device and the portable device; and updating the user interface based on the change in the initial relative pose.

[0095] In Example 15, the subject matter according to Example 14 includes, wherein determining the change in the initial relative pose includes: capturing additional images including a representation of the light source located in the device housing; and determining an updated relative pose between the head-mounted device and the portable device based on the additional images.

[0096] In Example 16, the subject matter according to Examples 14 to 15 includes, wherein determining the change in the initial relative pose includes: obtaining pose information of the head-mounted device based on a first VIO; obtaining second VIO-based pose information of the portable device; and determining a change in the initial relative pose between the head-mounted device and the portable device based on the first VIO-based pose information and the second VIO-based pose information.

[0097] In Example 17, the subject matter according to Example 16 includes: after a specified interval, capturing additional images including a representation of the light source located in the device housing; determining an updated relative pose between the head-mounted device and the portable device based on the additional images; and setting the initial relative pose to the updated relative pose.

[0098] In Example 18, the subject matter according to Examples 14 to 17 includes, wherein the operation further includes: inductively powering the light source in the device housing from the portable device.

[0099] In Example 19, the subject matter according to Examples 14 to 18 includes, wherein the operation further includes: determining a change in the absolute pose of the portable device; and updating the user interface based on the change in the absolute pose of the portable device.

[0100] In Example 20, the subject matter according to Examples 14 to 19 includes, wherein the light source is located at a corner of the device housing.

[0101] Example 21 is at least one machine-readable medium including instructions that, when executed by a processing circuit, cause the processing circuit to perform operations as described in any one of Examples 1 to 20.

[0102] Example 22 is a device that includes means for implementing any one of Examples 1 to 20. Example 23 is a system that implements any one of Examples 1 to 20. Example 24 is a method that implements any one of Examples 1 to 20.

[0103] Example 25 is a device housing having a perimeter and a recess defined therein for receiving a device, and one or more light sources are located in the perimeter of the device. Example 26 is the device housing according to Example 25, further including an induction pad and control electronics for receiving power from a corresponding induction pad in the device, and the power can directly power the control electronics or charge a battery in the housing.

[0104] Figure 7 is a block diagram showing an example messaging system 700 for exchanging data (e.g., messages and associated content) over a network. The messaging system 700 includes multiple instances of a portable device 328 that host several applications including a messaging client 702 and other applications 704. The messaging client 702 is communicatively coupled via a network 330 (e.g., the Internet) to other instances of the messaging client 702 (e.g., hosted on corresponding other portable devices 328), a messaging server system 706, and a third-party server 708. The messaging client 702 can also communicate with the locally hosted applications 704 using an application programming interface (API).

[0105] The messaging client 702 is capable of communicating and exchanging data via the network 330 with other messaging clients 702 and with the messaging server system 706. The data exchanged between the messaging clients 702 and between the messaging client 702 and the messaging server system 706 includes functionality (e.g., commands for activating functionality) and payload data (e.g., text, audio, video, or other multimedia data).

[0106] The messaging server system 706 provides server - side functionality to a particular messaging client 702 via the network 330. Although some functions of the messaging system 700 are described herein as being performed by the messaging client 702 or by the messaging server system 706, whether some functions reside within the messaging client 702 or within the messaging server system 706 can be a design choice. For example, it may be technically preferable to initially deploy some technologies and functions within the messaging server system 706, but then migrate the technologies and functions to the messaging client 702 when the portable device 328 has sufficient processing power.

[0107] The messaging server system 706 supports various services and operations provided to the messaging client 702. Such operations include transmitting data to the messaging client 702, receiving data from the messaging client 702, and processing data generated by the messaging client 702. As an example, the data can include message content, user device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. Data exchange within the messaging system 700 is activated and controlled through functions available via the user interface (UI) of the messaging client 702.

[0108] Turning now specifically to the messaging server system 706, an application programming interface (API) server 710 is coupled to an application server 714 and provides a programming interface to the application server 714. The application server 714 is communicatively coupled to a database server 716, which facilitates access to a database 720 that stores data associated with messages processed by the application server 714. Similarly, a web server 724 is coupled to the application server 714 and provides a web - based interface to the application server 714. To this end, the web server 724 processes incoming network requests via the Hyper - Text Transfer Protocol (HTTP) and several other related protocols.

[0109] The Application Programming Interface (API) server 710 receives and transmits message data (e.g., commands and message payloads) between the portable device 328 and the application server 714. Specifically, the Application Programming Interface (API) server 710 provides a set of interfaces (e.g., routines and protocols) that can be invoked or queried by the messaging client 702 to activate the functions of the application server 714. The Application Programming Interface (API) server 710 exposes various functions supported by the application server 714, including: account registration; login functionality; sending a message from a particular messaging client 702 to another messaging client 702 via the application server 714, sending a media file (e.g., an image or video) from the messaging client 702 to the messaging server 712, and possible access by another messaging client 702; setting a collection of media data (e.g., a story); retrieving a friend list of the user of the portable device 328; retrieving such a collection; retrieving messages and content; adding and deleting entities (e.g., friends) to and from an entity graph (e.g., a social graph); locating friends in the social graph; and opening application events (e.g., related to the messaging client 702).

[0110] The application server 714 hosts a number of server applications and subsystems, including, for example, the messaging server 712, the image processing server 718, and the social network server 722. The messaging server 712 implements a number of message handling techniques and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 702. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 702. Given the hardware requirements for such processing, other processor- and memory-intensive processing of data can also be performed by the messaging server 712 on the server side.

[0111] The application server 714 also includes an image processing server 718 that is dedicated to performing various image processing operations, typically on images or videos within the payload of messages sent from or received at the messaging server 712.

[0112] The social network server 722 supports various social networking functions and services and makes these functions and services available to the messaging server 712. To this end, the social network server 722 maintains and accesses an entity graph within the database 720. Examples of functions and services supported by the social network server 722 include identifying other users in the messaging system 700 with whom a particular user has a relationship or whom the particular user is "following", and also include identifying the interests of a particular user and other entities.

[0113] The messaging client 702 can notify a user of the portable device 328 or other users associated with such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 702 can provide a notification related to the current or recent use of a game by one or more members of a user group to participants in a conversation (e.g., a chat session) in the messaging client 702. One or more users can be invited to join an active session or initiate a new session. In some examples, the shared session can provide a shared augmented reality experience in which multiple people can collaborate or participate.

[0114] Figure 8 FIG. 800 is a block diagram showing a software architecture 804 that can be installed on any one or more of the devices described herein. The software architecture 804 is supported by hardware such as a machine 802 that includes a processor 820, a memory 826, and I / O components 838. In this example, the software architecture 804 can be conceptually thought of as a stack of layers, where each layer provides a specific function. The software architecture 804 includes layers such as an operating system 812, libraries 808, frameworks 810, and applications 806. In operation, the application 806 makes an API call 850 through the software stack and receives a message 852 in response to the API call 850.

[0115] The operating system 812 manages the hardware resources and provides common services. The operating system 812 includes, for example: a kernel 814, services 816, and drivers 822. The kernel 814 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 814 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. The services 816 can provide other common services to other software layers. The drivers 822 are responsible for controlling or interfacing with the underlying hardware. For example, the drivers 822 can include a display driver, a camera driver, or a low-power driver, a flash driver, a serial communication driver (e.g., a Universal Serial Bus (USB) driver), drivers, an audio driver, a power management driver, etc.

[0116] The library 808 provides low-level common infrastructure used by the application 806. The library 808 may include a system library 818 (e.g., the C standard library), which provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, the library 808 may include an API library 824, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), High Efficiency Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., the OpenGL framework for presenting graphical content in two-dimensional (2D) and three-dimensional (3D) on a display, GLMotif for implementing 3D user interfaces), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite that provides various relational database functions), a web library (e.g., WebKit that provides web browsing functions), etc. The library 808 may also include various other libraries 828 to provide many other APIs to the application 806.

[0117] The framework 810 provides high-level common infrastructure used by the application 806. For example, the framework 810 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 810 may provide a wide range of other APIs that can be used by the application 806, some of which may be specific to a particular operating system or platform.

[0118] In an example, the application 806 may include a home application 836, a contacts application 830, a browser application 832, a book reader application 834, a location application 842, a media application 844, a messaging application 846, a gaming application 848, and various other applications such as third-party applications 840. The application 806 is a program that executes functions defined in the program. One or more of the applications 806 can be created using various programming languages, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., the C language or assembly language). In a specific example, the third-party application 840 (e.g., an application developed using the ANDROID TM or IOS TM software development kit (SDK)) can be an application on platforms such as IOS TM 、ANDROID TM 、 Mobile software running on the mobile operating system of a Phone or another mobile operating system. In this example, third-party application 840 can call API call 850 provided by operating system 812 to facilitate the functions described herein.

[0119] Figure 9 is a diagrammatic representation of a machine 900 or computing device in which instructions 910 (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine 900 to perform any one or more of the methods discussed herein. For example, instructions 910 can cause the machine 900 to perform any one or more of the methods described herein. Instructions 910 transform the general unprogrammed machine 900 into a particular machine 900 programmed to perform the described and illustrated functions in the described manner. Machine 900 can operate as a stand-alone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 900 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 900 can include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), PDAs, entertainment media systems, cellular telephones, smart phones, mobile devices, head-mounted devices (e.g., smart watches), smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 910 specifying actions to be taken by machine 900. Further, although a single machine 900 is shown, the term "machine" can also be regarded as including a collection of machines that individually or jointly execute instructions 910 to perform any one or more of the methods discussed herein.

[0120] Machine 900 can include a processor 902, a memory 904, and I / O components 906, which can be configured to communicate with each other via a bus 944. In an example, processor 902 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a processor 908 and a processor 912 that execute instructions 910. The term "processor" is intended to include multi-core processors that can include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 9A number of processors 902 are shown, but machine 900 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0121] Memory 904 includes main memory 914, static memory 916, and storage unit 918, all of which are accessible by processor 902 via bus 944. Main memory 904, static memory 916, and storage unit 918 store instructions 910 for implementing any one or more of the methods or functions described herein. During execution of instructions 910 by networking system 300, instructions 910 may also reside, in whole or in part, within main memory 914, within static memory 916, within machine-readable medium 920 within storage unit 918, within one or more of processors 902 (e.g., within a cache memory of the processor), or in any suitable combination thereof.

[0122] I / O components 906 can include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, etc. The specific I / O components 906 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that I / O components 906 can include Figure 9 many other components not shown. In various examples, I / O components 906 can include output components 928 and input components 932. Output components 928 can include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. Input components 932 can include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instrument), haptic input components (e.g., a physical button, a touch screen that provides the location and / or force of a touch or touch gesture, or other haptic input components), audio input components (e.g., a microphone), etc.

[0123] In additional examples, I / O component 906 can include: a biometric component 934, an inertial measurement unit 936, an environmental component 938, or a location component 940, as well as various other components. For example, biometric component 934 includes components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biometric signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retina recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), and so on. Inertial measurement unit 936 includes acceleration sensor components (e.g., accelerometers), gravity sensor components, rotational sensor components (e.g., gyroscopes), and so on. Environmental component 938 includes, for example, lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting the concentration of hazardous gases for safety or measuring pollutants in the atmosphere), or other components that can provide indications, measurement results, or signals corresponding to the surrounding physical environment. Location component 940 includes location sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect the air pressure from which altitude can be obtained), orientation sensor components (e.g., magnetometers), and so on.

[0124] A variety of techniques can be used to implement communication. I / O component 906 also includes a communication component 942, which is operable to couple networked system 300 to network 922 or device 924 via coupling 930 and coupling 926, respectively. For example, communication component 942 can include a network interface component that interfaces with network 922 or another suitable device. In additional examples, communication component 942 can include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, components (e.g., low power), components, and other communication components that provide communication via other modalities. Device 924 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0125] In addition, communication component 942 can detect an identifier or include components operable to detect an identifier. For example, communication component 942 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., for detecting optical sensors such as one-dimensional barcodes, e.g., Universal Product Code (UPC) barcodes; multi-dimensional barcodes, e.g., Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). Additionally, various information can be obtained via communication component 942, such as a location obtained via Internet Protocol (IP) geolocation, a location obtained via position triangulation of signals, a location obtained via detecting an NFC beacon signal that can indicate a specific location, etc.

[0126] Various memories (e.g., memory 904, main memory 914, static memory 916, and / or the memory of processor 902) and / or storage unit 918 can store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instruction 910), when executed by processor 902, cause various operations to implement the disclosed examples.

[0127] Instructions 910 can be transmitted or received over network 922 via a network interface device (e.g., the network interface component included in communication component 942), using a transmission medium and using any one of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 910 can be transmitted or received to device 924 via coupling 926 (e.g., a peer-to-peer coupling) using a transmission medium.

[0128] A "carrier signal" refers to any non-transitory medium capable of storing, encoding, or carrying instructions executable by a machine and includes digital or analog communication signals or other non-transitory media to facilitate the communication of such instructions. Instructions can be transmitted or received over a network via a network interface device using a transmission medium.

[0129] "User equipment" or "client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other user equipment or client devices. User equipment or client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, laptop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0130] "Communication network" refers to one or more parts of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), plain old telephone service (POTS) network, cellular telephone network, wireless network, network, another type of network, or a combination of two or more such networks. For example, a network or a part of a network can include a wireless network or a cellular network, and the coupling can be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other types of cellular or wireless couplings. In this example, the coupling can implement any data transmission technology among various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolved data optimized (EVDO) technology, general packet radio service (GPRS) technology, GSM enhanced data rates for GSM evolution (EDGE) technology, 3rd Generation Partnership Project (3GPP) including 3G, 4th Generation Wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), worldwide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other data transmission technologies defined by various standards setting organizations, other long distance protocols, or other data transmission technologies.

[0131] "Component" refers to a device, physical entity, or logic having boundaries defined by functionality or subroutine calls, branch points, APIs, or other technical definitions provided for partitioning or modularizing a particular processing or control function. Components can be combined with other components via their interfaces to perform machine processing. A component can be an encapsulated functional hardware unit designed to be used with other components and a program for a particular function that typically performs related functions. Components can constitute software components (e.g., code implemented on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit capable of performing some operations and can be configured or arranged in a particular physical manner. In various examples, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or a portion of an application) to operate to perform some operations as described herein as a hardware component. A hardware component can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic permanently configured to perform some operations. A hardware component can be a dedicated processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component can also include programmable logic or circuitry that is temporarily configured by software to perform some operations. For example, a hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a particular component of a machine) customized to perform the configured function and is no longer a general-purpose processor. It will be understood that the decision to implement a hardware component mechanically in dedicated and permanently configured circuitry or in circuitry that is temporarily configured (e.g., configured by software) can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, i.e., an entity that is physically constructed, permanently configured (e.g., hard-wired), or temporarily configured (e.g., programmed) to operate in a particular manner or to perform some operations described herein. Considering an example where a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any given time. For example, in the case where a hardware component includes a general-purpose processor that is configured by software to become a dedicated processor, the general-purpose processor can be configured at different times to be respective different dedicated processors (e.g., including different hardware components). The software accordingly configures one or more particular processors to, for example, constitute a particular hardware component at one time and a different hardware component at a different time. A hardware component can provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components can be considered communicatively coupled.In the case where multiple hardware components exist simultaneously, communication can be achieved through signal transmission between two or more hardware components or within two or more hardware components (e.g., via appropriate circuits and buses). In an example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving the information from the memory structure. For example, one hardware component can perform an operation and store the output of the operation in a memory device communicatively coupled thereto. Then, another hardware component can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device and can operate on resources (e.g., collection of information). The various operations of the example methods described herein can be performed by one or more processors temporarily configured (e.g., via software) or permanently configured to perform the relevant operations. Whether temporarily configured or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more of the operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be partially implemented by a processor, where a particular one or more processors are examples of hardware. For example, some of the operations of the method can be performed by one or more processors or processor-implemented components. In addition, one or more processors can also operate to support the execution of relevant operations in a "cloud computing" environment or operate as "software as a service" (SaaS). For example, some of the operations can be performed by a group of computers (as an example of a machine including processors), where these operations can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of some of the operations can be distributed among processors, reside within a single machine, and be deployed across multiple machines. In some examples, the processor or processor-implemented components can be located in a single geographical location (e.g., within a home environment, an office environment, or a server farm). In other examples, the processor or processor-implemented components can be distributed across multiple geographical locations.

[0132] "Computer-readable medium" or "machine storage medium" refers to a single or multiple non-transitory storage devices and / or media (e.g., a centralized or distributed database, and / or associated cache and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory as well as optical and magnetic media, including memory internal or external to a processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and may be used interchangeably in the present disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, some of which are covered by the term "signal medium".

[0133] "Processor" refers to any circuit or virtual circuit (physical circuits simulated by logic executed on an actual processor) that manipulates data values according to control signals (e.g., "commands", "opcodes", "machine codes", etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor can be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor can also be a multi-core processor having two or more independent processors (sometimes called "cores") that can execute instructions simultaneously.

[0134] "Signal medium" refers to any intangible medium that can store, encode, or carry instructions executable by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" can be regarded as including any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose one or more characteristics are set or changed in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in the present disclosure.

[0135] Without departing from the scope of the present disclosure, changes and modifications can be made to the disclosed examples. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A computer-implemented method for providing gesture user input to a user interface displayed by a head-mounted device using the movement of a portable device within a device housing: Capturing, by the head-mounted device, an image including a representation of a light source located within the device housing and illuminated; Determining an initial relative pose between the head-mounted device and the portable device based on the image; Determining a change in the initial relative pose resulting from relative movement between the head-mounted device and the portable device; And Updating the user interface based on the change in the initial relative pose.

2. The computer-implemented method according to claim 1, Wherein, Determining the change in the initial relative pose includes: Capturing an additional image including a representation of the light source located within the device housing; and Determining an updated relative pose between the head-mounted device and the portable device based on the additional image.

3. The computer-implemented method according to claim 1, Wherein, Determining the change in the initial relative pose includes: Obtaining first pose information of the head-mounted device; Obtaining second pose information of the portable device; and Determining the change in the initial relative pose between the head-mounted device and the portable device based on the first pose information and the second pose information.

4. The computer-implemented method according to claim 3, further Including: After a specified interval, capturing an additional image including a representation of the light source located within the device housing; Determining an updated relative pose between the head-mounted device and the portable device based on the additional image; And Setting the initial relative pose to the updated relative pose.

5. The computer-implemented method according to claim 1, further Including: Inductively powering the light source within the device housing from the portable device.

6. The computer-implemented method according to claim 1, further Including: Determining a change in the absolute pose of the portable device; And Updating the user interface based on the change in the absolute pose of the portable device.

7. The computer-implemented method according to claim 1, Wherein, The light source is located at a corner of the device housing.

8. A non-transitory computer-readable storage medium including instructions that, when executed by a processor, cause the processor to perform operations for providing gesture user input to a user interface displayed by a head-mounted device using the movement of a portable device within a device housing, the operations Including: Capturing, by the head-mounted device, an image including a representation of a lighted light source located within the device housing; Determining an initial relative pose between the head-mounted device and the portable device based on the image; Determining a change in the initial relative pose resulting from relative movement between the head-mounted device and the portable device; And Updating the user interface based on the change in the initial relative pose.

9. The non-transitory computer-readable storage medium according to claim 8, Wherein, Determining the change in the initial relative pose includes: Capture additional images including a representation of the light source located in the device housing; and Determine an updated relative pose between the head-mounted device and the portable device based on the additional images.

10. The non-transitory computer-readable storage medium according to claim 8, wherein, the operations further include: Obtain first pose information of the head-mounted device; Obtain second pose information of the portable device; and Determine a change in the initial relative pose between the head-mounted device and the portable device based on the first pose information and the second pose information.

11. The non-transitory computer-readable storage medium according to claim 10, wherein, the operations further include: After a specified interval, capture additional images including a representation of the light source located in the device housing; Determine an updated relative pose between the head-mounted device and the portable device based on the additional images; and Set the initial relative pose to the updated relative pose.

12. The non-transitory computer-readable storage medium according to claim 8, wherein, the operations further include: Inductively power the light source in the device housing from the portable device.

13. The non-transitory computer-readable storage medium according to claim 8, wherein, the operations further include: Determine a change in the absolute pose of the portable device; and Update the user interface based on the change in the absolute pose of the portable device.

14. A computing system, comprising: A head-mounted device; A processor and a memory storing instructions; and A device housing for accommodating a portable device, the device housing including a light source located therein, wherein the instructions stored by the memory, when executed by the processor, configure the system to perform operations for providing gesture user input to a user interface displayed by the head-mounted device when the light source on the device housing is lit, the operations including: The head-mounted device captures an image including a representation of the light source located in the device housing; Determine an initial relative pose between the head-mounted device and the portable device based on the image; Determine a change in the initial relative pose resulting from relative movement of the head-mounted device and the portable device; and Update the user interface based on the change in the initial relative pose.

15. The computing system according to claim 14, wherein, Determining the change in the initial relative pose includes: Capturing additional images including a representation of the light source located in the device housing; and Determining an updated relative pose between the head-mounted device and the portable device based on the additional images.

16. The computing system according to claim 14, wherein, Determining the change in the initial relative pose includes: Obtaining first VIO-based pose information of the head-mounted device; Obtaining second VIO-based pose information of the portable device; and Determine a change in the initial relative pose between the head-mounted device and the portable device based on the first VIO-based pose information and the second VIO-based pose information.

17. The computing system according to claim 16, further comprising: After a specified interval, capture additional images including representations of the light source located in the device housing; Determine an updated relative pose between the head-mounted device and the portable device based on the additional images; and Set the initial relative pose to the updated relative pose.

18. The computing system according to claim 14, wherein the operation further comprises: Inductively power the light source in the device housing from the portable device.

19. The computing system according to claim 14, wherein the operation further comprises: Determine a change in the absolute pose of the portable device; and Update the user interface based on the change in the absolute pose of the portable device.

20. The computing system according to claim 14, wherein the light source is located at a corner of the device housing.

Images