Method for object motion tracking and mixed reality system

Abstract

An example of a mixed reality computing device that utilizes a remote sensor and a local sensor as an input device is disclosed. In one example, a mixed reality computing device includes an image sensor, a remote input device, a processor, and a storage including stored instructions. The stored instructions are executable by the processor to perform the following operations: perform object motion tracking and environment tracking based on output from the image sensor, and adjust parameters of motion tracking while maintaining environment tracking in response to detecting that the remote input device is in use.

Description

[0001] This application is a divisional application of an application with application date of 2017 / 01 / 16, application number 201780004806.X (international application number PCT / US2017 / 013638), and name “Method for tracking object motion using a remote device of a mixed reality system and a mixed reality system”. Background Art

[0002] Various types of user input mechanisms may be used to provide control signals to a computing device. For example, a computing device may utilize natural user input in the form of sensed body movements performed by a user (e.g., body gestures, eye gaze, voice commands), which are converted into control signals for the computing device.

[0003] Overview

[0004] An example of a mixed reality computing device that utilizes a remote sensor and a local sensor as an input device is disclosed. In one example, the mixed reality computing device includes an image sensor, a remote input device, a processor, and a storage including stored instructions. The stored instructions are executable by the processor to perform the following operations: perform object motion tracking and environment tracking based on output from the image sensor; and adjust parameters of motion tracking while maintaining environment tracking in response to detecting that the remote input device is in use.

[0005] This summary is provided to introduce in simplified form a selection of concepts that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages mentioned in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 and Figure 2 An example use environment is shown in which a user wears an example mixed reality computing device.

[0007] Figure 3 is a flow chart illustrating an example method for performing object motion tracking using one or more of a mixed reality computing device and a remote input device.

[0008] Figure 4 A block diagram of an example remote input device is shown.

[0009] Figure 5 and Figure 6 An example mixed reality computing device is schematically illustrated. Detailed Description

[0010] Wearable mixed reality computing systems may rely on hand and arm gestures to control various actions performed by the mixed reality computing system. These hand and arm gestures may be sensed and measured using sensors that are capable of measuring the depth of objects in front of them, such as image sensors (e.g., depth image sensors or stereo image sensor configurations). These image sensors tend to have a limited sensing field of view, and because the image sensors are placed on the wearable computing system, the sensor field of view is typically centered near the operator's head. As such, the user may frequently raise and lower their arms to place their hands within the sensor field of view for detection.

[0011] Thus, according to the embodiments disclosed herein, a remote input device configured to be held in the hand of an operator of a mixed reality computing system can be selectively used to perform gesture input. By allowing gesture input to be detected via a sensor of the remote input device, the need for the operator to repeatedly perform motion gestures in front of an image sensor of the wearable computing system can be reduced.

[0012] As described in more detail below, gesture motion tracking performed via a remote input device (e.g., via an inertial motion sensor) can be less power-intensive than gesture motion tracking performed based on image sensor data. Thus, one or more parameters related to gesture motion tracking based on an image sensor can be adjusted when the system detects that the remote input device is in use. For example, gesture motion tracking using an onboard image sensor of a mixed reality computing device can be disabled when remote motion tracking is in use, or the frame rate of image sensor gesture motion tracking can be reduced for gesture motion tracking. More generally, the capabilities and power consumption characteristics of each of a plurality of sensors capable of performing gesture motion tracking placed across a wearable computing system and a remote input device can be considered when performing gesture motion tracking to adjust sensor usage in various object tracking contexts. This can help achieve a suitable balance between power consumption and motion tracking accuracy in each object tracking context, thereby saving power where lower accuracy is required while achieving higher accuracy where appropriate.

[0013] Figure 1 An example use environment 100 is shown in which a user 102 can interact with an example mixed reality (MR) computing device. Figure 1 104, but other examples may take any other suitable form. Figure 5 and Figure 6 To describe example hardware configurations separately.

[0014] The HMD 104 provides the user 102 with a see-through view of the environment 100. The HMD also displays an augmented reality image to the user. In one example, the HMD is a stereoscopic display device, where two separate augmented reality images are each displayed on a respective left-eye and right-eye display of the HMD. When viewed by a wearer of the HMD (e.g., user 102), the two augmented reality images together form an augmented reality object that can be perceived by the wearer as part of the environment 100. Figure 1 An example augmented reality object 106a is depicted.

[0015] The HMD 104 may include one or more image sensors capable of imaging the environment 100. For example, the HMD 104 may include a depth camera that uses time of flight, structured light, or other suitable depth sensing techniques to image the environment 100. Based on the output from the depth camera, an environment model may be generated, and an augmented reality image may be displayed based on the environment model.

[0016] To image the environment, the depth camera of the HMD 104 may transmit imaging light at a first relatively high brightness so as to image the environment at a relatively long distance from the HMD, such as 30 feet. Because the environment is unlikely to change frequently, the imaging light transmitted to perform modeling of the environment (also referred to herein as environment tracking) may be output at a relatively low frame rate (e.g., 1 Hz).

[0017] In contrast, when the HMD 104 transmits light to perform object tracking (which may include tracking a user's hand or finger), imaging light may be transmitted at a second relatively low brightness to image the environment within a shorter distance from the HMD (such as within three feet). Because the position of the object being tracked may change frequently, the imaging light transmitted to perform object tracking may be transmitted at a relatively high frame rate (e.g., 30 Hz).

[0018] exist Figure 1 In the example illustrated in , user 102 is creating an augmented reality drawing via gesture input. As shown, user 102 is creating a first drawing (depicted as augmented reality object 106a) along wall 108. However, this augmented reality drawing is a non-limiting example of an action that may be performed in response to gesture input, and other actions are possible. For example, gesture input may be used to control a user interface or other suitable actions. Figure 1 As shown in , user 102 is performing gesture input with his or her hands, and one or more sensors of HMD 104 (such as the depth sensors described above) detect the movement of the user's hands.

[0019] Figure 2A second example use environment 200 is shown in which the user 102 is performing gesture input to create another drawing (depicted as augmented reality object 106b) along the wall 108. The gesture motion performed by the user 102 to create the drawing is detected based on one or more sensors of the remote input device 110 communicatively coupled to the HMD 104.

[0020] The remote input device 110 may include any suitable one or more sensors capable of detecting user gestures and / or user actuated inputs. In one example, the remote input device 110 includes an inertial measurement unit (IMU) including one or more accelerometers, gyroscopes, momentary switches, and / or magnetometers. The remote input device 110 may further include user actuable input devices such as buttons, tracking wheels, joysticks, etc. The remote input device 110 may communicate with the HMD device 104 via a suitable mechanism (such as a wireless connection). Movement data (e.g., vertical and horizontal displacements relative to a reference, pitch, roll, yaw) and / or user actuated inputs sensed by the IMU may be communicated to the HMD directly or via a remote computing system and used by the HMD to control actions on the HMD, such as converting gesture movements into user interface controls.

[0021] As described above, the remote input device 110 can selectively adjust the operation of the depth sensor system of the HMD. This allows the power drawn by the HMD to be reduced because no power is used for the imaging system when the imaging system is turned off. More specifically, the HMD 104 may include a depth image sensor configured to transmit imaging light (e.g., infrared light) and determine the depth of each object in the sensor's field of view based on the reflected imaging light. The movement of each object in the field of view (especially the user's hand, arm, or finger) can be tracked to control various aspects of the HMD using the detected gestures as input. However, due to the output imaging light, the reception of the reflected imaging light, and the processing of the signal from the received light at a relatively high frame rate to determine the object movement, such gesture / object motion tracking may be resource intensive. Thus, when the remote input device is in use, the gesture / object motion tracking based on the HMD image sensor can be adjusted to reduce power consumption. In other examples, the image sensor of the HMD can be used in combination with the sensor of the remote input device to achieve a suitable balance between accuracy and power consumption.

[0022] Figure 3 A flow chart depicting an example method 300 for performing object motion tracking using one or both of: a computing system image sensor; and one or more sensors of a remote input device communicatively coupled to the computing system. As an example, the method 300 may be implemented by a mixed reality computing system such as Figure 1 and Figure 2 HMD 104) combined with a remote input device (such as Figure 2remote input device 110) to execute.

[0023] At 302, method 300 includes performing environment tracking at a first light source brightness and a first frame rate based on output from an image sensor. As explained above, a computing system (e.g., an HMD) may include an image sensor, such as a depth sensor, that can perform both environment and object tracking. When performing environment tracking, the image sensor system may transmit imaging light at a first brightness to image the environment at a relatively long distance from the image sensor at a lower frame rate. The received reflected imaging light may be used to construct an environment model, which in turn may be used to generate an augmented reality object for display via a display device of the HMD.

[0024] At 304, method 300 includes performing object motion tracking at a second light source brightness and a second frame rate based on output from the image sensor. Imaging light transmitted from the image sensor system of the computing system may also be transmitted at a lower brightness for tracking objects within a close distance to the image sensor. Object tracking may be performed at a relatively high frame rate. The received imaging light may be used to track the movement of an object (such as a user's hand) to, for example, perform gesture control of the HMD.

[0025] At 306, method 300 determines whether the remote input device is in use. The remote input device may be determined to be in use in any suitable manner. As an example, if an image sensor of the HMD detects that the remote input device is in its field of view (e.g., using object recognition analysis of image data via an edge recognition method or other suitable algorithm), the remote input device may be determined to be in use. As another example, the remote input device may be determined to be in use based on receiving a user input configured to activate the remote input device (e.g., in the case where the user actuates a switch or other input mechanism of the remote input device). As a further example, the remote input device may be determined to be in use based on an output received from a sensor of the remote input device (e.g., in the case where a motion signal from the remote input device indicates a change from a non-moving state to a moving state, an orientation change to an orientation associated with use, etc.).

[0026] If it is determined that the remote input device is not in use, the method 300 proceeds to 328, as described in more detail below. On the other hand, if it is determined that the remote input device is in use, the method 300 includes maintaining environment tracking based on output from the image sensor at the first light source brightness and the first frame rate at 308. Thus, regardless of which sensor(s) will be used to perform object motion tracking, environment tracking is maintained to continue to generate an accurate environment model.

[0027] In addition, at 310, method 300 includes adjusting one or more parameters of object motion tracking based on output from the image sensor. When the remote input device is in use, in some instances, it may be advantageous to disable or adjust object motion tracking performed by the HMD based on output from the HMD image sensor to save power usage. Any suitable method may be used to determine how to adjust the one or more parameters of object motion tracking. In some examples, a power function may be utilized, wherein the power function represents the accuracy and power consumption of each motion tracking sensor for various possible states of these sensors. This may allow object tracking parameters that appropriately balance accuracy and power consumption for the current use context to be selected for these sensors. Accordingly, at 310, method 300 determines a power function based at least in part on the power consumption of each sensor and the motion tracking accuracy. Data about the power consumption of each sensor may be predetermined and / or tracked over time. Similarly, motion tracking accuracy data may also be predetermined (e.g., during device development or manufacturing), and / or may be determined during use (e.g., via calibration).

[0028] Any suitable parameters for object tracking may be adjusted. For example, adjusting object motion tracking may include reducing the object motion tracking frame rate for image-based object tracking, as indicated at 314. As previously explained, object motion tracking performed by the HMD may be performed at a relatively high frame rate (e.g., 30 Hz). Thus, when the remote input device is in use, the frame rate of the object motion tracking may be reduced to save power. The frame rate may be reduced to any suitable rate. As a non-limiting example, a 30 Hz rate may be reduced to 10 Hz. In another example, adjusting object motion tracking may include deactivating image-based object motion tracking, as indicated at 316.

[0029] Adjusting object motion tracking may further include activating one or more gesture filters, as indicated at 318. In some instances, object tracking may be performed only by the remote input device. As such, the user will be less likely to place his or her hand in front of the image sensor of the HMD for the purpose of gesture input to control the HMD. Thus, the one or more gesture motion filters may be used to filter out motion detected in front of the image sensor of the HMD, thereby ignoring or discarding potential unnecessary or unintentional motion in front of the HMD and reducing power consumption. The one or more gesture motion filters may discard motion data that occurs within a threshold distance of the image sensor of the HMD, discard motion data that occurs only within a threshold time period, and / or perform any other suitable filtering.

[0030] In some examples, discrete predefined operating modes may be used in various object tracking contexts. Thus, adjusting one or more parameters of object motion tracking may be based on changes in the selected sensor operating mode, as indicated at 320. Various contextual factors may be used to determine the sensor operating mode, such as the characteristics of the object being tracked (e.g., large or small), the type of motion occurring (e.g., linear versus nonlinear), and the desired accuracy of the tracking (e.g., based on the current usage context, such as interacting with large, well-spaced graphical user interface features, or creating detailed, small-scale augmented reality images). Power function information may be used to select a mode. As a more specific example, a first sensor operating mode may include performing motion tracking using outputs from a remote input device IMU and not from an HMD image sensor. This mode may be used for object motion tracking situations where power conservation may be prioritized over accuracy, such as for tracking linear motion of an object. Similarly, an example second sensor operating mode may include performing motion tracking using outputs from an image sensor (with or without a remote input device IMU output). This mode may be used in situations where the accuracy of the remote input device IMU may outweigh the power conservation of the IMU.

[0031] At 322, method 300 includes performing object motion tracking based on output from the remote input device. This may include using output from an image sensor in combination with output from one or more sensors of the remote input device based on a selected sensor operating mode, as indicated at 324. This may also include correcting input device sensor output drift based on output from the image sensor, as indicated at 326. As explained above, the remote input device may include an IMU to detect movement of the remote input device. Although motion tracking based on output from an IMU may have relatively low power consumption compared to motion tracking based on output from a depth sensor, IMU motion data is susceptible to drift, which may accumulate over time and create errors in object tracking. Thus, when the IMU output begins to drift (e.g., the IMU indicates that an object is greater than a threshold distance from the location of the object as determined by output from the depth sensor), object tracking performed by the image sensor may be used to correct the IMU drift.

[0032] At 328, method 300 includes performing an action on the HMD based on the environment tracking and the object motion tracking. The action may include any suitable computing device action, such as selecting a user interface object, creating an augmented reality object, cursor control, game control, and / or other operating system or application interaction. Method 300 then returns.

[0033] Figure 4A block diagram of an example remote input device 400 configured to provide position / movement data to a mixed reality computing system (such as HMD 104) is schematically shown. The remote input device 400 includes an input subsystem 402 and a communication subsystem 404. The remote input device 400 may optionally include a logic device 406, a storage device 408, and / or Figure 4 Other components not shown.

[0034] The input subsystem 402 may include or interface with one or more user-actuable input devices such as buttons, a tracking wheel, a keyboard, a mouse, a touch screen, or a game controller. The input subsystem 402 may further include one or more sensors configured to determine the relative position and / or track motion of the remote input device 400. Examples of such sensors include, but are not limited to, an inertial motion unit (IMU), an image sensor, a momentary switch, a magnetometer, and / or other sensors.

[0035] The IMU may be configured to provide position and / or orientation data of the remote input device 400 to a logic subsystem, a controller of an HMD in communication with the remote input device 400, and / or a remote computing system. In one embodiment, the IMU may be configured as a three-axis or three-degree-of-freedom position sensor system. This example position sensor system may include, for example, three gyroscopes for indicating or measuring changes in the orientation of the remote input device 400 around three orthogonal axes (e.g., x, y, z) in 3D space.

[0036] In another example, the IMU can be configured as a six-axis or six-degree-of-freedom position sensor system. Such a configuration may include three accelerometers and three gyroscopes to indicate or measure changes in position of the remote input device 400 along three orthogonal axes and changes in device orientation (e.g., roll, pitch, yaw) about the three orthogonal axes.

[0037] The communication subsystem 404 can be configured to communicatively couple the remote input device 400 with one or more other computing devices, such as the HMD 104. The communication subsystem 404 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem can be configured to communicate via a wireless telephone network or a wired or wireless local or wide area network. In some embodiments, the communication subsystem can allow the remote input device 400 to send and / or receive messages to and / or from other devices via a network, such as the Internet.

[0038] Data from each remote input device sensor may be sent individually or processed locally to determine combined sensor data, and this data may be sent to the HMD 104 for the purpose of experience control. In one example, pitch and yaw data may be derived from the fusion of these motion sensors. When the momentary switch is pressed, the precise orientation of the system is measured and used as the 'zero' point for both pitch and yaw. When the device is angularly displaced, all angular data is reported relative to the starting orientation.

[0039] In other embodiments, a capacitive touch sensor located below the area where the user's thumb would normally rest is used. When the thumb (or other part of the body) is pressed against the surface of the capacitive touch sensor and moved, the horizontal and vertical position of the finger is sent to the HMD 104 via a wireless connection.

[0040] In further embodiments, a multi-directional switch (also known as a joystick) is used. When the joystick is moved, the horizontal and vertical positions of the joystick are sent to the HMD via a wireless connection. In different embodiments, various switches and wheel input devices (circular pads that can be continuously rotated with a finger, which report the amount and direction of rotation) can be included. When the wheel is moved, the rotation and switch data are sent to the HMD 104 via a wireless connection. In each of the embodiments described above, these switches can be configured so that they report the force applied.

[0041] The data connection between the remote input device 400 and the HMD can be bidirectional. This bidirectional data link can be used to send information that triggers a tactile event on the device. In the above embodiments, a tactile feedback system can be included to provide tactile feedback to the holder of the remote input device.

[0042] In a further example, the orientation and position of the remote input device 400 can be measured using multiple cameras that detect a surrounding scene, track the scene, and use the position of the scene to infer the position and orientation of the remote input device. Position and orientation calculations can be performed on the remote input device (e.g., via a logic and / or storage subsystem), or the data can be sent to a remote system for analysis and the results sent back to the remote input device and / or HMD. The position and orientation data (and changes to the data over time) can then be used to generate commands that are sent to the HMD via a wireless connection.

[0043] A fingerprint sensor may be included to allow user authentication. Authentication may also be accomplished via specific motion data (gestures) performed by the user instead of an alphanumeric password entered via a keyboard. The device may include a non-rechargeable battery or a rechargeable battery. Charging of these units (where these units are rechargeable) may be accomplished via a wired connection or via a wireless connection.

[0044] The power drawn by the device can be reduced by keeping it in a low power state when not in active use. This inactive state can be detected by using a capacitive sensor that detects the presence of a human hand. When the sensor no longer detects the presence of a hand, it can switch to a low power state. The device can also enter a low power state after no motion is detected (the motion sensor does not report motion after a preset period of time). It can then be awakened by detecting new motion or when an included momentary switch (or other suitable switch) is actuated.

[0045] The HMD 104 can provide visual feedback related to the motion and orientation of the remote input device. In addition, the motion and displacement data from the remote input device can control the speed of information scrolling on the HMD. The motion and displacement data can also control the absolute cursor position on the HMD.

[0046] Additionally, the remote input device may include a haptic feedback device.Haptic feedback may be generated locally based on device sensors and state information, or generated from the HMD based on data internal to the HMD.

[0047] Multiple finger actuated input devices may be included, such as sliders (reporting discrete digital actuation data or analog reporting continuous actuation data), pressure sensors for measuring and reporting finger / hand pressure between the finger / hand and the device, or pressure sensors for measuring force between the device and an external object (such as the surface of a table or desk).

[0048] When logic device 406 is included, logic device 406 includes one or more physical devices configured to execute instructions. For example, a logic device may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise obtain a desired result.

[0049] When storage device 408 is included, storage device 408 includes one or more physical devices configured to hold instructions executable by logic devices to implement the methods and processes described herein. When implementing such methods and processes, the state of storage device 408 may be transformed—for example, to store different data.

[0050] Figure 5A non-limiting example of an HMD device 500 is shown in the form of a pair of wearable glasses having a see-through display 502. In other examples, the HMD device may take any other suitable form in which a transparent, semi-transparent, and / or opaque display is supported in front of one or both eyes of a viewer. The HMD device 500 is a non-limiting example of a mixed reality computing device that may be used with a remote input device.

[0051] HMD device 500 includes a controller 504 configured to control the operation of a perspective display 502. The perspective display 502 can cause images (such as holographic objects) to be delivered to the eyes of the wearer of the HMD device 500. The perspective display 502 can be configured to visually enhance the appearance of a real-world physical environment to the wearer who observes the physical environment through the transparent display. For example, the appearance of the physical environment can be enhanced to create a mixed reality environment by the graphical content presented via the perspective display 502. In one example, the display can be configured to display one or more visual digital content items. In some cases, the digital content items can be virtual objects covered in front of the real world environment. Similarly, in some cases, the digital content items can be elements of real-world objects of the real world seen by the perspective display 502.

[0052] Any suitable mechanism may be used to display images via the see-through display 502. For example, the see-through display 502 may include an image generating element (such as, for example, a see-through organic light emitting diode (OLED) display) located within the lens 506. As another example, the see-through display 502 may include a light modulator located within the frame of the HMD device 500. In this example, the lens 506 may serve as a light guide for delivering light from the light modulator to the wearer's eyes. Such a light guide may enable the wearer to perceive 3D holographic images located within the physical environment that the wearer is viewing, while also allowing the wearer to view physical objects in the physical environment, thus creating a mixed reality environment.

[0053] HMD device 500 may also include various sensors and related systems for providing information to controller 504. Such sensors may include, but are not limited to, microphone 512, one or more outward-facing image sensors 508, and inertial measurement unit (IMU) 510.

[0054] Microphone 512 may be any suitable microphone configured to capture sound, and may include a single microphone or an array of microphones placed at suitable locations(s) on HMD device 500. As a non-limiting example, the microphone array may include six microphones located on different portions of HMD device 500.

[0055] One or more outward-facing image sensors 508 may be configured to capture virtual data from the physical environment in which the HMD device 500 is located. For example, the outward-facing sensor 508 may be configured to detect movement within the field of view of the perspective display 502, such as movement performed by a wearer or person or physical object within the field of view. In one example, the outward-facing sensor 508 may detect a remote input device, such as the remote input device 400, used by a user to input gesture movements. The outward-facing sensor may also capture 2D image information and depth information from the physical environment and physical objects within the environment. For example, the outward-facing image sensor 508 may be configured to collect image data of the physical environment, illumination data about ambient light conditions of the physical environment, depth data of surfaces in the physical environment, material property data of surfaces in the physical environment, and the like. In one example, the one or more outward-facing image sensors 508 may include an infrared time-of-flight depth camera. In another example, the one or more outward-facing image sensors 508 may include an infrared structured light depth camera.

[0056] Data from the one or more outward-facing image sensors 508 may be used by the controller 504 to detect movement within the field of view of the see-through display 502. Examples of such movement include gesture-based input or other movement performed by the wearer or a person or physical object within the field of view. In one example, data from the outward-facing image sensor 508 may be used to detect gesture input (e.g., pinching fingers, clenching fists, etc.) performed by the wearer of the HMD to indicate a virtual interaction with a visually presented user interface via the HMD. Data from the outward-facing image sensor may be used by the controller 504 to determine direction / position and orientation data (e.g., from imaging environment features), which enables position / motion tracking of the HMD 500 in the real world environment. Data from the outward-facing image sensor 508 may be used by the controller 504 to construct a still image and / or video image of the surrounding environment from the perspective of the HMD 500.

[0057] Controller 504 may be configured to identify the surfaces of the physical space in any suitable manner. In one example, the surface of the physical space may be identified based on a depth map derived from depth data provided by a depth camera and / or image sensor. Additionally or alternatively, information from an outward-facing image sensor may be communicated to a remote computer responsible for generating and / or updating a model of the physical space. In either case, the relative position and / or orientation of HMD 500 relative to the physical space may be evaluated so that an augmented reality image may be accurately displayed in a desired real-world position with a desired orientation. In one example, controller 504 may be configured to perform simultaneous localization and mapping (SLAM) of the physical space using information provided by a surface sensor alone or in combination with other sensors of HMD 500. Specifically, controller 504 may be configured to generate a 3D model of the physical space, including surface reconstruction information that may be used to identify a surface in the physical space.

[0058] The IMU 510 may be configured to provide position and / or orientation data of the HMD device 500 to the controller 504. In one embodiment, the IMU 510 may be configured as a three-axis or three-degree-of-freedom position sensor system. This example position sensor system may include, for example, three gyroscopes for indicating or measuring changes in the orientation of the HMD device 500 around three orthogonal axes (e.g., x, y, z) in 3D space. The orientation derived from the sensor signals of the IMU may be used to determine the direction of a user who has interacted with the wearer of the HMD device in a conversation.

[0059] In another example, the IMU 510 may be configured as a six-axis or six-degree-of-freedom position sensor system. Such a configuration may include three accelerometers and three gyroscopes to indicate or measure changes in position of the HMD device 500 along three orthogonal axes and changes in device orientation (e.g., roll, pitch, yaw) about the three orthogonal axes. In some embodiments, the position and orientation data from the image sensor 508 and the IMU 510 may be used in combination to determine the position and orientation of the HMD device 500.

[0060] HMD 500 may also support other suitable positioning technologies, such as GPS or other global navigation systems. In addition, although specific examples of position sensor systems are described, it will be understood that any other suitable sensor system may be used. For example, head posture and / or movement data may be determined based on sensor information from any combination of sensors worn on the wearer and / or external to the wearer, including but not limited to any number of gyroscopes, accelerometers, inertial measurement units, GPS devices, barometers, magnetometers, cameras (e.g., visible light cameras, infrared cameras, time-of-flight depth cameras, structured light depth cameras, etc.), communication devices (e.g., WIFI antennas / interfaces), etc.

[0061] The HMD device 500 may also include speakers 524 and 526 configured to output sound to the wearer of the HMD device. The speakers 524 and 526 may be positioned on each side frame portion of the HMD device close to the wearer's ears. For example, the speakers 524 and 526 may play audio content such as music or a soundtrack to the visual content displayed via the perspective display 502. In some cases, the volume of the speakers may be reduced or muted in response to a conversation between the wearer and another person detected.

[0062] Controller 504 may include logic and memory for communicating with the display and various sensors of HMD 500, as described below with respect to Figure 6 discussed in more detail.

[0063] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0064] HMD device 500 is one example of a computing system on which the examples disclosed herein may be implemented. More generally, Figure 6 A block diagram of an example non-limiting embodiment of a computing system 600 that can perform one or more of the above methods and processes is schematically shown. The computing system 600 is shown in a simplified form. In addition to the HMD device described herein, the computing system 600 can take the form of one or more personal computing devices, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices.

[0065] The computing system 600 includes a logic device 602 and a storage device 604. The computing system 600 may optionally include a display subsystem 605, an input subsystem 608, a communication subsystem 610, and / or Figure 6 Other components not shown.

[0066] Logic device 602 includes one or more physical devices configured to execute instructions. For example, a logic device may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, convert the state of one or more components, implement technical effects, or otherwise obtain desired results.

[0067] The logic device may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic device may include one or more hardware or firmware logic devices configured to execute hardware or firmware instructions. The processor of the logic device may be single-core or multi-core, and the instructions executed thereon may be configured for serial, parallel and / or distributed processing. The individual components of the logic device may optionally be distributed between two or more separate devices, which may be remotely located and / or configured for collaborative processing. Various aspects of the logic device may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0068] Storage device 604 includes one or more physical devices configured to hold instructions executable by logic devices to implement the methods and processes described herein. When implementing such methods and processes, the state of storage device 604 may be transformed—for example, to store different data.

[0069] The storage device 604 may include removable and / or built-in devices. The storage device 604 may include optical storage (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor storage (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic storage (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), etc. The storage device 604 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices.

[0070] It is understood that storage device 604 includes one or more physical devices. However, various aspects of the instructions described herein may alternatively be propagated via a communication medium (eg, electromagnetic signals, optical signals, etc.) that is not retained by a physical device for a finite duration.

[0071] Aspects of logic device 602 and storage device 604 may be integrated together into one or more hardware logic components. These hardware logic components may include, for example, field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0072] The term "program" may be used to describe an aspect of a computing system 600 that is implemented to perform a particular function. In some cases, a program may be instantiated via a logic device 602 that executes instructions held by a storage device 604. It will be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" are intended to cover single or grouped executable files, data files, libraries, drivers, scripts, database records, etc.

[0073] It should be understood that a "service" as used herein is an application program that is executable across multiple user sessions. A service can be used for one or more system components, programs, and / or other services. In some implementations, a service can run on one or more server computing devices.

[0074] When including display subsystem 605, display subsystem 605 can be used to present a visual representation of the data maintained by storage device 604. This visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored by the storage device, and thus convert the state of the storage device, the state of display subsystem 605 can also be converted to visually represent the change of the underlying data. Display subsystem 605 may include one or more display devices using substantially any type of technology. Such display devices can be combined in a shared package with logic device 602 and / or storage device 604, or such display devices can be peripheral display devices.

[0075] When input subsystem 608 is included, input subsystem 608 includes or interfaces with one or more user input devices such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the transduction and / or processing of input actions may be handled on-board or off-board. Examples of NUI components may include microphones for language and / or speech recognition; infrared, color, stereoscopic display, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity.

[0076] When the communication subsystem 610 is included, the communication subsystem 610 can be configured to communicatively couple the computing system 600 with one or more other computing devices. The communication subsystem 610 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network. In some embodiments, the communication subsystem can allow the computing system 600 to send messages to other devices and / or receive messages from other devices via a network such as the Internet.

[0077] Another example provides a mixed reality computing device, including: an image sensor; a remote input device; a processor; and storage including stored instructions, which can be executed by the processor to perform the following operations: perform object motion tracking and environment tracking based on output from the image sensor; and adjust the parameters of motion tracking while maintaining environment tracking in response to detecting that the remote input device is in use. Such examples may additionally or alternatively include instructions that can be executed to perform the following operations: adjust the parameters of motion tracking by reducing the frame rate of motion tracking while maintaining the frame rate of environment tracking. Such examples may additionally or alternatively include instructions that can be executed to adjust the parameters of motion tracking by executing instructions for the following operations: deactivate motion tracking based on output from the image sensor; and perform motion tracking based on output from the remote input device. Such examples may additionally or alternatively include a light source and instructions that can be executed to perform the following operations: perform object motion tracking by activating the light source at a first brightness; and perform environment tracking by activating the light source at a second brightness. The remote input device may alternatively or additionally include an inertial measurement unit (IMU) and a user-actuable input device. Such examples may additionally or alternatively include instructions executable to detect that the remote input device is in use based on one or more of: detecting that the remote input device is in the field of view of the image sensor; receiving user input via the user-actuatable input device of the remote input device; and receiving output from the IMU of the remote input device. Such examples may additionally or alternatively include instructions executable to perform the following operations: in response to detecting that the remote input device is in use, perform object motion tracking based on output from the IMU of the remote input device. The remote input device may alternatively or additionally further include one or more image sensors. Any or all of the examples described above may be combined in various implementations in any suitable manner.

[0078] Another example provides a mixed reality computing system, including: a head mounted display device; a remote input device; a plurality of sensors distributed between the head mounted display device and the remote input device; a storage including stored instructions; and a processor configured to execute instructions for the following operations: receiving outputs from the plurality of sensors, obtaining a power function based at least in part on the power consumption and accuracy of the plurality of sensors, selecting a sensor operation mode based at least in part on the power function, and performing object motion tracking according to the selected sensor operation mode. In such examples, the sensor operation mode may additionally or alternatively determine the selection of sensor outputs to be used to perform object motion tracking. In such examples, the sensors may additionally or alternatively include one or more of an accelerometer, a gyroscope, and an image sensor. In such examples, the image sensor may additionally or alternatively be located on the head mounted display device, and wherein the accelerometer and the gyroscope may additionally or alternatively be located on the remote input device. Such examples may additionally or alternatively include instructions executable to perform the following operations: using one or more of the accelerometer and the gyroscope to perform object motion tracking according to a first selected sensor operating mode. Such examples may additionally or alternatively include instructions executable to perform the following operations: correcting motion tracking errors introduced by one or more of the accelerometer and the gyroscope based on output from the image sensor during object motion tracking according to the first selected sensor operating mode. Such examples may additionally or alternatively include instructions executable to perform the following operations: using the image sensor to perform object motion tracking according to a second selected sensor operating mode. Any or all of the examples described above may be combined in various implementations in any suitable manner.

[0079] Another example provides a mixed reality computing system, including: a head mounted display device; a remote input device; a plurality of sensors distributed between the head mounted display device and the remote input device; a processor; and a storage including stored instructions executable by the processor to perform the following operations: activating one or more gesture motion filters in response to determining that the remote input device is in use and performing object motion tracking based on outputs from one or more of the plurality of sensors and the one or more gesture motion filters; and deactivating the one or more gesture motion filters in response to determining that the remote input device is not in use. In such examples, the one or more gesture motion filters may additionally or alternatively include one or more of the following: a first gesture motion filter for filtering out detected object motion occurring within a threshold distance of the image sensor and a second gesture motion filter for filtering out detected object motion that stays within a field of view of the image sensor for less than a threshold amount of time. Such examples may additionally or alternatively include instructions executable to perform the following operations: reducing a frame rate of object motion tracking in response to determining that the remote input device is in use and increasing a frame rate of object motion tracking in response to determining that the remote input device is not in use. In such examples, the remote input device may additionally or alternatively include an inertial measurement unit (IMU) and a user-actuatable input device, and instructions executable to detect that the remote input device is in use based on one or more of: detecting that the remote input device is in the field of view of the image sensor, receiving user input via the user-actuatable input device of the remote input device, and receiving output from the IMU of the remote input device. Such examples may additionally or alternatively include instructions executable to perform the following operations: perform an action based on the object motion tracking. Any or all of the examples described above may be combined in various implementations in any suitable manner.

[0080] It will be understood that the configuration and / or mode described herein are exemplary in nature, and these specific embodiments or examples herein should not be considered as limiting, because many variations are possible. The specific routines or methods described herein can represent one or more of any number of processing strategies. Thus, the various actions illustrated and / or described can be performed in the order shown and / or described, in other orders, in parallel, or omitted. Likewise, the order of the above process can be changed.

[0081] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A mixed reality computing system, comprising: A head mounted display device having an image sensor; Remote input device; a plurality of sensors distributed between the head mounted display device and the remote input device; processor; as well as comprising storage of stored instructions executable by the processor to: determining that the remote input device is in use; In response to determining that the remote input device is in use, reducing a frame rate of object motion tracking, activating one or more gesture motion filters and performing the object motion tracking based on outputs from one or more of the plurality of sensors and the one or more gesture motion filters; as well as In response to determining that the remote input device is not in use, increasing a frame rate of the object motion tracking and deactivating the one or more gesture motion filters; The remote input device comprises an inertial measurement unit (IMU) and a user-actuatable input device, and wherein the instructions are further executable to detect that the remote input device is in use based on one or more of: detecting that the remote input device is in a field of view of the image sensor, receiving user input via the user-actuatable input device of the remote input device, and receiving an output from the IMU of the remote input device.

2. The mixed reality computing system according to claim 1, wherein: The one or more gesture motion filters include one or more first gesture motion filters to filter out detected object motion occurring within a threshold distance of the image sensor and second gesture motion filters to filter out detected object motion remaining within the field of view of the image sensor for less than a threshold amount of time.

3. The mixed reality computing system of claim 1, wherein: The instructions are also executable to perform an action based on the object motion tracking.

4. A method for adjusting a gesture motion filter in a mixed reality computer system, the mixed reality computer system comprising a head mounted display device having an image sensor, a remote input device, and a plurality of sensors distributed between the head mounted display device and the remote input device, the method comprising: Determining that the remote input device is in use includes one or more of the following: detecting that the remote input device is in a field of view of the image sensor; receiving user input via a user-actuatable input device of the remote input device; and receiving an output of an inertial measurement unit IMU from the remote input device; Reduce the frame rate of object motion tracking; activating one or more gesture motion filters; performing the object motion tracking based on outputs from one or more sensors of the plurality of sensors and the one or more gesture motion filters; Determining that the remote input device is not in use; increasing the frame rate of the object motion tracking; as well as The one or more gesture motion filters are disabled.

5. The method according to claim 4, characterized in that Also included is one or more of the following steps: filtering out detected object motion that occurs within a threshold distance of the image sensor, or filtering out detected object motion that stays within the field of view of the image sensor for less than a threshold amount of time.

6. The method according to claim 4, characterized in that Also included is performing an action based on the object motion tracking.

Images