Dynamic noise model for pose estimation

Abstract

A method may receive operational motion sensor data from a head mounted device in use. A method may determine a motion-correlated noise sigma value based on the operational motion sensor data and a noise-to-motion relationship characterizing a relationship between a calibration noise sigma value for a calibration head mounted device and calibration motion sensor data. A method may determine an estimated pose using the operational motion sensor data and the motion-correlated noise sigma value.

Description

DYNAMIC NOISE MODEL FOR POSE ESTIMATIONTECHNICAL FIELD

[0001] This description relates to estimating a pose of a device via sensor fusion.BACKGROUND

[0002] Visual inertial odometry is the process of estimating the pose of a device (e.g., an extended reality device) using the input of camera and inertial measurement unit (IMU) sensors attached to the device. The pose is the 3 degrees of freedom (3dof) of a device. Estimation methods are frequently used to estimate a pose of a device based on sensor data received.SUMMARY

[0003] The disclosure describes a way to improve a pose estimation for a head mounted device by determining a motion-correlated noise sigma value for the device. The motion-correlated noise sigma value represents noise in the operational motion sensor data (such as image data and IMU data) that is due to motion in the head mounted device. The motion-correlated noise sigma value is generated using operational motion sensor data from one or more sensors on the head mounted device and a predetermined noise-to-motion relationship generated based on calibration data generated in a calibration lab sitting.

[0004] The noise-to-motion relationship may be used to produce a noise estimation for the head mounted device that accounts for changes of the physical condition of the head mounted device. In implementations, the dynamic noise may be related to any combination of motion, changing temperature, changing camera settings, etc.

[0005] The calibration data is generated by exposing the head mounted device to varying physical conditions (such as temperature and movement) and / or sensor settings (such as shutter speed) while operating the device. If the relationship between noise in the calibration data and a physical condition is linear, a least squares fit (or some other method) may be performed to the calibration data to generate the noise-to-motion relationship. If a noise model is nonlinear, however, then the noise-to-motion relationship may include a lookup table (or some other relationship).

[0006] The noise-to-motion relationship may then be applied to operational motion sensor data generated while a user operates the head mounted device to estimate a dynamicnoise signal value for the operational motion sensor data. The dynamic noise signal value may then be used to generate an estimated pose of the head mounted device using an estimation method, such as a Kalman filter, to generate the estimated pose.

[0007] In some aspects, the techniques described herein relate to a method, including: receiving operational motion sensor data from a head mounted device in use; determining a motion-correlated noise sigma value based on the operational motion sensor data and a noise- to-motion relationship characterizing a relationship between a calibration noise sigma value for a calibration head mounted device and calibration motion sensor data; and determine an estimated pose using the operational motion sensor data and the motion-correlated noise sigma value. The method can additionally comprise the following steps: determining a calibration noise sigma value based on calibration sensor data from a calibration head mounted device and calibration motion sensor data generated during a calibration activity; and generating a noise-to-motion relationship between the calibration noise sigma value and the calibration motion sensor data characterizing a relationship between a calibration noise sigma value for a calibration head mounted device and calibration motion sensor data.

[0008] In some aspects, the techniques described herein relate to a device including: a processor; and a memory' configured with instructions to: receive operational motion sensor data from a head mounted device in use; determine a motion-correlated noise sigma value based on the operational motion sensor data and a noise-to-motion relationship characterizing a relationship between a calibration noise sigma value for a calibration head mounted device and calibration motion sensor data; and determine an estimated pose using the operational motion sensor data and the motion-correlated noise sigma value.

[0009] In some aspects, the techniques described herein relate to a system including: a data receiving module configured to receive operational motion sensor data from a head mounted device in use; a dynamic noise module configured to determine a motion-correlated noise sigma value based on the operational motion sensor data and a noise-to-motion relationship characterizing a relationship between a calibration noise sigma value for a calibration head mounted device and calibration motion sensor data; and a pose estimation module configured to determine an estimated pose using the operational motion sensor data and the motion-correlated noise sigma value. The system can additionally comprise: a noise calculation module configured to determine a calibration noise sigma value based on calibration sensor data from a calibration head mounted device and calibration motion sensor data generated during a calibration activity; anda noise model generation module configured to generate a noise-to-motion relationship between the calibration noise sigma value and the calibration motion sensor data.

[0010] In some aspects, the techniques described herein relate to a method including: determining a calibration noise sigma value based on calibration sensor data from a calibration head mounted device and calibration motion sensor data generated during a calibration activity; and generating a noise-to-motion relationship between the calibration noise sigma value and the calibration motion sensor data.

[0011] In some aspects, the techniques described herein relate to a system including: a noise calculation module configured to determine a calibration noise sigma value based on calibration sensor data from a calibration head mounted device and calibration motion sensor data generated during a calibration activity; and a noise model generation module configured to generate a noise-to-motion relationship between the calibration noise sigma value and the calibration motion sensor data.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG. 1 A illustrates a frontal view of a head mounted device, in accordance with implementations of the disclosure.

[0013] FIG. IB is a rear view of head mounted device, in accordance with implementations of the disclosure.

[0014] FIG. 2A depicts a block diagram of a device, in accordance with implementations of the disclosure.

[0015] FIG. 2B depicts a block diagram of a computing device, in accordance with implementations of the disclosure.

[0016] FIG. 3 depicts a block diagram, in accordance with implementations of the disclosure.

[0017] FIG. 4A depicts a calibration lab set up, in accordance with implementations of the disclosure.

[0018] FIG. 4B depicts an image target, in accordance with implementations of the disclosure.

[0019] FIG. 4C depicts a calibration lab set up, in accordance with implementations of the disclosure.

[0020] FIG. 4D depicts an image reproj ection diagram, in accordance with implementations.

[0021] FIG. 4E depicts an image reprojection error diagram, in accordance withimplementations.

[0022] FIG. 4F depicts an image reprojection error diagram, in accordance with implementations.

[0023] FIG. 5A depicts a system 500, in accordance with implementations of the disclosure.

[0024] FIG. 5B depicts a system 550, in accordance with implementations of the disclosure.

[0025] FIG. 6A depicts a method 600, in accordance with implementations of the disclosure.

[0026] FIG. 6B depicts a method 650, in accordance with implementations of the disclosure.DETAILED DESCRIPTION

[0027] Extended reality (XR) devices, such as augmented reality, mixed reality, and virtual reality devices, include head-mounted displays that can place virtual objects in front of a field of view of a user. Often those objects are placed in the field of view of the user with respect to the 3D space and / or objects in the 3D space around the user. In implementations, in an XR application a video game character may be positioned sitting in a real chair in a room. In order to make object placements seem realistic and / or avoid jitter that would frustrate the user experience, XR devices first determine a pose (an orientation and velocity) of the head mounted device before determining where to place an object. As the use of some head mounted devices expands into new applications, such as real time navigation, there is a need for increasingly accurate pose estimation that avoids jitter and results in an enjoyable user experience.

[0028] While the present disclosure describes a method to determine a dynamic noise of a head mounted device to improve a pose estimation of a device, other types of devices are contemplated. The methods described herein may be used to estimate a pose of any device or object, such as a pose of a car or a robot.

[0029] Current methods use an estimation of static noise as an input when generating a pose estimate for a device. Estimation algorithms generate a weighted average of estimated poses based on operational motion sensor data, the operational motion sensor data being weighted based on the noise input. Therefore, using a static noise as input, which under current methods is a constant determined in a lab setting, does not allow the weighted algorithm to vary the weights of the operational motion sensor data when estimating poses.

[0030] To generate the static noise estimate used by current methods, operational motion sensor data (such as IMU and image data) are generated as the head mounted device rests on a lab bench at room temperature. While the static noise determined in the lab may represent lab conditions well, in operation head mounted device is subjected to a much greater variety of physical conditions, such as varied temperatures, senor orientations and movements. In the implementation of a camera, illumination levels and shutter speed are also important variables in the sensor data.

[0031] At least one technical problem is that current methods do not adequately account for noise in operational motion sensor data due to changing physical conditions around the head mounted device, such as: movement of the head mounted device, a changing temperature environment around the head mounted device, and changing illumination and exposure conditions in a camera of the head mounted device. Because the dynamic noise is not adequately accounted for, the pose estimation may be somewhat jittery, and therefore a virtual object placed in the field of view of a head-mounted device may appear to jump or shake.

[0032] At least one technical solution described herein is to improve a pose estimation for a head mounted device by determining a motion-correlated noise sigma value for the device representing noise in the operational motion sensor data due to motion or other phy sical changes in the head mounted device. The motion-correlated noise sigma value is generated using operational motion sensor data from one or more sensors on the head mounted device and a predetermined noise-to-motion relationship generated based on calibration data generated in a calibration lab sitting.

[0033] FIG. 1 A illustrates a frontal view of head mounted device 100 in the form of smart glasses, or augmented reality glasses, or mixed reality’ glasses, including display capability, eye / gaze tracking capability, and computing / processing capability. FIG. IB is a rear view of head mounted device 100 shown in FIG. 1A. Head mounted device 100 includes a headset frame 110. The headset frame 110 includes a front frame portion 120, and a pair of side portions 130 rotatably coupled to the front frame portion 120 by respective hinge portions 140. The front frame portion 120 includes rim portions 123 surrounding respective optical portions in the form of lenses 127, with a bridge portion 129 connecting the rim portions 123. The side portions 130 may be coupled pivotably or rotatably coupled, to the front frame portion 120 at peripheral portions of the respective rim portions 123.

[0034] Head mounted device 100 may include a display device 104 that can output visual content, such as, at an output coupler 105, so that the visual content is visible to theuser. In the implementation shown, the display device 104 is included in one of the two arm portions 130. simply for purposes of discussion and illustration. Display devices 104 may be included in each of the two arm portions 130 to include for binocular output of content. In some implementations, the display device 104 may be a see-through near eye display. In some implementations, the display device 104 may be configured to project light from a display source onto a portion of teleprompter glass functioning as a beam splitter seated at an angle (e.g., 30-45 degrees). The beam splitter may allow for reflection and transmission values that allow the light from the display source to be partially reflected while the remaining light is transmitted through. Such an optic design may allow a user to see both phy sical items in the world, such as through the lenses 127, next to content (such as digital images, user interface elements, virtual content, and the like) output by the display device 104. In some implementations, waveguide optics may be used to depict content on the display device 104.

[0035] In implementations, the head mounted device 100 may include any combination of an illumination device 108 and an outward facing camera 116. In implementations, the head mounted device 100 may include a sensing system 111. The sensing system 111 may include at least one IMU, a motion sensor operating in one to three dimensions which may include any combination of: accelerometer, gy roscope, and / or magnetometer.

[0036] Each sensor in head mounted device 100 features static noise when the sensor is at rest in stable conditions. In implementations, the static noise may include angle random walk noise for gy roscope data and velocity random w alk noise for accelerometer data. In implementations, the static noise may also include bias sigma such as rate random walk for gyroscope data and acceleration random walk for accelerometer data. Each sensor also features additional dynamic noise when the sensor is in motion or other physical conditions are changing.

[0037] IMU dynamic noise models, including for both intrinsic error and noise, determine a motion-correlated noise sigma value based on parameter changes of the static noise model and the dynamic changes of the gyroscope and accelerometer intrinsic parameters, such as scale factors, biases and misalignment angles (also known as cross axis sensitivity). The IMU dynamic model parameters are functions of inputs such as: temperature, rotation rate and acceleration.

[0038] The image data static noise model includes pixel noise sigma values. Image motion-correlated noise sigma values may be determined using motion dynamics, asmeasured by the IMU, in addition to lighting conditions, including exposure time data and light intensity data.

[0039] FIG. 2A depicts a block diagram of device 200 in accordance with an implementation. Device 200 may be used to determine a motion-correlated noise sigma value of head mounted device 100. In implementations, device 200 may be an instance of head mounted device 100. In further implementations, however, device 200 may be a computing device in communication with head mounted device 100. Device 200 may be a handheld device, a mobile phone, a laptop computer, a desktop computer, a wrist worn computing device, a server, or any other t pe of computing device in communication with head mounted device 100.

[0040] In the implementation where device 200 is an instance of head mounted device 100, the block view of device 200 omits some components depicted in FIGs. 1 A and IB for the sake of brevity. In implementations where device 200 is an instance of head mounted device 100, device 200 may further include any combination of components depicted in Figures 1A and IB.

[0041] Device 200 includes a processor 202, a memory 204, a communications interface 206, and a dynamic noise module 218. In implementations, device 200 may further include any combination of: an IMU 208, a camera 210, an IMU data receiving module 212, an image data receiving module 214, a temperature data receiving module 216. a dynamic noise module 218, and a pose estimation module 220.

[0042] In implementations, processor 202 may include multiple processors, and memory' 204 may include multiple memories. Processor 202 may be in communication with any cameras, sensors, and other modules and electronics of head mounted device 100. Processor 202 is configured by instructions (e.g., software, application, modules, etc.) to determine a motion-correlated noise sigma value. The instructions may include non-transitory computer readable instructions stored in, and recalled from, memory 204. In implementations, the instructions may be communicated to processor 202 from a computing device or from a network via a communications interface 206. Processor 202 may be configured with instructions to receive data from any combination of IMU 208 or camera 210.

[0043] Communications interface 206 of device 200 may be operable to facilitate communication between device 200 and head mounted device 100 or between device 200 and calibration computing device 250. In implementations, communications interface 206 may utilize Bluetooth, Wi-Fi, Zigbee, or any other wireless or wired communication methods.

[0044] In implementations where device 200 is head mounted device 100, device 200 may include IMU 208. IMU 208 may comprise a motion sensor operating in one to three dimensions which may include any combination of: accelerometer, gyroscope, and / or magnetometer. In implementations, IMU 208 may be implemented as a three-axis motion sensor such as a three-axis accelerometer or a three-axis gyroscope, where the motion signals captured by the motion sensor describe three translation movements (i. e.. x-direction, y- direction, and z-direction) along axes of a world coordinate system. In implementations, IMU 208 may be implemented as a six-axis motion sensor which can describe three translation movements (i.e., x-direction, y-direction, and z-direction) along axes of a world coordinate system and three rotation movements (i.e., pitch, yaw. roll) about the axes of the world coordinate system. In implementations, IMU 208 may produce a vibrational spectrum.

[0045] In implementations, IMU 208 may be a first IMU and there may be further IMUs integrated into head mounted device 100. In implementations, IMU 208 may be positioned anywhere within or on headset frame 110. In implementations, IMU 208 mayproduce IMU temperature data to processor 202 from a temperature sensor internal to IMU 208.

[0046] Device 200 may include camera 210. Camera 210 may send image data to processor 202. In implementations where device 200 is head mounted device 100, camera 210 may comprise outward facing camera 116.

[0047] In implementations, camera 210 may include an auto ISO feature that may adjust a camera shutter speed based on changing light conditions to obtain an optimal exposure of image data. As part of the auto ISO feature, camera 210 may include an internal light sensor. In implementations, camera 210 may measure light intensity from the internal light sensor to processor 202. In implementations, the light intensity data may be a luminous intensity. In implementations, camera 210 may provide shutter speed data to processor 202. In implementations, it may be possible for processor 202 to command a shutter speed by sending a command to camera 210.

[0048] In implementations, camera 210 may provide camera temperature data to processor 202 from a temperature sensor internal to camera 210.

[0049] Device 200 may include IMU data receiving module 212. In implementations where device 200 is head mounted device 100, IMU data receiving module 212 may receive operational motion sensor data from IMU 208. Operational motion sensor data is any data received from a sensor within head mounted device 100 while a user is operating the device in a non-calibration lab environment. In implementations where device 200 is an additionalcomputing device or a server, however, operational motion sensor data from any combination of IMUs internal to head mounted device 100 may be received over communications interface 206.

[0050] The IMU data may take many possible formats. In one implementation, the IMU data may include gyroscope angular rotation data, gy roscope rotational rate data, accelerometer linear acceleration data, and / or accelerometer vibration spectrum data.

[0051] Device 200 may include image data receiving module 214. In implementations where device 200 is head mounted device 100, image data receiving module 214 may receive operational motion sensor data from camera 210. In implementations where device 200 is an additional computing device or a server, however, operational motion sensor data from any camera internal to head mounted device 100 may be received over communications interface 206.

[0052] In implementations, image data receiving module 214 may receive any combination of image data, light intensity data, and / or camera shutter exposure time.

[0053] Device 200 may include temperature data receiving module 216. In implementations, module 216 may receive operational motion sensor data from any sensor internal to head mounted device 100. In implementations, a temperature sensor may be integrated into IMU 208 or camera 210, thereby providing an IMU temperature or a camera temperature, respectively. In implementations, a temperature sensor may be integrated into any other component of head mounted device 100. In implementations, a temperature sensor may measure the temperature of any sub-component of head mounted device 100 or the temperature of head mounted device 100 itself.

[0054] Device 200 may include dynamic noise module 218, operable to determine a motion-correlated noise sigma value based on operational motion sensor data and a noise-to- motion relationship.

[0055] In implementations, the motion-correlated noise sigma value is a noise sigma value that accounts for movement of head mounted device 100. In current methods, only- static noise may be determined for head mounted device 100. Static noise does not account for translational, rotational, or vibrational motion of a device, each of which may generate their own unique noise signatures in the operational motion sensor data. In implementations, the motion-correlated noise sigma value may further account for other physical conditions of head mounted device 100, such as any combination of temperature, image exposure time, illumination, etc.

[0056] The motion-correlated noise sigma value determined for head mounted device100, which may be in use by a user, is based on a noise-to-motion relationship for the calibration head mounted device. The noise-to-motion relationship is a relationship that may be used to determine dynamic noise signal value based on data characterizing some kind of motion of head mounted device 100. In implementations, the data used to generate the noise- to-motion relationship for the calibration head mounted device may have been generated by operating the calibration head mounted device in a calibration or lab environment. In implementations, the calibration head mounted device may have been selected to represent a class of head mounted devices that have similar noise-to-motion relationships than the calibration head mounted device.

[0057] The noise-to-motion relationship represents the correlation between motion of the head mounted device sensor and noise in that sensor data. In implementations, image data may have more blurring when head mounted device is in motion versus when it is stable. The blurring in the image data may also correlate to the degree of motion of the head mounted device.

[0058] In implementations, the noise-to-motion relationship may characterize a relationship between a calibration noise sigma value for a calibration head mounted device and calibration motion sensor data. Calibration motion sensor data may include any data that provides information about the motion of the head mounted device and / or head mounted device sensors. In implementations, calibration motion sensor data may include data from sensors external to a head mounted device that independently measure the position of the head mounted device, data from instruments controlled and / or programmed to move the head mounted device, and / or data from sensors internal to the head mounted device that capture the motion of the head mounted device.

[0059] A calibration noise sigma value is a noise sigma value measured in a calibration environment. A noise sigma value for a sensor may change with motion or other conditions. In implementations, noise sigma value may vary based on rotational or translational motion, acceleration, and / or orientation. In implementations, the noise sigma value may further vary based on changing temperature, light intensity, image exposure time, or any other environmental condition. In implementations, noise sigma value may vary based on any combination of the conditions described.

[0060] FIG. 3 depicts block diagram 300, in accordance with an implementation. FIG. 3 depicts the inputs and outputs of dynamic noise module 218. Dynamic noise module 218 determines the motion-correlated noise sigma value using operational motion sensor data as input. In implementations, operational motion sensor data may comprise any combination of:IMU data, including gyroscope data 302 and / or accelerometer data 304, image data 306, temperature data (such as camera temperature 312 or IMU temperature 314). light intensity data 316, or camera shutter exposure time 318, as described with respect to IMU data receiving module 212, image data receiving module 214, and temperature data receiving module 216 above.

[0061] The creation of the noise-to-motion relationship used by dynamic noise module 218 to determine the motion-correlated noise sigma value is further described below with respect to modules 258 to 282 of computing device 250.

[0062] In implementations, determining a motion-correlated noise sigma value may comprise determining one or more motion-correlated noise sigma values for one or more measurements taken with one or more sensors. Returning to FIG. 3, any combination of gyroscope data 302, accelerometer data 304 and image data 306 may determine a dynamic sigma noise value 310 for any combination of sensor measurements. In implementations, one or more noise-to-motion relationship models including one or more different types of sensor input may be used to determine one or more instances of motion-correlated noise sigma value 310.

[0063] Dynamic noise module 218 may include a gyroscopic dynamic noise model operable to determine a motion-correlated noise sigma value for an angular orientation or a rotational rate. In implementations, the gyro motion-correlated noise sigma value may include two or three of the following terms: gyro_noise = gyro_static_noise + fl (temp, omega, accel) + vl (vib_spectrum) (Equation 1) where temp is IMU temperature 314, omega is an IMU gyroscope rotational position, accel is an IMU accelerometer measurement, and the vib spectrum is an IMU accelerometer vibrational spectrum. The term gyro_static_noise, gyroscopic static noise, may be a constant, as described above. In implementations, generating and applying the fl or the vl relationships may inherently include the gryo_static_noise term, or the gyro static noise term may be uncorrelated and need to be added separately. In implementations, the gyroscopic motion-correlated noise sigma value for an angular orientation or rotational rate may be determined for each of three dimensions.

[0064] The term fl (temp, omega, accel) may comprise a motion-correlated noise sigma value for a rotational rate based on temperature, rotational rate, and acceleration. The term vl (vib spectrum) may comprise a gyroscope angular orientation dynamic noise model may be used to determine a motion-correlated noise sigma value for an angular orientationbased on vibrational spectrum received from the IMU. The generation of the fl and vl terms are described with respect to noise model generation module 282 below.

[0065] Dynamic noise module 218 may include an accelerometer dynamic noise model operable to determine a motion-correlated noise sigma value for a linear acceleration or a vibration spectrum. In implementations, the accelerometer motion-correlated noise sigma value may include two or three of the following terms: accel_noise = accel_static_noise + f2(temp, rate, accel) + v2 (vib_spectrum) (Equation 2) where temp is an IMU temperature, rate is an IMU gyroscope rotational speed, accel is an IMU accelerometer measurement, and the vib_spectrum is the IMU accelerometer vibrational spectrum. The term accel static noise, accelerometer static noise, may be a constant, as described above. In implementations, generating and applying the f2 or the v2 relationships may inherently include the accel_static_noise term, or the accel_static_noise term may be uncorrelated and need to be added separately. In implementations, the accelerometer motion-correlated noise sigma values for a linear acceleration may be determined for each of three dimensions.

[0066] Dynamic noise module 218 may include an image dynamic noise model operable to determine a motion-correlated noise sigma value for an image. In implementations, the image dynamic noise sigma may be determined via either Equation 3 or 4. Equation 3 may in include two or three of the terms listed: image_sigma_l= image_static_noise + f3(lum_intensity, exp_time, rot_speed) + v3 (vib spectrum)(Equation 3)(Equation 4) where lumjntensity is a camera light intensity data, exp_time is a camera shutter exposure time, rot_speed is rotational speed, and est_speed is an estimated speed from a pose estimation module (described below). The term image_static_noise, image static noise, may be a constant, as described above. In implementations, generating and applying the f3 or the v3 relationships may inherently include the image_static_noise term, or the image_static_noise term may be uncorrelated and need to be added separately. Inimplementations, the image motion-correlated noise sigma values may be determined for each of three dimensions.

[0067] One or more motion-correlated noise sigma values determined in dynamic noise module 218 may be used to generate a more accurate pose estimation via pose estimation module 220. Pose estimation module 220 may generate an estimated pose 308 using the operational motion sensor data and the motion-correlated noise sigma value. FIG. 3 depicts the inputs and outputs of pose estimation module 220.

[0068] Pose estimation module 220 receives data about the current pose of head mounted device 100 in the form of current operational motion sensor data (gyroscope data 302, accelerometer data 304. and / or image data 306). In implementations, gyroscope data 302 and / or accelerometer data 304 may be received from IMU data receiving module 212, as described above. In implementations, image data 306 may be received from image data receiving module 214, as described above. Pose estimation module 220 uses the current operational motion sensor data to generate an estimated pose 308 for the next sample period. Estimated pose 308 is preferred over determining the current pose of a device for XR applications because estimated pose 308 is an estimate of pose without noise, and therefore objects placed in the user’s field-of-view may appear much less jittery.

[0069] In implementations, pose estimation module 220 may be a Kalman filter, such as a multistate constrain Kalman filter. In implementations, pose estimation module 220 maybe a least squares bundle adjustment, or any other optimal estimation algorithm. In implementations such as a Kalman filter, the average may be weighted based on the estimated noise input so that when the operational motion sensor data noise is higher, the nosier data will be weighted less than data with lower levels of noise. This may provide for a more robust estimated pose 308.

[0070] Pose estimation module 220 receives a motion-correlated noise sigma value 310. Motion-correlated noise sigma value 310 is determined by7dynamic noise module 218. In implementations, dynamic noise module 218 may receive current operational motion sensor data, such as any combination of gyroscope data 302, accelerometer data 304. and or image data 306. In implementations, dynamic noise module 218 may also receive temperature data, such as camera temperature 312 and / or IMU temperature 314. In implementations, dynamic noise module 218 may receive information about image data 306 exposure, such as any combination of light intensity data 316 and / or camera shutter exposure time 318.

[0071] In implementations, estimated pose 308 may be estimated based on a previous speed 320 determined from a previous estimated pose, as will be further described below.

[0072] FIG. 2B depicts a block diagram of computing device 250, in accordance with an implementation. Computing device 250 may be used to determine a noise-to-physical condition relationship of a calibration head mounted device.

[0073] In implementations, computing device 250 may be a desktop computer, a server, or any other computing device. Computing device 250 may be used to do any combination of commanding and controlling lab equipment, receiving telemetry, signals and data from lab equipment, sending commands to and receiving data from the calibration head mounted device, and / or performing data analysis on telemetry, signals and data, calculating the calibration noise sigma value in one or more sensors of the calibration head mounted device. Using the calibration noise sigma value, computing device 250 may generate a noise- to-physical condition relationship, as will be further described below.

[0074] Computing device 250 includes a processor 252, a memory 254, a communication interface 256. In embodiments, processor 252 may have any combination of features described with respect to processor 202, memory 254 may have any combination of features described with respect to memory 204, and communication interface 256 may have any combination of features described with respect to communications interface 206 above.

[0075] In implementations, computing device 250 may further include any combination of a displacement mechanism command module 258. a vibration table command module 260, a temperature control module 262, a camera operation module 264, a light command module 266, an IMU data receiving module 268, an image data receiving module 270, a temperature data receiving module 272, a displacement mechanism position data receiving module 274, a vibration table data receiving module 276, a calibration data storage module 278, a noise calculation module 280, and a noise model generation module 282.

[0076] In implementations, computing device 250 may include a displacement mechanism command module 258. Displacement mechanism command module 258 may provide commands to a device that is operable to move the calibration head mounted device translationally, rotationally, or a combination thereof. In implementations, displacement mechanism command module 258 may command a robot arm, a linear rail, a rate table, or any other ty pe of device operable to displace the calibration head mounted device.

[0077] FIG. 4A depicts a calibration lab set up 400 in accordance with implementations. Lab set up 400 includes a platform 404 upon which a robot arm 402 ismounted. Calibration head mounted device 403 is coupled to an end of robot arm 402. An image target frame 406 may be structured around calibration head mounted device 403 and an upper portion of robot arm 402. An image target, which may be coupled to image target frame 406, is not depicted in FIG. 4A for the sake of simplicity.

[0078] In implementations, calibration head mounted device 403 may be a representative implementation of a class of head mounted device 100 devices or an individual instance of calibration head mounted device 403. In implementations, calibration head mounted device 403 may be calibration head mounted device 403.

[0079] Calibration lab set up 400 further includes computing device 250 (not depicted in FIG. 4A). Computing device 250 may be configured to sequence a calibration of calibration head mounted device 403 by sending commands to robot arm 402 and / or calibration head mounted device 403. Computing device 250 may further facilitate the determination of calibration noise sigma values and a noise-to-motion relationship by receiving telemetry and / or data from calibration head mounted device 403 and robot arm 402.

[0080] FIG. 4B depicts the image target 408. Image target 408 includes an image that may be used to help determine a position of calibration head mounted device 403 when imaged by camera 210. In implementations, image target 408 may cover five sides of a box configured to mount to image target frame 406. In implementations, image target 408 may be configured to surround much of calibration head mounted device 403 and a top portion of robot arm 402. In implementations, image target 408 may include a series of pattern keypoints in var ing sizes. In implementations, image target 408 may feature an image that is nonrepeating throughout the surface area of image target 408 so that an orientation of calibration head mounted device 403 may be determined from image data based on the unique content of the image data.

[0081] Computing device 250 may include a vibration table command module 260, which may command a vibration table in a calibration lab. The vibration table may be used to apply mechanical vibrations to calibration head mounted device 403. In implementations, the vibration table may provide very little or de minimums displacement to calibration head mounted device 403. In implementations, the vibrations may be periodic. In implementations, the vibrations may vary in frequency and / or amplitude. In implementations, the vibration table may provide telemetry and / or data including information about the frequencies and / or amplitudes that the vibration table oscillates at.

[0082] FIG. 4C depicts a test setup 450, according to implementations. Test setup 450 includes calibration head mounted device 403 positioned on top of a vibration table 452.Computing device 250 may sequence a calibration of calibration head mounted device 403 by sending commands to each of calibration head mounted device 403 and vibration table 452. Computing device 250 may further receive telemetry and / or data from head mounted device 100 and vibration table 452.

[0083] In implementations, processor 202 may include temperature control module 262. Temperature control module 262 may allow computing device 250 to command a temperature controller to heat and / or cool the lab environment during a calibration activity. The temperature controller may provide telemetry and / or data about the temperature setpoint or lab temperature back to processor 202.

[0084] In implementations, processor 202 may further include camera operation module 264. Camera operation module 264 may allow for the command of camera 210. In implementations, computing device 250 may be able to command camera 210 (via calibration head mounted device 403) to perform any combination of generating image data, changing a shutter speed, and using an auto ISO setting.

[0085] In implementations, processor 202 may further include light command module 266. Light command module 266 may allow for computing device 250 to control a light source. In implementations, the light source may illuminate image target 408 within a field of view of camera 210. By varying the levels of illumination upon image target 408, it may be possible to generate image data with different combinations of light and exposure time.

[0086] In implementations, processor 202 may further include IMU data receiving module 268. IMU data receiving module 268 may be configured to receive IMU data from calibration head mounted device 403. In implementations, the IMU data may include gy roscope angular rotation data, gy roscope rotational rate data, accelerometer linear acceleration data, and / or accelerometer vibration spectrum data. In implementations. IMU data receiving module 268 may receive other types of data from IMU 208.

[0087] In implementations, processor 202 may further include image data receiving module 270. Image data receiving module 270 may receive image data from camera 210. Image data receiving module 270 may also receive information from camera 210 relating to exposure times and / or luminous intensity.

[0088] Computing device 250 may further include temperature data receiving module 272. Temperature data receiving module 272, may receive temperature data from one or more sensors internal to or external to calibration head mounted device 403.

[0089] Computing device 250 may further include displacement mechanism position data receiving module 274. In implementations, displacement mechanism position datareceiving module 274 may receive data from a robot arm indicating the robot arm’s position determined via encoder position data or stepper motor position data. The position data may be used to independently determine a pose of calibration head mounted device 403, which may be used to help determine the level of dynamic noise in sensor data received from calibration head mounted device 403.

[0090] In implementations, computing device 250 may further include vibration table data receiving module 276. Vibration table data receiving module 276 may receive data from vibration table 452 indicating a frequency and / or amplitude of vibration. In implementations, the frequency data may include commanded or measured frequencies and / or amplitudes. The frequencies and / or amplitudes may be used to help determine a level of dynamic noise in sensor data received from head mounted device 100.

[0091] In implementations, any combination of modules 258-266 may be used to gather calibration data during a calibration exercise. Such as:• Data for a gyroscope dynamic noise calibration may be collected by rotating robot arm 402 at different rates and determining the calibration noise sigma value of the gyroscope data at the different rotational speeds. In implementations, the axis of rotation may be parallel to a horizontal plane.• Data for a gyroscope dynamic noise calibration may be collected by rotating robot arm 402 at constant speed and adjusting the angles of the axis of rotation with respect to the gravity direction to calibrate the gyroscopic noise sensitivity to the gravity direction.• Data for accelerometer dynamic noise calibration may be collected by stepping robot arm 402 through various orientations with respect to the gravity direction to calibrate the accelerometer noise with regard to gravity.• Data for a static temperature noise calibration may be collected by stepping calibration head mounted device 403 through a temperature profile, such as by raising the temperature in the lab around calibration head mounted device 403 and allowing the device to get to steady state. In implementations, the temperature steps may differ by 5 C and span from 10 C to 60 C. At each temperature step, any combination of image data, accelerometer data and gy roscope data may be collected to calibrate static temperature noise.• Data for a dynamic temperature noise calibration may be collected by increasing the temperature of calibration head mounted device 403 at a steady rate, such as by 0.5, 1, 1.5 or 2 C per minute, while collecting any combination of image data, accelerometer data and gyroscope data may be collected to calibrate static temperature noise.• Data for a vibrational noise calibration may be collected by commanding vibration table 452 to generate random vibrations at different band widths (spectrum), such as 1Hz to 10 Hz, 10 Hz to 100 Hz and 100 Hz to 200 Hz and 200 Hz to 1000 Hz, and so forth. Data may be collected from vibration table 452, along with any combination of image data, accelerometer data and gyroscope data to calibrate IMU 208 and / or camera 210 for vibrational noise.• Data for an image motion blur calibration may be collected by commanding the light to change the lighting conditions around calibration head mounted device 403 while collecting data from calibration head mounted device 403 sensors. Such as, the lighting may be set to a specific luminosity level while the exposure time of camera 210 is changed and / or robot arm 402 is operated to move calibration head mounted device 403. Data for an image motion blur calibration may be further collected by setting camera 210 in an ISO auto exposure mode while changing the lighting conditions around calibration head mounted device 403. Data collected may include any combination of image data, gyroscope data, accelerometer data, light intensity data, camera shutter exposure time data, displacement mechanism position data, and / or vibration table data.

[0092] In implementations, computing device 250 may further include calibration data storage module 278. Calibration data storage module 278 may store one or more types of data generated during a calibration lab activity. In implementations, calibration data storage module 278 may include any combination of data relating to gyroscope measurements (such as angular rotation, rotational rates data, etc.), accelerometer measurements (such as linear acceleration rates, vibration spectrum data, etc.), image data, light sensor data (such as luminous intensity data), camera exposure time data, temperature data (such as camera temperature data, IMU temperature data, etc.), displacement mechanism position data, and / or vibration table data. In implementations, the data may include timestamps.

[0093] In implementations, computing device 250 may further include noise calculation module 280. Noise calculation module 280 may determine the noise, such as the standard deviation or sigma levels, of sensor data received from calibration head mounted device 403 based on data stored in calibration data storage module 278. Noise calculation module 280 may receive IMU (gyroscope and / or accelerometer) data and / or image data and determine the level of noise in that data. The noise may include any combination of static noise, dynamic noise (noise to movement of a sensor), and / or vibrational noise (noise due to vibrations of a sensor).

[0094] According to implementations, noise calculation module 280 may determine an independent pose for calibration head mounted device 403 using instruments external to calibration head mounted device 403 to compare to the pose determined using sensors internal to calibration head mounted device 403 for the purposes of determining one or more the motion-correlated noise sigma values in the data. In implementations, a displacement mechanism determined pose may be determined for the displacement mechanism that head mounted device 403 is mounted to in the calibration lab.

[0095] In implementations, robot arm 402 may include one or more sections connected to one or more articulating joints. Robot arm 402 may include one or more encoders to measure the rotation of one or more articulating joints. Using information about the dimensions of various sections of robot arm 402, the encoder-determined rotational positions of the one or more joints of robot arm 402, and the orientation of calibration head mounted device 403 coupled to robot arm 402, it may be possible to determine an independent displacement mechanism pose for calibration head mounted device 403 which can be compared to a pose determined using IMU 208 and / or camera 210 data to calculate noise in the IMU 208 and / or camera 210 data or pose. The rotational and / or translational positions determined via the displacement mechanism may be differentiated to determine rotational and / or translational rates or accelerations. In implementations, the displacement mechanism determined pose may be used to calculate the noise, including dynamic noise in data from IMU 208 and / or camera 210.

[0096] In implementations a 6DoF offset may first be determined between the displacement mechanism determined pose and a pose determined using data from IMU 208 before performing the noise calculation. Robot arm 402 may rotate calibration head mounted device 403 at constant speed along its 3 axes to generate a set of data to determine the 6DoF offset. The 6DoF offset determined may be used in subsequent dynamic noise calculations.

[0097] The lab generated position, rate, and acceleration data may subsequently be used to calculate gyroscope, accelerometer, and image motion-correlated noise sigma values. The independent position data may be used to calculate the noise in IMU 208 and / or camera 210 data.

[0098] According to implementations, noise calculation module 280 may determine an image data determined pose using image data from camera 210 of image target 408. Dynamic image data pose information may be determined during any of the previously described dynamic calibration activities (with calibration head mounted device 403 moving, lighting changing, temperature changing) and used to calculate the noise in an IMU datadetermined pose. The rotational and / or translational positions determined using image data of image target 408 may be differentiated to determine rotational and / or translational rates or accelerations. In implementations, the accuracy of the image data determined poses may be improved by providing adequate lighting conditions to support shorter image exposure times, thereby minimizing motion blur. In implementations, an image exposure time of 1 ms or less may minimize motion blur.

[0099] In implementations, noise may be calculated in the image data generated pose by comparing the image generated pose to the displacement mechanism generated pose described above. In implementations, the noise calculation may be an image reprojection error calculation. Camera 210 may be used to generate images of image target 408.

[0100] FIG. 4D depicts image reprojection diagram 470, according to implementations. In diagram 470, planar point / within image target 408 is being observed by camera 210 at a first camera position having a pose C 1 and a second camera position having a pose C2. N represents a normal vector to image target 408. Z1 and Z2 represent the depth of planar point. N represents the normal of the plane of image target 408, and distances dl, d2 represent the respective distances between pose C 1 and image target 408 and pose C2 and image target 408, respectively.

[0101] By determining the connection between a normalized image coordinate of planar point f observed by a moving camera at pose Cl and pose C2, it is possible to determine the camera generated pose of the camera at each of Cl and C2. Equation 5 represents the normalized image coordinates below:(Equation 5)where represents a planar point location with respect to an observed camera pose C, and zi represents the depth of the planar point.

[0102] Because planar point lies on a plane within image target 408, it is possible to establish constraints between Cip^ and plane parameters N, dl, and d2. Distances dl and d2 may be independently determined using displacement mechanism determined poses and knowledge about the relative positions of the displacement mechanism and image target 408. Therefore, using the displacement mechanism determined poses, it is possible to measure the noise in the image data determined poses.

[0103] FIG. 4E depicts a histogram projection error graph 480, and FIG. 4F depicts reproduction error diagram 490 for a set of calibration image data, the calibration image dataprojection error being determined in accordance with the methods described herein. In the figures, it may be seen that the average reproductive is less than 0.5.

[0104] In implementations, it may be further possible to use the image data derived poses determined with the use of image target 408, as described above, to determine the noise in an IMU derived pose.

[0105] In implementations, a properly designed filter may be used to smooth out gyroscope measurements, such as angular position and rate, and / or accelerometer measurements, or accelerometer measurements, such as speed and acceleration measurements. The smooth gyroscope and accelerometer measurements may be used as a reference to calculate noise in gy roscope and accelerometer data that has not been smoothed. In implementations, the following equations may be used to calculate the noise: gyro noise = gyro_raw_measurement - smoothed_gyro_measurement (Equation 6) accelerometer_noise = accelerometer_raw_measurement - smoothed_accelerometer_measurement. (Equation 7)

[0106] In implementations, noise in pose data or raw data may be determined based on temperature. In implementations where calibration head mounted device 403 is stepped through a series of temperatures in the lab, noise due to temperature may be calculated by comparing a standard deviation observed in sensor data at different temperatures. In implementations where displacement mechanism data determined pose is available, noise may be calculated in the IMU data determined pose and / or the image data determined pose by comparing those pose calculations at different temperatures to the displacement mechanism data determined pose.

[0107] In implementations, any combination of the above methods may be used to determine noise content of sensor data for calibration head mounted device 403. In implementations, further combinations and variations of the noise calculating methods described herein are possible.

[0108] In implementations, computing device 250 may further include noise model generation module 282. Noise model generation module 282, receives the calibration noise sigma values calculated by noise calculation module 280 and data representing motion of the head mounted device generated during calibration activity' and generates a noise to motion relationship for head mounted device 100. Noise model generation module 282 may generate the models described with regards to gyroscope, accelerometer, and / or image noise for dynamic noise module 218 above.

[0109] In implementations, noise model generation module 282 may fit a least-squares curve to the data representing motion of the head mounted device. In implementations, noise model generation module 282 may generate a lookup table of data. It may be possible to interpolate between calibration noise sigma values in the lookup table using an interpolation method such as a linear interpolation method or is spline interpolation method. Once noise model generation module 282 has generated a noise to motion relationship, device 200 and / or head mounted device 100 may use the noise to motion relationship to determine a motion-correlated noise sigma value via dynamic noise module 218. The motion-correlated noise sigma value may be used by pose estimation module 220 to determine a more accurate pose estimation for head mounted device 100.

[0110] By determining a motion-correlated noise sigma value based on a noise to motion relationship, it may be possible to dynamically account for a greater variety’ of physical situations that generate different noise circumstances. Using a more accurate noise estimation may help pose estimation module 220 generate a more accurate pose estimation. This may allow for more stable image placement in a field-of-view of a user wearing an XR head mounted device, thereby allowing for a new set of applications such as real-time navigation while walking or driving.

[0111] FIG. 5A depicts system 500, in accordance with implementations. System 500 may be executed to determine motion-correlated noise sigma value 310 and / or estimated pose 308, as described above.

[0112] System 500 may include any combination of data receiving module 502, dynamic noise module 218, and pose estimation module 220. Data receiving module 502 may include any combination of IMU data receiving module 212, image data receiving module 214, and temperature data receiving module 216, as described above.

[0113] FIG. 5B depicts system 550, and accordance with implementations. System 550 may be executed to generate a noise-to-motion relationship between calibration noise sigma value and data representing the motion of calibration head mounted device 403, as described above.

[0114] System 550 may include any combination of command sending module 552, noise calculation module 280, and noise model generation module 282. Command sending module 552 may comprise any combination of displacement mechanism command module 258, vibration table command module 260, temperature control module 262, camera operation module 264, and / or light command module 266, as described above.

[0115] FIG. 6A depicts method 600. in accordance with an implementation of the description. Method 600 may be used to determine motion-correlated noise sigma value 310and / or estimated pose 308, as described above.

[0116] Method 600 begins with step 602. In step 602. operational motion sensor data may be received from head mounted device 100. In implementations, any combination of gyroscope data 302, accelerometer data 304, image data 306, camera temperature 312, IMU temperature 314, light intensity' data 316, and / or camera shutter exposure time 318 may be received from head mounted device 100, as described above.

[0117] Method 600 may continue with step 604. In step 604, the motion-correlated noise sigma value may be determined based on the operational motion sensor data and a noise to motion relationship. In implementations, step 604 may comprise executing dynamic noise module 218, as described above.

[0118] In implementations, executing step 604 may further comprise executing step 606. In step 606, a first motion-correlated noise sigma value and the second motion- correlated noise sigma value received from a lookup table may be interpolated between, as is described above.

[0119] Method 600 may continue with step 608. In step 608 an estimated pose may be determined using the operational motion sensor data and the motion-correlated noise sigma value. In implementations, executing step 608 may comprise executing pose estimation module 220, as described above.

[0120] FIG. 6B depicts method 650, in accordance with an implementation of the description. Method 650 may be executed to generate a noise-to-motion relationship between calibration noise sigma value and data representing the motion of calibration head mounted device, as described above.

[0121] Method 650 may begin with step 652. In step 652, a command may be sent to at least one of: operate as temperature controller, operate a displacement mechanism, operate a vibration table, or operate a light. In implementations, executing step 652 may comprise executing displacement mechanism command module 258, vibration table command module 260, temperature control module 262, or light command module 266, as described above.

[0122] Method 650 may continue with step 654. In step 654, calibration noise sigma value may be determined based on calibration sensor data from a calibration head mounted device and calibration motion sensor data gathered during a calibration activity. In implementations, executing step 654 may comprise executing noise calculation module 280, as described above.

[0123] Method 650 may continue with step 656. In step 656, a noise to motion relationship may be generated between the calibration noise sigma value and the calibrationmotion sensor data. In implementations, executing step 656 may comprise executing noise model generation module 282. as described above.

[0124] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. Various implementations of the systems and techniques described here can be realized as and / or generally be referred to herein as a circuit, a module, a block, or a system that can combine software and hardware aspects. A module may include the functions / acts / computer program instructions executing on a processor or some other programmable data processing apparatus.

[0125] Some of the above implementations are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations as sequential processes, many of the operations may be performed in parallel, concurrently or simultaneously. In addition, the order of operations may be re-arranged. The processes may be terminated when their operations are completed but may also have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0126] Methods discussed above, some of which are illustrated by the flow charts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine or computer readable medium such as a storage medium. A processor(s) may perform the necessary tasks.

[0127] Specific structural and functional details disclosed herein are merely representative for the purposes of describing implementations. Implementations, however, have many alternate forms and should not be construed as limited to only the implementations set forth herein.

[0128] It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. A first element may betermed a second element, and, similarly, a second element may be termed a first element, without departing from the scope of implementations. As used herein, the term and / or includes any and all combinations of one or more of the associated listed items.

[0129] The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of implementations. As used herein, the singular forms a, an, and the are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms comprises, comprising, includes and / or including, when used herein, specify the presence of stated features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0130] It should also be noted that in some alternative implementations, the functions / acts noted may occur out of the order noted in the figures. Two figures show n in succession may in fact be executed concurrently or may sometimes be executed in the reverse order, depending upon the functionality / acts involved.

[0131] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which implementations belong. It will be further understood that terms, e.g., those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0132] Portions of the above implementations and corresponding detailed description are presented in terms of softw are, or algorithms and symbolic representations of operation on data bits within a computer memory. These descriptions and representations are the ones by which those of ordinary skill in the art effectively convey the substance of their work to others of ordinary skill in the art. An algorithm, as the term is used here, and as it is used generally, is conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0133] In the above illustrative implementations, reference to acts and symbolic representations of operations (e g., in the form of flow-charts) that may be implemented asprogram modules or functional processes include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types and may be described and / or implemented using existing hardware at existing structural elements. Such existing hardware may include one or more Central Processing Units (CPUs), Graphics Processing Units (GPUs), digital signal processors (DSPs), application-specific- integrated-circuits, field programmable gate arrays (FPGAs) computers or the like.

[0134] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, or as is apparent from the discussion, terms such as processing or computing or calculating or determining of displaying or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical, electronic quantities within the computer system’s registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0135] Note also that the software implemented aspects of the implementations are typically encoded on some form of non-transitory program storage medium or implemented over some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or a hard drive) or optical (e.g., a compact disk read only memory, or CD ROM), and may be read only or random access. Similarly, the transmission medium may be twisted wire pairs, coaxial cable, optical fiber, or some other suitable transmission medium known to the art. The implementations not limited by these aspects of any given implementation.

[0136] Lastly, it should also be noted that whilst the accompanying claims set out particular combinations of features described herein, the scope of the present disclosure is not limited to the particular combinations hereafter claimed, but instead extends to encompass any combination of features or implementations herein disclosed irrespective of whether or not that particular combination has been specifically enumerated in the accompanying claims at this time.

[0137] In some aspects, the techniques described herein relate to a method, wherein the operational motion sensor data includes at least one of: inertial measurement unit data or image data.

[0138] In some aspects, the techniques described herein relate to a method, wherein the motion-correlated noise sigma value is further determined based on at least one of: temperature data, exposure time data, or light intensity data.

[0139] In some aspects, the techniques described herein relate to a method, wherein the calibration noise sigma value and the calibration motion sensor data are generated during a calibration activity.

[0140] In some aspects, the techniques described herein relate to a method, wherein the calibration motion sensor data is related to a motion of a displacement mechanism coupled to the calibration head mounted device.

[0141] In some aspects, the techniques described herein relate to a method, wherein the noise-to-motion relationship includes a least squares fit curve.

[0142] In some aspects, the techniques described herein relate to a method, wherein the noise-to-motion relationship includes a lookup table.

[0143] In some aspects, the techniques described herein relate to a method, wherein determining the motion-correlated noise sigma value further includes: interpolating between a first motion-correlated noise sigma value and a second motion-correlated noise sigma value received from the lookup table.

[0144] In some aspects, the techniques described herein relate to a method, wherein the estimated pose is estimated using a Kalman filter.

[0145] In some aspects, the techniques described herein relate to a method, wherein the estimated pose is estimated based on a previous speed determined from a previous estimated pose.

[0146] In some aspects, the techniques described herein relate to a device, wherein the operational motion sensor data includes at least one of: inertial measurement unit data or image data.

[0147] In some aspects, the techniques described herein relate to a device, wherein the motion-correlated noise sigma value is further determined based on at least one of: temperature data, exposure time data, or light intensity data.

[0148] In some aspects, the techniques described herein relate to a device, wherein the calibration noise sigma value and the calibration motion sensor data are generated during a calibration activity.

[0149] In some aspects, the techniques described herein relate to a device, wherein the calibration motion sensor data is related to a motion of a displacement mechanism coupled to the calibration head mounted device.

[0150] In some aspects, the techniques described herein relate to a device, wherein the noise-to-motion relationship includes a least squares fit curve.

[0151] In some aspects, the techniques described herein relate to a device, wherein the noise-to-motion relationship includes a lookup table.

[0152] In some aspects, the techniques described herein relate to a device, wherein the memory is further configured with instructions to determine the motion-correlated noise sigma value by interpolating between a first motion-correlated noise sigma value and a second motion-correlated noise sigma value received from the lookup table.

[0153] In some aspects, the techniques described herein relate to a device, wherein the memory is further configured with instructions to determine the estimated pose using a Kalman filter.

[0154] In some aspects, the techniques described herein relate to a device, wherein the estimated pose is further determined based on a previous speed determined from a previous estimated pose.

[0155] In some aspects, the techniques described herein relate to a system, wherein the operational motion sensor data includes at least one of: inertial measurement unit data or image data.

[0156] In some aspects, the techniques described herein relate to a system, wherein the motion-correlated noise sigma value is further determined based on at least one of: temperature data, exposure time data, or light intensity data.

[0157] In some aspects, the techniques described herein relate to a system, wherein the calibration noise sigma value and the calibration motion sensor data are generated during a calibration activity.

[0158] In some aspects, the techniques described herein relate to a system, wherein the calibration motion sensor data generated during the calibration activity is related to a motion of a displacement mechanism coupled to the calibration head mounted device.

[0159] In some aspects, the techniques described herein relate to a system, wherein the noise-to-motion relationship includes a least squares fit curve.

[0160] In some aspects, the techniques described herein relate to a system, wherein the noise-to-motion relationship includes a lookup table.

[0161] In some aspects, the techniques described herein relate to a system, wherein the dynamic noise module is further configured to determine the motion-correlated noise sigma value by interpolating between a first motion-correlated noise sigma value and a second motion-correlated noise sigma value received from the lookup table.

[0162] In some aspects, the techniques described herein relate to a system, wherein the pose estimation module determines the estimated pose using a Kalman filter.

[0163] In some aspects, the techniques described herein relate to a system, wherein the estimated pose is determined based on a previous speed determined from a previous estimated pose.

[0164] In some aspects, the techniques described herein relate to a method, wherein the calibration sensor data from the calibration head mounted device includes at least one of: inertial measurement unit data or image data.

[0165] In some aspects, the techniques described herein relate to a method, wherein the calibration motion sensor data is related to a motion of a displacement mechanism coupled to the calibration head mounted device.

[0166] In some aspects, the techniques described herein relate to a method, wherein the calibration motion sensor data is generated from a sensor that is separate from the calibration head mounted device.

[0167] In some aspects, the techniques described herein relate to a method, wherein the noise-to-motion relationship includes a least squares fit curve.

[0168] In some aspects, the techniques described herein relate to a method, wherein the noise-to-motion relationship includes a lookup table.

[0169] In some aspects, the techniques described herein relate to a method, further including: sending a command to at least one of: operate a temperature controller, operate a displacement mechanism, operate a vibration table, or operate a light.

[0170] In some aspects, the techniques described herein relate to a method, wherein the calibration sensor data includes image data of a calibration target.

[0171] In some aspects, the techniques described herein relate to a system, wherein the calibration sensor data from the calibration head mounted device includes at least one of: inertial measurement unit data or image data.

[0172] In some aspects, the techniques described herein relate to a system, wherein the calibration sensor data includes at least one of: temperature data, exposure time data, or light intensity data.

[0173] In some aspects, the techniques described herein relate to a system, wherein the calibration motion sensor data is related to a motion of a displacement mechanism coupled to the calibration head mounted device.

[0174] In some aspects, the techniques described herein relate to a system, wherein the calibration motion sensor data is generated from a sensor that is separate from the calibration head mounted device.

[0175] In some aspects, the techniques described herein relate to a system, wherein the noise model generation module is further configured to determine the noise-to-motion relationship using a least squares fit curve.

[0176] In some aspects, the techniques described herein relate to a system, wherein the noise model generation module is further configured to determine the noise-to-motion relationship using a lookup table.

[0177] In some aspects, the techniques described herein relate to a system, further including: a command sending module configured to send a command to at least one of: operate a temperature controller, operate a displacement mechanism, operate a vibration table, or operate a light.

[0178] In some aspects, the techniques described herein relate to a system, wherein the calibration sensor data includes image data of a calibration target.

Claims

WHAT IS CLAIMED IS:

1. A method, comprising: receiving operational motion sensor data from a head mounted device in use; determining a motion-correlated noise sigma value based on the operational motion sensor data and a noise-to-motion relationship characterizing a relationship between a calibration noise sigma value for a calibration head mounted device and calibration motion sensor data; and determining an estimated pose using the operational motion sensor data and the motion-correlated noise sigma value.

2. The method of claim 1, wherein the operational motion sensor data comprises at least one of: inertial measurement unit data or image data.

3. The method of claim 1 or 2, wherein the motion-correlated noise sigma value is further determined based on at least one of: temperature data, exposure time data, or light intensity data.

4. The method of any of the preceding claims, wherein the calibration noise sigma value and the calibration motion sensor data are generated during a calibration activity.

5. The method of claim 4, wherein the calibration motion sensor data is related to a motion of a displacement mechanism coupled to the calibration head mounted device.

6. The method of any of the preceding claims, wherein the noise-to-motion relationship comprises a least squares fit curve.

7. The method of any of claims 1 to 5, wherein the noise-to-motion relationship comprises a lookup table.

8. The method of claim 7, wherein determining the motion-correlated noise sigma value further comprises: interpolating between a first motion-correlated noise sigma value and a second motion-correlated noise sigma value received from the lookup table.

9. The method of any of the preceding claims, wherein the estimated pose is estimated using a Kalman filter.

10. The method of any of the preceding claims, wherein the estimated pose is estimated based on a previous speed determined from a previous estimated pose.

11. A device comprising: a processor; and a memory configured with instructions to perform the method of any one of claims 1 to 10.

12. A system comprising: a data receiving module configured to receive operational motion sensor data from a head mounted device in use; a dynamic noise module configured to determine a motion-correlated noise sigma value based on the operational motion sensor data and a noise-to-motion relationship characterizing a relationship between a calibration noise sigma value for a calibration head mounted device and calibration motion sensor data; and a pose estimation module configured to determine an estimated pose using the operational motion sensor data and the motion-correlated noise sigma value.

13. The system of claim 12, wherein the operational motion sensor data comprises at least one of: inertial measurement unit data or image data.

14. The system of claim 12 or 13, wherein the motion-correlated noise sigma value is further determined based on at least one of: temperature data, exposure time data, or light intensity data.

15. The system of any of claims 12 to 23, wherein the calibration noise sigma value and the calibration motion sensor data are generated during a calibration activity.

16. The system of claim 15, wherein the calibration motion sensor data generated during the calibration activity is related to a motion of a displacement mechanism coupled to the calibration head mounted device.

17. The system of any of claims 12 to 25, wherein the noise-to-motion relationship comprises a least squares fit curve.

18. The system of any of claims 12 to 25, wherein the noise-to-motion relationship comprises a lookup table.

19. The system of claim 18, wherein the dynamic noise module is further configured to determine the motion-correlated noise sigma value by interpolating between a first motion- correlated noise sigma value and a second motion-correlated noise sigma value received from the lookup table.

20. The system of any of claims 12 to 28, wherein the pose estimation module determines the estimated pose using a Kalman filter.

21. The system of any of claims 12 to 29, wherein the estimated pose is determined based on a previous speed determined from a previous estimated pose.

22. A method comprising: determining a calibration noise sigma value based on calibration sensor data from a calibration head mounted device and calibration motion sensor data generated during a calibration activity; and generating a noise-to-motion relationship between the calibration noise sigma value and the calibration motion sensor data characterizing a relationship between a calibration noise sigma value for a calibration head mounted device and calibration motion sensor data.

23. The method of claim 22, wherein the calibration sensor data from the calibration head mounted device comprises at least one of: inertial measurement unit data or image data.

24. The method of claim 22 or 23, wherein the calibration motion sensor data is related to a motion of a displacement mechanism coupled to the calibration head mounted device.

25. The method of any of claims 22 to 33, wherein the calibration motion sensor data is generated from a sensor that is separate from the calibration head mounted device.

26. The method of any of claims 22 to 34, wherein the noise-to-motion relationship comprises a least squares fit curve.

27. The method of any of claims 22 to 34, wherein the noise-to-motion relationship comprises a lookup table.

28. The method of any of claims 22 to 36, further comprising: sending a command to at least one of: operate a temperature controller, operate a displacement mechanism, operate a vibration table, or operate a light.

29. The method of any of claims 22 to 37, wherein the calibration sensor data includes image data of a calibration target.

30. A system comprising: a processor; and a memory configured with instructions to perform the method of any one of claims 22 to 29.

Images