Long-term on-site IMU temperature calibration

Abstract

A method for calibrating a vision-inertial tracking system is described. In one aspect, a method includes measuring a temperature of an inertial measurement unit (IMU) of a vision-inertial tracking system, identifying an IMU intrinsic parameter estimate corresponding to the temperature from an IMU parametric model of an IMU calibration module, determining an online IMU intrinsic parameter estimate by operating the vision-inertial tracking system with the IMU intrinsic parameter estimate, and determining an online IMU intrinsic parameter estimate by operating the vision-inertial tracking system with the IMU intrinsic parameter estimate. The online IMU intrinsic parameter estimates are provided to an IMU calibration module, and the IMU parameter model is updated and incorporated with the online IMU intrinsic parameter estimates.

Description

[0001] Priority interest

[0002] This application claims the benefit of priority to U.S. patent application serial number 18 / 063,450, filed on December 8, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The subject matter disclosed herein generally relates to visual tracking systems. In particular, the present disclosure provides systems and methods for calibrating an inertial measurement unit of a visual-inertial tracking system. Background Art

[0004] The performance quality of a mixed reality (MR) system generally depends on the accuracy of its calibration parameters, and specifically on calibration parameters related to the operation and interoperation between image sensors and non-image sensors (accelerometers, gyroscopes, magnetometers) of the device implementing the MR system. Currently, most MR systems are calibrated using an averaging technique (sometimes called "average calibration"), which involves performing a factory calibration of a large number of devices of the same model and averaging the resulting calibration parameters of each device. The average parameters are then used for systems that have not been factory calibrated. The average calibration of a given MR system often lacks accuracy, especially when compared to the factory calibration of the MR system. In addition, due to mechanical stresses and temperature changes in the MR system, the factory calibration parameters may drift over time when the user wears the MR system. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] To easily identify the discussion of any particular element or act, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0006] Figure 1 is a block diagram illustrating an environment for operating a display device according to an example embodiment.

[0007] Figure 2 is a block diagram illustrating a display device according to an example embodiment.

[0008] Figure 3 is a block diagram illustrating a visual-inertial tracking system according to one example implementation.

[0009] Figure 4 is a block diagram illustrating an IMU calibration module according to an example implementation.

[0010] Figure 5 is a block diagram illustrating a calibration process according to one example implementation.

[0011] Figure 6 is a flowchart showing a method for generating a lookup table according to an example embodiment.

[0012] Figure 7 is a block diagram showing a software architecture in which the present disclosure may be implemented according to an example embodiment.

[0013] Figure 8 is a graphical representation of a machine in the form of a computer system within which a set of instructions may be executed to cause the machine to perform any one or more of the methods discussed herein. Detailed Description

[0014] The following description describes systems, methods, techniques, instruction sequences, and computer program products that illustrate example embodiments of the present subject matter. In the following description, for purposes of explanation, numerous specific details are set forth to provide an understanding of the various embodiments of the present subject matter. However, it will be apparent to those skilled in the art that embodiments of the present subject matter may be practiced without some or other of these specific details. The examples merely represent possible variations. Unless otherwise explicitly stated, structures (e.g., structural components such as modules) are optional and may be combined or subdivided, and operations (e.g., in a process, algorithm, or other function) may vary in order or be combined or subdivided.

[0015] As used herein, the term “MR system” includes augmented reality (AR), virtual reality (VR). The term “augmented reality” (AR) refers to an interactive experience of a real-world environment in which physical objects residing in the real world are “augmented” or enhanced by computer-generated digital content (also referred to as virtual content or synthetic content). AR may also refer to a system that realizes the combination of the real and virtual worlds, real-time interaction, and 3D registration of virtual and real objects. A user of an AR system perceives virtual content that appears to be attached to or interact with real-world physical objects. The term “virtual reality” (VR) refers to a simulated experience of a virtual world environment that is completely different from the real-world environment. The computer-generated digital content is displayed in the virtual world environment. VR also refers to a system that enables a user of a VR system to be fully immersed in the virtual world environment and interact with virtual objects presented in the virtual world environment.

[0016] Mixed Reality (MR) systems typically employ some form of motion tracking to keep a record of the motion of the MR system in terms of its position and orientation as the MR system moves through an area. A Visual-Inertial Odometry (VIO) system is used to track the motion. The MR system uses VIO to estimate its position and orientation (referred to as "pose") in real time based on image data captured by one or more image sensors and further based on Inertial Measurement Unit (IMU) data (e.g., a combination of one or more of accelerometer measurements, gyroscope measurements, and magnetometer measurements) obtained via the IMU sensors of the MR system. If the MR system cannot accurately identify its pose, the quality of the user experience associated with the MR system may be degraded. The performance quality of the MR system depends on the accuracy of the calibration parameters of the MR system: the accuracy of the calibration parameters associated with its VIO system (sometimes referred to herein as "VIO calibration parameters"). The VIO system calibrates the sensors of the MR system to accurately determine the pose of the MR system.

[0017] Rather than calibrating each individual system at the factory, traditional MR systems rely on average calibration for setting the VIO calibration parameters, where the average value of such VIO calibration parameters corresponding to the factory calibrations across multiple MR systems is calculated and then used as the VIO calibration parameters for a given MR system that has not been factory calibrated.

[0018] However, compared to system-specific calibration, such average calibration typically introduces an undesired increase in the number of motion tracking errors and tracking failures (e.g., including VIO resets and VIO initialization failures). For example, although the tracking system of an MR system may be factory calibrated, non-intrinsic / intrinsic parameters may change over time (e.g., due to mechanical stress, temperature variations). The tracking system mitigates these changes by gradually updating the parameter values during the runtime of AR / VR applications. However, the more the parameter values deviate from the factory calibration, the longer it takes for the tracking system to "catch up" and obtain an accurate new estimate of the parameter values (e.g., the convergence time becomes longer). It is important to keep this convergence time short because as long as the estimates are inaccurate, the tracking performance will be negatively affected. Additionally, the parameter values can be used by components other than the tracking system. For applications that rely on these components, it is important that the tracking system operates accurately from the start (e.g., when the AR / VR application starts or goes online).

[0019] Precise localization may come with a high computational cost. IMUs contribute to a more efficient and low-power VIO system. The more accurately all IMU parameters are known, the better the accuracy of the tracking system. More accurate IMU measurements enable increasing the precise time of the system without external calibration. Thus, precise IMU parameters enable reducing the computational cost of the expensive pose correction using the images of the camera device. However, IMU parameters are not constant over time and are affected by many factors such as temperature. Usually, the underlying filtering / optimization algorithms use a general model that accounts for random variations to estimate the changing IMU parameters on the fly. However, the changes caused by temperature are systematic and larger than the non-systematic changes. Since they are systematic, they do not have to be modeled by an inaccurate general model that is designed to estimate the smaller non-systematic variations of the bias and other intrinsic IMU parameters. Temperature calibration of the IMU can be performed during the manufacturing process of the device. However, calibration at the time of manufacturing requires a large amount of time and specialized equipment such as a large temperature chamber. This application describes an online estimation process that uses the natural temperature variations to which the device is exposed during operation to estimate the temperature model on the fly. Online estimation enables compensating for the systematic variations and making the part that accounts for random variations as small as possible. Such online estimation achieves better IMU performance and unlocks more efficient and precise VIO algorithms.

[0020] This application describes a method for reducing the convergence time of online parameter estimation at the start of a tracking system. The tracking system measures the current temperature of the IMU and identifies an IMU bias estimate corresponding to that temperature based on a parameter model (e.g., a look-up table). By leveraging the on-the-fly estimation performed in a Visual-Inertial Odometry (VIO) system, the tracking system performs online calibration of the dependence of the IMU parameters on the temperature of the IMU (or the tracking system). When high accuracy is achieved through online estimation, the intrinsic parameters are stored. The result is a look-up table / parameter temperature model that can be used to estimate the IMU bias (or dynamically adjust the random walk model to compensate for larger errors). The advantages of temperature-precise intrinsic IMU calibration enable the tracking system to operate closer to the correct values, and thus: (1) reduce the systematic errors introduced in the tracking system, (2) reduce the computational power required to estimate the intrinsic parameters, and (3) provide better localization accuracy.

[0021] In one example embodiment, the present application describes a method for calibrating a visual-inertial tracking system, including: measuring the temperature of an inertial measurement unit (IMU) of the visual-inertial tracking system, identifying an IMU intrinsic parameter estimate corresponding to the temperature from an IMU parameter model of an IMU calibration module, determining an online IMU intrinsic parameter estimate by operating the visual-inertial tracking system using the IMU intrinsic parameter estimate, providing the online IMU intrinsic parameter estimate to the IMU calibration module, and updating and incorporating the IMU parameter model with the online IMU intrinsic parameter estimate.

[0022] As a result, one or more of the methods described herein help to address the technical problems of power consumption savings and efficient calibration by generating and updating a look-up table for bias parameters based on IMU temperature. The currently described methods provide an improvement to the functional operation of a computer by providing power consumption reduction and the fastest calibration calculations. Thus, one or more of the methods described herein can avoid the need for certain efforts or computational resources. Examples of such computational resources include processor cycles, network traffic, memory usage, data storage capacity, power consumption, network bandwidth, and cooling capacity.

[0023] Figure 1 is a network diagram showing an environment 100 suitable for operating a display device 106 according to some example embodiments. The environment 100 includes a user 102, a display device 106, and a physical object 104. The user 102 operates the display device 106. The user 102 can be a human user (e.g., a human), a machine user (e.g., a computer configured by a software program to interact with the display device 106), or any suitable combination thereof (e.g., a human assisted by a machine or a machine supervised by a human). The user 102 is associated with the display device 106.

[0024] The display device 106 can be a computing device having a display, such as a smart phone, a tablet computer, or a wearable computing device (e.g., a watch or glasses). The computing device can be handheld or can be removably mounted to the head of the user 102. In one example, the display includes a screen for displaying an image captured by a camera device of the display device 106. In another example, the display of the device can be transparent, such as in the lenses of wearable computing glasses. In other examples, the display can be opaque, partially transparent, or partially opaque. In other examples, the display can be worn by the user 102 to cover the field of view of the user 102.

[0025] The display device 106 includes an AR application that generates virtual content based on an image detected by a camera device of the display device 106. For example, the user 102 can direct the camera device of the display device 106 to capture an image of a physical object 104 (e.g., a physical item having a visual structure / feature). The AR application generates virtual content corresponding to the recognized object in the image (e.g., the physical object 104) and presents the virtual content in the display of the display device 106.

[0026] The display device 106 includes a visual-inertial tracking system 108. The visual-inertial tracking system 108 uses, for example, optical sensors (e.g., a depth-enabled 3D camera device, an image camera device), inertial sensors (e.g., gyroscopes, accelerometers), wireless sensors (Bluetooth, Wi-Fi), GPS sensors, and audio sensors (e.g., a microphone array) to track the pose (e.g., location and orientation) of the display device 106 relative to the real-world environment 110. In one example, the display device 106 displays virtual content based on the pose of the display device 106 relative to the real-world environment 110 and / or the physical object 104.

[0027] Figure 1 Any one of the machines, databases, or devices shown can be implemented in a general-purpose computer that has been modified by software (e.g., configured or programmed) to be a special-purpose computer to perform one or more of the functions described herein for that machine, database, or device. For example, a computer system capable of implementing any one or more of the methods described herein is discussed below. As used herein, a "database" is a data storage resource and can store data structured as text files, tables, spreadsheets, relational databases (e.g., object-relational databases), triple stores, hierarchical data stores, or any suitable combination thereof. Additionally, Figure 7 Any two or more of the machines, databases, or devices shown can be combined into a single machine, and the functions described herein for any single machine, database, or device can be subdivided among multiple machines, databases, or devices. Figure 1

[0028] The display device 106 can operate over a computer network. A computer network can be any network that enables communication between or among machines, databases, and devices. Thus, a computer network can be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. A computer network can include one or more portions that make up a private network, a public network (e.g., the Internet), or any appropriate combination thereof.

[0029] Figure 2 ​is a block diagram showing modules (e.g., components) of a display device 106 according to some example embodiments. The display device 106 includes a sensor 202, a display 204, a processor 208, and a storage device 206. Examples of the display device 106 include a wearable computing device, a mobile computing device, a navigation device, a portable media device, or a smart phone.

[0030] The sensor 202 includes, for example, an optical sensor 212 (e.g., a camera device such as a color camera device, a thermal camera device, a depth sensor, and one or more grayscale, global / rolling shutter tracking camera devices), a temperature sensor 224, and an inertial sensor 214 (e.g., a gyroscope, an accelerometer, a magnetometer). Other examples of the sensor 202 include a proximity sensor or a position sensor (e.g., near field communication, GPS, Bluetooth, Wi-Fi), an audio sensor (e.g., a microphone), or any suitable combination thereof. Note that the sensor 202 described herein is for illustrative purposes, and thus the sensor 202 is not limited to the sensors described above.

[0031] The display 204 includes a screen or a monitor configured to display an image generated by the processor 208. In one example embodiment, the display 204 may be transparent or semi-opaque such that the user 102 can view through the display 204 (in an AR use case). In another example embodiment, the display 204 covers the eyes of the user 102 and blocks the entire field of view of the user 102 (in a VR use case). In another example, the display 204 includes a touch screen display configured to receive user input via a contact on the touch screen display. In other examples, the display device 106 includes a touch pad for receiving user input.

[0032] The processor 208 includes a mixed reality application 210, a visual-inertial tracking system 108, and an IMU calibration module 216. The mixed reality application 210 uses computer vision to detect and recognize a physical environment or physical object 104. The mixed reality application 210 retrieves virtual content (e.g., a 3D object model) based on the recognized physical object 104 or physical environment. The mixed reality application 210 presents virtual objects in the display 204. In one example implementation, the mixed reality application 210 includes a local rendering engine that generates a visualization of virtual content that overlays (e.g., is superimposed on or otherwise displayed in cooperation with) an image of the physical object 104 captured by the optical sensor 212. The visualization of the virtual content can be manipulated by adjusting the positioning of the physical object 104 relative to the optical sensor 212 (e.g., its physical position, orientation, or both). Similarly, the visualization of the virtual content can be manipulated by adjusting the pose of the display device 106 relative to the physical object 104. For VR applications, the mixed reality application 210 displays virtual content in the display 204 at a position (in the display 204) determined based on the pose of the display device 106.

[0033] The visual-inertial tracking system 108 estimates the pose of the display device 106. For example, the visual-inertial tracking system 108 uses image data and corresponding inertial data from the optical sensor 212 and the inertial sensor 214 to track the position and pose of the display device 106 relative to a reference frame (e.g., the real-world environment 110).

[0034] The IMU calibration module 216 calibrates the IMU sensors of the inertial sensor 214 based on IMU bias parameter values from the IMU intrinsic model 220. When the mixed reality application 210 is running, the visual-inertial tracking system 108 can be referred to as online. When the mixed reality application 210 stops running, the visual-inertial tracking system 108 can be referred to as offline. When the visual-inertial tracking system 108 is online, the visual-inertial tracking system 108 runs an online estimation of the IMU bias parameters until a new IMU bias estimate is obtained. The IMU calibration module 216 accesses the temperature of the inertial sensor 214 measured by the temperature sensor 224. The IMU calibration module 216 stores and updates the new IMU bias estimate and the corresponding measured temperature in the IMU intrinsic model 220 in the storage device 206. When the visual-inertial tracking system 108 is running, the IMU calibration module 216 re-uses the most recent IMU bias estimate based on the temperature of the IMU as a more recent starting point for another parameter estimation. Using this new estimate as a starting point for another parameter estimation results in a reduced convergence time and higher tracking accuracy immediately after the visual-inertial tracking system 108 is started.

[0035] The storage device 206 stores virtual content 218, factory calibration parameters 222, and an IMU intrinsic model 220. The virtual content 218 includes, for example, a database of visual references (e.g., images of physical objects) and corresponding experiences (e.g., three-dimensional virtual object models). The IMU intrinsic model 220 includes, for example, a look-up table that calculates the most recent IMU bias estimation parameter values of the visual-inertial tracking system 108 and the corresponding IMU temperature. In one example, the IMU intrinsic model 220 updates the most recent bias estimation parameter values based on the most recently measured IMU temperature and the most recently estimated parameter values.

[0036] Any one or more of the modules described herein may be implemented using hardware (e.g., a processor of a machine) or a combination of hardware and software. For example, any module described herein may configure a processor to perform the operations described herein for that module. Additionally, any two or more of these modules may be combined into a single module, and the functions described herein for a single module may be subdivided among multiple modules. Further, according to various example embodiments, modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0037] Figure 3 A visual-inertial tracking system 108 is shown in accordance with one example embodiment. The visual-inertial tracking system 108 includes, for example, an inertial sensor module 302, an optical sensor module 304, and a pose estimation module 306. The inertial sensor module 302 accesses inertial sensor data from the inertial sensor 214. The optical sensor module 304 accesses optical sensor data from the optical sensor 212.

[0038] The pose estimation module 306 determines the pose (e.g., position, orientation, orientation) of the display device 106 relative to a reference frame (e.g., the real-world environment 110). In one example embodiment, the pose estimation module 306 includes a visual odometry system that estimates the pose of the display device 106 based on a 3D map of feature points from images captured by the optical sensor 212 and inertial sensor data captured by the inertial sensor 214. The optical sensor module 304 accesses image data from the optical sensor 212.

[0039] In one example embodiment, the pose estimation module 306 calculates the position and orientation of the display device 106. The display device 106 includes one or more optical sensors 212 and one or more inertial sensors 214 mounted on a rigid platform (the frame of the display device 106). The optical sensors 212 may be mounted with non-overlapping (distributed aperture) or overlapping (stereo or more) fields of view.

[0040] In some example embodiments, the pose estimation module 306 includes an algorithm that combines inertial information from the inertial sensor 214 and image information from the optical sensor 212, where the optical sensor 212 is coupled to a rigid platform (e.g., the display device 106) or a rig. In one embodiment, the rig may consist of multiple camera devices mounted on a rigid platform and an inertial navigation unit (e.g., the inertial sensor 214). Thus, the rig may have at least one inertial navigation unit and at least one camera device.

[0041] Figure 4 is a block diagram showing an IMU calibration module 216 according to an example embodiment. The IMU calibration module 216 includes an online IMU intrinsic estimation component 402 and an IMU intrinsic temperature component 404.

[0042] The online IMU intrinsic estimation component 402 operates the visual-inertial tracking system 108 when the mixed reality application 210 is running. In one example embodiment, the online IMU intrinsic estimation component 402 reads the current temperature of the inertial sensor 214 and identifies the latest IMU bias estimate from the IMU intrinsic model 220 based on the measured temperature.

[0043] The IMU intrinsic temperature component 404 generates and updates the IMU intrinsic model 220. For example, the IMU intrinsic temperature component 404 determines whether the latest bias estimate (corresponding to the measured temperature of the inertial sensor 214) is still accurate or valid (using the visual-inertial tracking system 108 to detect drift). The IMU intrinsic temperature component 404 runs the visual-inertial tracking system 108 to perform an online bias estimate based on the latest IMU bias estimate (from the IMU intrinsic model 220) or the factory calibration bias estimate depending on whether the latest bias estimate is still accurate. When the bias estimate has converged, the IMU intrinsic temperature component 404 updates the IMU intrinsic model 220. The operation of the IMU intrinsic temperature component 404 is described in more detail below with reference to Figure 5 and Figure 6 The operation of the IMU intrinsic temperature component 404 is described in more detail.

[0044] Figure 5 is a block diagram showing an example process according to an example embodiment. The visual-inertial tracking system 108 receives sensor data from the sensor 202 (e.g., the inertial sensor 214) to determine the pose of the display device 106. The IMU calibration module 216 detects the current temperature of the inertial sensor 214 from the sensor 202.

[0045] The IMU calibration module 216 accesses the IMU bias estimate corresponding to the current temperature of the inertial sensor 214 from the IMU intrinsic model 220. The IMU calibration module 216 determines whether the IMU bias estimate is still accurate or valid. If the IMU bias estimate is no longer valid, the IMU calibration module 216 retrieves the factory calibration parameters 222 (e.g., IMU factory calibration). The IMU calibration module 216 runs the visual-inertial tracking system 108 for online bias estimation and detects whether the IMU bias estimate has converged. Once converged, the IMU calibration module 216 stores and updates the converged bias estimate for the measured temperature in the IMU intrinsic model 220.

[0046] The IMU calibration module 216 calibrates the sensor data (from the inertial sensor 214) based on the IMU bias estimate corresponding to the measured IMU temperature from the IMU intrinsic model 220. The mixed reality application 210 retrieves the virtual content 218 from the storage device 206 and causes the virtual content 218 to be displayed at a specific location based on the geometric model of the display device 106.

[0047] Figure 6 An example routine 600 for updating and incorporating the IMU intrinsic model is shown. Although the example routine 600 depicts a specific sequence of operations, the sequence can be changed without departing from the scope of the present disclosure. For example, some of the depicted operations can be performed in parallel or in a different order that does not substantially affect the functionality of the routine 600. In other examples, different components of an example device or system implementing the routine 600 can perform functions substantially simultaneously or in a specific order.

[0048] The operations in routine 600 can be performed by the visual-inertial tracking system 108 using the components (e.g., modules, engines) described above with respect to Figure 2 Thus, routine 600 is described by way of example with reference to the IMU calibration module 216. However, it should be understood that at least some of the operations in routine 600 can be deployed on various other hardware configurations or performed by similar components residing elsewhere.

[0049] According to some examples, the method includes starting at 602, accessing the current IMU temperature at block 604, obtaining the intrinsic value from the IMU calibration module at block 606, running the tracking system / online estimation of the intrinsic value at block 608, providing the IMU intrinsic value - estimate to the IMU calibration module at block 610, and updating and incorporating the IMU intrinsic model with the new estimated IMU intrinsic value - estimate at block 612.

[0050] Figure 7FIG. 700 is a block diagram showing a software architecture 704 that may be installed on any one or more of the devices described herein. The software architecture 704 is supported by hardware such as a machine 702 that includes a processor 720, a memory 726, and I / O components 738. In this example, the software architecture 704 may be conceptualized as a stack of layers, where each layer provides a particular functionality. The software architecture 704 includes layers such as an operating system 712, libraries 710, frameworks 708, and applications 706. In operation, the application 706 makes API calls 750 through the software stack and receives messages 752 in response to the API calls 750.

[0051] The operating system 712 manages hardware resources and provides common services. The operating system 712 includes, for example, a kernel 714, services 716, and drivers 722. The kernel 714 serves as an abstraction layer between the hardware and the other software layers. For example, the kernel 714 provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functions. The services 716 may provide other common services to the other software layers. The drivers 722 are responsible for controlling or interfacing with the underlying hardware. For example, the drivers 722 may include a display driver, a camera driver, a BLUETOOTH or BLUETOOTH low energy driver, a flash drive, a serial communication driver (e.g., a Universal Serial Bus (USB) driver), a WI-FI driver, an audio driver, a power management driver, and the like.

[0052] The libraries 710 provide low-level common infrastructure used by the applications 706. The libraries 710 may include system libraries 718 (e.g., a C standard library) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. Additionally, the libraries 710 may include API libraries 724, such as media libraries (e.g., libraries for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., an OpenGL framework for 2D and 3D rendering of graphical content on a display), database libraries (e.g., SQLite that provides various relational database functions), web libraries (e.g., WebKit that provides web browsing functions), and the like. The libraries 710 may also include various other libraries 728 to provide many other APIs to the applications 706.

[0053] The framework 708 provides a high-level common infrastructure used by the application 706. For example, the framework 708 provides various graphical user interface (GUI) functions, high-level resource management, and high-level location services. The framework 708 may provide a wide range of other APIs that can be used by the application 706, some of which may be specific to a particular operating system or platform.

[0054] In an example implementation, the application 706 may include a home application 736, a contacts application 730, a browser application 732, a book reader application 734, a location application 742, a media application 744, a messaging application 746, a gaming application 748, and various other applications such as third-party applications 740. The application 706 is a program that executes functions defined in a program. One or more of the applications 706 can be created using various programming languages in various ways, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C language or assembly language). In a specific example, the third-party application 740 (e.g., an application developed using an ANDROID TM or IOS TM software development kit (SDK)) can be mobile software that runs on a mobile operating system such as IOS TM , ANDROID TM , WINDOWS Phone or another mobile operating system. In this example, the third-party application 740 can call the API calls 750 provided by the operating system 712 to facilitate the functions described herein.

[0055] Figure 8is an illustrative representation of a machine 800 within which instructions 808 (e.g., software, programs, applications, applets, apps, or other executable code) can be executed to cause the machine 800 to perform any one or more of the methods discussed herein. For example, the instructions 808 can cause the machine 800 to perform any one or more of the methods described herein. The instructions 808 transform the general, unprogrammed machine 800 into a particular machine 800 programmed to perform the described and illustrated functions in the described manner. The machine 800 can operate as a stand-alone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 800 can operate in a server-client network environment as a server machine or a client machine, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 800 can include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), PDAs, entertainment media systems, cellular telephones, smartphones, mobile devices, wearable devices (e.g., smart watches), smart home devices (e.g., smart appliances), other smart devices, web appliances, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing the instructions 808 specifying actions to be taken by the machine 800. Further, while only a single machine 800 is shown, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute the instructions 808 to perform any one or more of the methods discussed herein.

[0056] The machine 800 can include a processor 802, a memory 804, and I / O components 842 configured to communicate with each other via a bus 844. In an example implementation, the processor 802 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), other processors, or any suitable combination thereof) can include, for example, a processor 806 and a processor 810 that execute the instructions 808. The term "processor" is intended to include multi-core processors that can include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 8 multiple processors 802 are shown, the machine 800 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0057] The memory 804 includes a main memory 812, a static memory 814, and a storage unit 816, all of which are accessible by the processor 802 via a bus 844. The main memory 812, the static memory 814, and the storage unit 816 store instructions 808 that embody any one or more of the methods or functions described herein. The instructions 808 may also reside, completely or partially, within the main memory 812, within the static memory 814, within a machine-readable medium 818 within the storage unit 816, within at least one of the processors 802 (e.g., within a cache memory of the processor), or in any suitable combination thereof during execution by the machine 800.

[0058] The I / O component 842 may include a variety of components that receive input, provide output, generate output, transfer information, exchange information, capture measurement results, and so on. The specific I / O components 842 included in a particular machine will depend on the type of the machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine is less likely to include such a touch input device. It should be understood that the I / O component 842 may include many other components not shown in Figure 8 In various example embodiments, the I / O component 842 may include an output component 828 and an input component 830. The output component 828 may include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), tactile components (e.g., a vibration motor, a resistance mechanism), other signal generators, and so on. The input component 830 may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), pointing-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instruments), tactile input components (e.g., physical buttons, a touch screen that provides the location and / or force of a touch or touch gesture, or other tactile input components), audio input components (e.g., a microphone), and so on.

[0059] In other example embodiments, the I / O component 842 can include a biometric component 832, a motion component 834, an environmental component 836, or a positioning component 838, as well as a variety of other components. For example, the biometric component 832 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biometric signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retina recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), and so on. The motion component 834 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), and the like. The environmental component 836 includes, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an auditory sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of hazardous gases to ensure safety or measuring pollutants in the atmosphere), or other components that can provide an indication, measurement, or signal corresponding to the surrounding physical environment. The positioning component 838 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer for detecting the air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), and the like.

[0060] A variety of techniques can be used to implement communication. The I / O component 842 also includes a communication component 840, which is operable to couple the machine 800 to the network 820 or the device 822 via the couplings 824 and 826, respectively. For example, the communication component 840 can include a network interface component or another suitable device to interface with the network 820. In other examples, the communication component 840 can include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, components (e.g., low power consumption), components, and other communication components for providing communication via other modalities. The device 822 can be another machine or any peripheral device among a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0061] In addition, the communication component 840 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 840 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be derived via the communication component 840, such as a location via Internet Protocol (IP) geolocation, a location via Wi-Fi, and a location of a user. The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, and the like.

[0062] Various memories (e.g., memory 804, main memory 812, static memory 814, and / or memory of processor 802) and / or storage unit 816 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 808) when executed by processor 802 cause various operations to implement the disclosed embodiments.

[0063] Instructions 808 may be sent or received over network 820 using a transmission medium via a network interface device (e.g., a network interface component included in communications component 840) and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 808 may be sent or received to device 822 via coupling 826 (e.g., a peer-to-peer coupling) using a transmission medium.

[0064] Example

[0065] Example 1 is a method comprising: measuring the temperature of an inertial measurement unit (IMU) of a visual-inertial tracking system, identifying an IMU intrinsic parameter estimate corresponding to the temperature from an IMU parameter model of an IMU calibration module, determining an online IMU intrinsic parameter estimate by operating the visual-inertial tracking system using the IMU intrinsic parameter estimate, providing the online IMU intrinsic parameter estimate to the IMU calibration module, and updating and incorporating the IMU parameter model with the online IMU intrinsic parameter estimate.

[0066] Example 2 includes the method of Example 1, further comprising: storing online IMU intrinsic parameter estimates and temperature in the IMU parameter model.

[0067] Example 3 includes the method described in Example 1, and further includes: storing the IMU parameter model in a storage device of the visual-inertial tracking system.

[0068] Example 4 includes the method described in Example 1, wherein updating and incorporating the IMU parameter model includes: updating the IMU intrinsic parameter estimate corresponding to temperature with an online IMU intrinsic parameter estimate.

[0069] Example 5 includes the method described in Example 2, wherein the IMU parameter model includes a look-up table, and wherein the look-up table maps the IMU intrinsic parameter estimate to temperature.

[0070] Example 6 includes the method described in Example 1, wherein the IMU parameter model includes an IMU temperature model.

[0071] Example 7 includes the method described in Example 1, wherein the online IMU intrinsic parameter estimate includes an online IMU bias estimate.

[0072] Example 8 includes the method described in Example 1, wherein measuring the temperature of the IMU: includes measuring the temperature of the IMU of the visual-inertial tracking system during operation of the visual-inertial tracking system.

[0073] Example 9 includes the method described in Example 1, wherein measuring the temperature of the IMU includes: periodically measuring the temperature of the IMU of the visual-inertial tracking system.

[0074] Example 10 includes the method described in Example 1, wherein the visual-inertial tracking system is part of an augmented reality display device.

[0075] Example 11 is a computing device, including: a processor; and a memory storing instructions that, when executed by the processor, configure the device to: measure the temperature of an inertial measurement unit (IMU) of a visual-inertial tracking system, identify an IMU intrinsic parameter estimate corresponding to the temperature from an IMU parameter model of an IMU calibration module, determine an online IMU intrinsic parameter estimate by operating the visual-inertial tracking system using the IMU intrinsic parameter estimate, provide the online IMU intrinsic parameter estimate to the IMU calibration module, and update and incorporate the IMU parameter model with the online IMU intrinsic parameter estimate.

[0076] Example 12 includes the computing device according to Example 11, wherein the instructions further configure the device to: store the online IMU intrinsic parameter estimate and the temperature in the IMU parameter model.

[0077] Example 13 includes the computing device according to Example 11, wherein the instructions further configure the device to: store the IMU parameter model in a storage device of the visual-inertial tracking system.

[0078] Example 14 includes the computing device described in Example 11, wherein updating and incorporating the IMU parameter model includes: updating the IMU intrinsic parameter estimate corresponding to temperature with an online IMU intrinsic parameter estimate.

[0079] Example 15 includes the computing device described in Example 12, wherein the IMU parameter model includes a look-up table that maps an IMU intrinsic parameter estimate to temperature.

[0080] Example 16 includes the computing device described in Example 11, wherein the IMU parameter model includes an IMU temperature model.

[0081] Example 17 includes the computing device described in Example 11, wherein the online IMU intrinsic parameter estimate includes an online IMU bias estimate.

[0082] Example 18 includes the computing device described in Example 11, wherein measuring the temperature of the IMU includes: measuring the temperature of the IMU of the visual-inertial tracking system during operation of the visual-inertial tracking system.

[0083] Example 19 includes the computing device described in Example 11, wherein measuring the temperature of the IMU includes: periodically measuring the temperature of the IMU of the visual-inertial tracking system.

[0084] Example 20 is a non-transitory computer-readable storage medium that includes instructions that, when executed by a computer, cause the computer to: measure the temperature of an inertial measurement unit (IMU) of a visual-inertial tracking system, identify an IMU intrinsic parameter estimate corresponding to the temperature from an IMU parameter model of an IMU calibration module, determine an online IMU intrinsic parameter estimate by operating the visual-inertial tracking system using the IMU intrinsic parameter estimate, provide the online IMU intrinsic parameter estimate to the IMU calibration module, and update and incorporate the IMU parameter model with the online IMU intrinsic parameter estimate.

[0085] Although embodiments have been described with reference to specific example embodiments, it will be apparent that various modifications and changes can be made to these embodiments without departing from the broader scope of the disclosure. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The accompanying drawings, which form a part of this invention, illustrate specific embodiments in which the subject matter can be practiced by way of illustration and not by way of limitation. The illustrated embodiments are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments can be utilized and derived therefrom, such that structural and logical substitutions and changes can be made without departing from the scope of the disclosure. Accordingly, the specific embodiments should not be construed in a limiting sense, and the scope of the various embodiments is defined only by the appended claims and the full scope of equivalents to which such claims are entitled.

[0086] These embodiments of the subject matter of the invention may be referred to herein individually and / or collectively by the term "invention" merely for convenience and are not intended to voluntarily limit the scope of this application to any single invention or inventive concept if in fact more than one invention or inventive concept is disclosed. Accordingly, although specific embodiments have been shown and described herein, it should be understood that any arrangement designed to achieve the same purpose may be substituted for the specific embodiments shown. The disclosure is intended to cover any and all adaptations or variations of various embodiments. After reviewing the above description, combinations of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art.

[0087] The abstract of the disclosure is provided to enable the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the foregoing detailed description, for the purposes of simplifying the disclosure, various features are grouped together in a single embodiment. This method of the disclosure should not be construed as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as reflected by the appended claims, the subject matter of the invention lies in less than all of the features of a single disclosed embodiment. Thus, the claims are hereby incorporated into the detailed description, where each claim stands on its own as a separate embodiment.

Claims

1. A method, comprising: Measuring the temperature of an inertial measurement unit (IMU) of a visual-inertial tracking system; Identifying an IMU intrinsic parameter estimate corresponding to the temperature from an IMU parameter model of an IMU calibration module; Determining an online IMU intrinsic parameter estimate by operating the visual-inertial tracking system using the IMU intrinsic parameter estimate; Providing the online IMU intrinsic parameter estimate to the IMU calibration module; And Updating and incorporating the IMU parameter model with the online IMU intrinsic parameter estimate.

2. The method according to claim 1, further comprising: Storing the online IMU intrinsic parameter estimate and the temperature in the IMU parameter model.

3. The method according to claim 1, further comprising: Storing the IMU parameter model in a storage device of the visual-inertial tracking system.

4. The method according to claim 1, wherein, Updating and incorporating the IMU parameter model includes: Updating the IMU intrinsic parameter estimate corresponding to the temperature with the online IMU intrinsic parameter estimate.

5. The method according to claim 2, wherein, The IMU parameter model includes a look-up table, Wherein the look-up table maps the IMU intrinsic parameter estimate to the temperature.

6. The method according to claim 1, wherein The IMU parameter model includes an IMU temperature model.

7. The method according to claim 1, wherein The online IMU intrinsic parameter estimate includes an online IMU bias estimate.

8. The method according to claim 1, wherein Measuring the temperature of the IMU includes: Measuring the temperature of the IMU of the visual-inertial tracking system during operation of the visual-inertial tracking system.

9. The method according to claim 1, wherein Measuring the temperature of the IMU includes: Periodically measuring the temperature of the IMU of the visual-inertial tracking system.

10. The method according to claim 1, wherein, The visual-inertial tracking system is part of an augmented reality display device.

11. A computing device, comprising: A processor; And A memory storing instructions that, when executed by the processor, configure the device to: Measure the temperature of an inertial measurement unit (IMU) of a visual-inertial tracking system; Identify an IMU intrinsic parameter estimate corresponding to the temperature from an IMU parameter model of an IMU calibration module; Determine an online IMU intrinsic parameter estimate by operating the visual-inertial tracking system using the IMU intrinsic parameter estimate; Provide the online IMU intrinsic parameter estimate to the IMU calibration module; And Update and incorporate the IMU parameter model with the online IMU intrinsic parameter estimate.

12. The computing device according to claim 11, wherein, The instructions further configure the device to: Store the online IMU intrinsic parameter estimate and the temperature in the IMU parameter model.

13. The computing device according to claim 11, wherein, The instructions further configure the device to: Store the IMU parameter model in a storage device of the visual-inertial tracking system.

14. The computing device according to claim 11, wherein, Updating and incorporating the IMU parameter model includes: Updating the IMU intrinsic parameter estimate corresponding to the temperature with the online IMU intrinsic parameter estimate.

15. The computing device according to claim 12, wherein, The IMU parameter model includes a look-up table, Wherein the look-up table maps the IMU intrinsic parameter estimate to the temperature.

16. The computing device according to claim 11, wherein, The IMU parameter model includes an IMU temperature model.

17. The computing device according to claim 11, wherein, The online IMU intrinsic parameter estimate includes an online IMU bias estimate.

18. The computing device according to claim 11, wherein, Measuring the temperature of the IMU includes: measuring the temperature of the IMU of the visual-inertial tracking system during operation of the visual-inertial tracking system.

19. The computing device according to claim 11, wherein, Measuring the temperature of the IMU includes: periodically measuring the temperature of the IMU of the visual-inertial tracking system.

20. A non-transitory computer-readable storage medium, the computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to: Measure the temperature of an inertial measurement unit (IMU) of a visual-inertial tracking system; Identify an IMU intrinsic parameter estimate corresponding to the temperature from an IMU parameter model of an IMU calibration module; Determine an online IMU intrinsic parameter estimate by operating the visual-inertial tracking system using the IMU intrinsic parameter estimate; Provide the online IMU intrinsic parameter estimate to the IMU calibration module; And Update and incorporate the IMU parameter model with the online IMU intrinsic parameter estimate.

Images