Artificial Intelligence for IMU Calibration and Sensor Fusion

Abstract

Disclosed herein are systems and methods for computer-aided sensor calibration in which a computer receives raw accelerometer data, raw gyroscope data, and raw magnetometer data and performs spatial calibration. The calibration data is used to calculate orientation data, and the calibrated data is then used to provide corrected linear acceleration data that may be used by the sensor to provide more accurate readings, for example, of physiological data. In one embodiment, the sensor calibration further uses a 9-axis IMU ("Inertial Measurement Unit") including an accelerometer, gyroscope, and magnetometer to provide the calibration data.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 401,448, filed August 26, 2022, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] In the prior art, calibration is typically performed online (i.e., in real time with a certain delay) because these measurements are used downstream in real-time applications. The time-critical nature of some real-time applications makes timely calibration essential. Therefore, existing calibration algorithms must be optimized for highly constrained hardware and software limitations. Therefore, existing algorithms sacrifice quality in exchange for reduced latency. There is interest in receiving highly accurate measurements before using them online to train AI / ML models and make important (e.g., health-related) inferences and decisions. Therefore, technically, calibration algorithms have no theoretical limits in terms of computational power or latency. One novel aspect of the present invention is the use of all the data necessary to effectively back-calibrate against erroneous measurements offline, thereby reducing the overall cumulative error.

[0003] Current state-of-the-art techniques for calibrating various IMU sensors do not account for dynamic changes in bias and sensitivity parameters due to time-varying factors such as temperature and shock. Typically, factory calibration is performed during device manufacturing and / or periodic recalibration routines are performed to proactively adjust for drift and other factors that can cause calibration to become invalid. Instead, one novel idea is to use some of the techniques discussed in this paragraph to systematically react to calibration discrepancies, enabling faster recovery from erroneous calibration parameter estimates. Another novel idea is to use suboptimal online calibration to provide time-dependent situational awareness, but perform offline corrections to provide more accurate inferences at a later time when time and computational power constraints are relaxed.

[0004] The ability to perform back-calibration of sensors affected by time-varying calibration parameters is also novel and does not currently exist, a deficiency remedied by the present invention. Summary of the Invention [Means for solving the problem]

[0005] Disclosed embodiments provide a system for sensor calibration that receives raw accelerometer data, raw gyroscope data, and raw magnetometer data and performs spatial calibration with an inertial measurement unit (IMU). The system then outputs a plurality of calibration parameters from the IMU. The calibrated data from the accelerometer, gyroscope, and magnetometer is sent to a sensor fusion module that calculates an orientation. The orientation and the calibrated accelerometer data are then sent to a gravity cancellation module that calculates a compensated linear acceleration.

[0006] In one embodiment, there are 18 calibration parameters, and in another embodiment, there may be up to 36 calibration parameters.

[0007] In another embodiment, the calibration is performed dynamically.

[0008] In yet another embodiment, the system further transmits the calibrated accelerometer data, the calibrated magnetometer data, and the calibrated gyroscope data from the sensor fusion module to the AI ​​algorithm module.

[0009] In another embodiment, the AI ​​algorithm module is trained to perform offline corrections using the calibrated accelerometer data, the calibrated magnetometer data, and the calibrated gyroscope data for subsequent iterations. [Brief explanation of the drawings]

[0010] A more complete understanding of the present invention and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings, wherein:

[0011] [Figure 1] FIG. 2 is a diagram of an exemplary embodiment of the hardware of the system of the present invention.

[0012] [Figure 2] 1 is a flowchart illustrating how the system processes raw IMU data to generate an orientation estimate from an exemplary sensor fusion algorithm that uses online calibration.

[0013] [Figure 3] 10 is a chart showing the evaluation of online calibration for IMU accelerometer bias error.

[0014] [Figure 4] 1 is a diagram illustrating the advantages of dynamic IMU calibration over static IMU calibration.

[0015] [Figure 5] 1 is a flowchart illustrating how the system processes raw IMU data to generate an orientation estimate from an exemplary sensor fusion algorithm that uses offline calibration.

[0016] [Figure 6] 10 is a chart showing an evaluation of offline calibration for IMU accelerometer bias error.

[0017] [Figure 7] 1 is a plot of cumulative accelerometer error over time for raw versus calibrated IMU signals.

[0018] [Figure 8] FIG. 1 illustrates how the system of the present invention operates for a wearable device. DETAILED DESCRIPTION OF THE INVENTION

[0019] In describing preferred embodiments of the invention illustrated in the drawings, specific terminology will be used for the sake of clarity. However, it is to be understood that the invention is not intended to be limited to the specific terminology so selected, and that each specific term includes all technical equivalents that similarly operate to accomplish a similar purpose. Several preferred embodiments of the invention have been described for purposes of illustration, and it will be understood that the invention may be embodied in other forms not specifically shown in the drawings.

[0020] Disclosed herein is a medical-grade wearable device comprised of physiological sensors, including PPG ("photoplethysmography"), GSR ("galvanic skin response"), and temperature sensors. In one embodiment, the device is utilized in combination with a custom data pipeline and backend to collect physiological data used to train AI / ML models to perform medically relevant diagnostic inferences and for patient monitoring. Rajant wants the most accurate and precise physiological sensor data the device can provide. To that end, the device is further equipped with a 9-axis IMU ("inertial measurement unit"), including one or more accelerometers, gyroscopes, and magnetometers. In one embodiment, the IMU may also include one or more temperature sensors.

[0021] Among other uses, IMUs are used to reduce motion artifacts in PPG sensor data induced by patient movement. In particular, accelerometer data is utilized to correct the PPG signal by removing motion-induced artifacts. Next, to calculate the device's true linear acceleration, the gravitational component of acceleration must be removed from the signal. This is achieved using a process called sensor fusion. This sensor fusion combines the outputs of the accelerometer, gyroscope, and magnetometer to determine the device's orientation (and orientation) and subtracts the contribution of gravity along each axis. Accelerometers, gyroscopes, and magnetometers produce noisy measurements that must be corrected. Furthermore, measurements are adversely affected if the sensors are not properly calibrated. Calibration is an additional correction step performed to account for sensor bias and sensitivity parameters and offset errors in the raw measurements.

[0022] To perform online calibration, one common approach is to classify each new acquired data point into a group (i.e., label) that indicates the data point's usefulness for calibrating a given sensor. Thus, once enough data points are collected, the sensor can be calibrated. The sensor measurements are then fused to provide a more accurate orientation estimate of the device using one of several estimation methods, such as a Kalman filter or particle filter (or variants thereof). A faster and more accurate estimate can be provided if the search for potential solutions focuses on a more likely solution space, taking into account the estimated trajectory of the device (e.g., a car traveling on a road). However, there is also interest in applications where the trajectory of the device in question may be ad hoc (e.g., human movement). One novel idea is to suggest a more appropriate and informative category / label for each data point, enabled by AI to assist in calibration and sensor fusion. For example, activity recognition (i.e., walking, running, etc.) can inform the calibration / fusion procedure of potential trajectories the device may take, enabling faster and more effective calibration / fusion (both online and offline).

[0023] FIG. 1 illustrates an exemplary embodiment of the system of the present invention. In the exemplary system 100, one or more peripheral devices 110 are connected to one or more computers 120 via a network 130. The peripheral devices / locations 110 may include smartphones, tablets, wearable devices, and other electronic devices known in the art that collect and transmit data over a network. The network 130 may be a wide area network such as the Internet or a local area network such as an intranet. Due to the network 130, the physical locations of the peripheral devices 110 and the computers 120 do not affect the functionality of the hardware and software of the present invention. Both implementations are described herein, and unless otherwise specified, it is contemplated that the peripheral devices 110 and the computers 120 may be in the same physical location or in different physical locations. Communication between the hardware in the system may be accomplished in a number of known ways, such as using network connectivity components such as modems and Ethernet adapters. Both the peripheral devices / locations 110 and the computers 120 include or are connected to communications devices. Communication may occur via industry-standard protocols such as HTTP or HTTPS.

[0024] Each computer 120 comprises a central processing unit 122, a storage medium 124, a user input device 126, and a display 128. Possible computers include commercially available personal computers, open-source computing devices (e.g., Raspberry Pi), commercially available servers, and commercially available portable devices (e.g., smartphones, smartwatches, tablets). In one embodiment, each peripheral 110 and each computer 120 in the system may have software associated with the system installed. In such an embodiment, system data may be stored locally on the networked computer 120 or on one or more remote servers 140 accessible from either the peripheral 110 or the networked computer 120 via the network 130. In an alternative embodiment, the software runs as an application on the peripheral 110 and includes web-based software, iOS-based, and Android-based mobile applications.

[0025] Online IMU Calibration and Sensor Fusion

[0026] Online calibration and sensor fusion utilizes real-time IMU measurements to update the bias and sensitivity parameters of the accelerometer, gyroscope, and magnetometer. Finding an appropriate solution for the parameters significantly reduces errors induced by bias and offset. Figure 2 illustrates the process of generating an accurate orientation estimate from the sensor fusion algorithm using raw IMU data. Bias and sensitivity parameters are estimated for each of the three sensors (accelerometer 202, gyroscope 204, and magnetometer 206). IMU calibration is then preferably performed (208), estimating a total of 18 calibration parameters 210 (bias and sensitivity for the x-, y-, and z-axes for each sensor). The parameters are updated when specific angular velocities and static orientations are measured for the accelerometer 212, gyroscope 214, and magnetometer 216. For example, the IMU rotates through an arc length of 120 degrees and stops at a specific angle relative to gravity.

[0027] Sensor Fusion

[0028] Sensor fusion 218 utilizes calibrated IMU data to estimate the device's orientation in world coordinates. Dynamic calibration makes sensor fusion estimates more accurate than static calibration for IMU bias and sensitivity parameters that vary over time and temperature. Sensor fusion with dynamic calibration achieves faster recovery. Sensor fusion detects inaccuracies in IMU parameter estimates.

[0029] gravity removal

[0030] The quaternion 222 describing the IMU's orientation from the sensor fusion algorithm and the calibrated accelerometer 220 are used to determine the acceleration due to gravity projected onto the x, y, and z axes of the IMU. The gravity projection is removed 224 from the accelerometer readings to provide a linear acceleration estimate 226 in world coordinates.

[0031] Figure 3 illustrates the ability of the calibration procedure to estimate IMU parameters. In this simulated example, the accelerometer static bias offset errors are 0.6, -0.2, and 0.1 mGe for the x, y, and z axes, respectively. The calibration is valid after approximately 24 seconds. After that point, the calibrated accelerometer signal has an error of less than 10 mGe for each axis, while the uncalibrated accelerometer measurements contain errors reflecting the bias.

[0032] Figure 4 illustrates the need for continuous calibration to ensure accurate IMU calibration. In this simulation, the accelerometer error is initially 100 mGe. At 24 seconds, valid calibration parameters are determined to be calculated, and the accelerometer error drops to 10 mGe. At 40 seconds, the IMU parameters change. This could be due to temperature changes or mechanical shock (or other phenomena). If calibration is performed only once and the IMU parameters change at some point in the future, the parameter change will likely produce an unacceptable heading estimate. Conversely, continuous calibration allows for recovery from IMU parameter fluctuations.

[0033] Offline IMU Calibration and Sensor Fusion

[0034] Real-time IMU calibration uses historical data only to update bias and sensitivity parameters. However, given a set of recorded raw IMU data, offline calibration can be performed to obtain more accurate IMU parameters. At any time, past and future IMU samples can be used to determine the optimal set of bias and sensitivity estimates. Similar to online applications, offline calibration allows tracking of dynamic IMU parameters as they change with time and temperature.

[0035] Offline calibration leverages machine learning (ML) and artificial intelligence (AI) algorithms to fully utilize large datasets. The AI ​​algorithms predict when calibration parameters are out of tolerance and quantify the quality of the calibration measurements. Figure 5 characterizes the process of generating accurate heading estimates from sensor fusion algorithms using raw IMU data.

[0036] Offline Storage

[0037] Data can be stored locally on an IMU device, smartphone, edge processing node, etc. until it is ready to be processed. Offline processing can occur on a backend machine or remote node.

[0038] As shown in FIG. 5, bias and sensitivity parameters are estimated for each of the three sensors (accelerometer 502, gyroscope 504, and magnetometer 506) and sent to offline storage 508. From offline storage 508, IMU calibration 516 is preferably performed using readings from sensors 502, 504, and 506, estimating a total of 18 calibration parameters 518 (bias and sensitivity for the x-, y-, and z-axes of each sensor) using raw acceleration data 510, raw gyroscope data 512, and raw magnetometer data 514. In one embodiment, there may be up to 36 calibration parameters. The parameters are updated when specific angular velocities and static orientations are measured for the calibrated accelerometer 520, calibrated gyroscope 522, and calibrated magnetometer 524. For example, the IMU rotates through an arc length of 120 degrees and stops at a specific angle relative to gravity.

[0039] Sensor Fusion

[0040] Sensor fusion 526 utilizes calibrated IMU data to estimate the device's orientation in world coordinates. Dynamic calibration makes sensor fusion estimates more accurate than static calibration for IMU bias and sensitivity parameters that vary over time and temperature. Sensor fusion with dynamic calibration achieves faster recovery. Sensor fusion detects inaccuracies in IMU parameter estimates.

[0041] The sensor fusion 526 data is sent to the AI ​​algorithm module 528, which is used to systematically train an AI / ML model to react to calibration discrepancies and enable faster recovery from erroneous calibration parameter estimates using some of the techniques described above. Suboptimal online calibration is used to provide time-dependent situational awareness, but offline correction may be performed to provide more accurate inference at a later time when time and computational power constraints are relaxed. As shown in Figures 5 and 6, offline calibration incorporates the entire history of IMU samples, demonstrating that all accelerometer errors approach zero. This contrasts with Figure 3, where online calibration error is minimized only after a complete set of calibration measurements has been captured.

[0042] gravity removal

[0043] The quaternion 532 describing the IMU orientation from the sensor fusion algorithm and the calibrated acceleration 530 are used to determine the acceleration due to gravity projected onto the x, y, and z axes of the IMU. The gravity projection is removed 534 from the accelerometer readings to provide a linear acceleration estimate 536 in world coordinates.

[0044] As shown in Figure 7, by evaluating the cumulative accelerometer error, the benefits of offline and online calibration are realized compared to the no-calibration scenario. In the online calibration case, the error accumulates at a significant rate over time until the calibration takes effect at 24 seconds. At that point, the error contribution is significantly reduced. In the offline case, bias and sensitivity errors are minimized across the entire data set because knowledge of both historical and future data is available. The substantial error contributed by the raw IMU signal continues to accumulate at a rate that is unacceptable for maintaining accurate IMU orientation.

[0045] FIG. 8 is a diagram illustrating how the system of the present invention operates for a wearable device in an online / offline hybrid implementation, according to an exemplary embodiment of the present invention. As shown in FIG. 8, the system comprises a wearable 802 and a backend 804. In one embodiment, the wearable 802 is the peripheral device 110 shown in FIG. 1, and the backend 140 may be one or more computers 120 as shown in FIG. 1. In one embodiment, the backend 140 may also be a smartphone or tablet. The wearable 110 comprises one or more IMU sensors 806, a data transfer service 808, an online calibration and sensor fusion (CNSF) 810, an external calibration updater 812, and a Bluetooth module 814. The data transfer service 808 and the online CNSF 810 have their own local stores 816 and 818, respectively.

[0046] During operation, the IMU sensor 806 collects various sensor data and transmits the raw IMU data to the online CNSF 810 and the data transfer service 808. The online CNSF 810 is primarily responsible for performing calculations to convert the raw IMU data into annotated, calibrated IMU data, while also calculating metrics and quality calibration values. Recalibration of the raw IMU data may be necessary due to factors such as changes in temperature, mechanical shock, and / or time. Other reasons for recalibration will be readily apparent to those skilled in the art. The online CNSF 810 stores the annotated, calibrated IMU data and calibration values ​​in a local store. The online CNSF 810 then transmits the annotated, calibrated IMU data and calibration values ​​to the data transfer service 808. The data transfer service 808 stores the annotated, calibrated IMU data, calibration values, and raw IMU data in a local store 818. The online CNSF 810 is also connected to an external calibration updater 812, which provides real-time external calibration data to the online CNSF 810. The external calibration data can be used by the online CNSF810 to calculate calibrated IMU data and calibration metrics and quality values.

[0047] The data transfer service 808 and the external calibration updater 812 are each connected to a Bluetooth module 814. The data transfer service 808 and the external calibration updater 812 send and receive one or more of raw IMU data, calibration data, annotated calibrated IMU data, and calibration values ​​with metrics and quality to and from the Bluetooth module 814. The Bluetooth module 814 can communicate with an edge app 820, which is an application that connects the wearable 110 to the backend 140. The wearable 802 and the backend 804 are in continuous bidirectional communication via the edge app 820.

[0048] The backend 804 is comprised of a recalibration service 824, a DB manager 826, an offline CNSF 828, and a remote store 830. The offline CNSF 828 receives calibration data from the edge app and compares it with the calibration data received from the DB manager 826. The DB manager 826 queries the remote store 830 for previously calculated historical data received from the wearable 802. If the calibration data received from the wearable 802 is determined to be less accurate than the historical data, the offline CNSF 828 signals the recalibration service 824 to request that the external calibration updater 812 perform an updated calibration and analysis of the IMU data at the wearable 802. The recalibration request is sent to the external calibration updater 812 via the edge app 820 and the Bluetooth module 814.

[0049] The foregoing description and drawings should be considered merely as illustrative of the principles of the invention. The invention is not intended to be limited to the preferred embodiment, but may be embodied in various ways apparent to those skilled in the art. Numerous applications of the present invention will readily occur to those skilled in the art. Therefore, it is not desired to limit the invention to the particular examples disclosed or to the exact construction and operation shown and described. Rather, all suitable modifications and equivalents should fall within the scope of the invention. All references cited herein are incorporated by reference.

Claims

1. 1. A computer-implemented method for sensor calibration, comprising: receiving raw accelerometer data, raw gyroscope data, and raw magnetometer data; performing a sensor calibration on an inertial measurement unit using the raw accelerometer data, the raw gyroscope data, and the raw magnetometer data; outputting a plurality of calibration parameters from the inertial measurement unit; sending the plurality of calibration parameters to a sensor fusion module that calculates calibrated accelerometer data and orientation; and transmitting the calibrated accelerometer data and the orientation to a gravity cancellation module that calculates corrected linear acceleration data.

2. The method of claim 1 , wherein there are a maximum of 36 calibration parameters.

3. The method of claim 1 , wherein the calibration is performed dynamically.

4. 10. The method of claim 1, further comprising transmitting calibrated accelerometer data, calibrated magnetometer data, and calibrated gyroscope data from the sensor fusion module to an AI algorithm module.

5. 5. The method of claim 4, wherein the AI ​​algorithm module is trained for subsequent iterations to perform offline corrections using the calibrated accelerometer data, the calibrated magnetometer data, and the calibrated gyroscope data.

6. comparing a plurality of previously calculated calibration parameters from the inertial measurement unit with the plurality of calibration parameters; 10. The method of claim 1, further comprising: updating the plurality of calibration parameters with an external calibration updater located on the wearable device before transmitting the plurality of calibration parameters to the sensor fusion module.

7. The method of claim 1 , wherein the sensor fusion module is used to determine acceleration due to gravity projected onto x, y, and z axes.

8. The method of claim 4 , wherein data from the sensor fusion module is used to train the AI ​​algorithm module.

9. The method of claim 1 , wherein historical calibration data is used to update bias and sensitivity parameters.

10. The method of claim 1 , further comprising transmitting the corrected linear acceleration data to a Bluetooth module for delivery to a second computer.

11. A non-transitory computer storage medium storing a program that causes a computer to execute a process, the process comprising: receiving raw accelerometer data, raw gyroscope data, and raw magnetometer data; performing a sensor calibration on an inertial measurement unit using the raw accelerometer data, the raw gyroscope data, and the raw magnetometer data; outputting a plurality of calibration parameters from the inertial measurement unit; sending the plurality of calibration parameters to a sensor fusion module that calculates calibrated accelerometer data and orientation; and transmitting the calibrated accelerometer data and the orientation to a gravity cancellation module that calculates corrected linear acceleration data.

12. The non-transitory computer medium of claim 11 , wherein there are up to 36 calibration parameters.

13. The non-transitory computer medium of claim 11 , wherein the calibration is performed dynamically.

14. 12. The non-transitory computer medium of claim 11, further comprising transmitting the calibrated accelerometer data, the calibrated magnetometer data, and the calibrated gyroscope data from the sensor fusion module to an AI algorithm module.

15. 15. The non-transitory computer medium of claim 14, wherein the AI ​​algorithm module is trained for subsequent iterations to perform offline corrections using the calibrated accelerometer data, the calibrated magnetometer data, and the calibrated gyroscope data.

16. The program further causes the computer to comparing a plurality of previously calculated calibration parameters from the inertial measurement unit with the plurality of calibration parameters; and updating the plurality of calibration parameters with an external calibration updater located on the wearable device before transmitting the plurality of calibration parameters to the sensor fusion module.

17. The non-transitory computer medium of claim 11 , wherein the sensor fusion module is used to determine acceleration due to gravity projected onto x, y, and z axes.

18. The non-transitory computer medium of claim 11 , wherein data from the sensor fusion module is used to train the AI ​​algorithm module.

19. The non-transitory computer medium of claim 11 , wherein historical calibration data is used to update bias and sensitivity parameters.

20. 12. The non-transitory computer medium of claim 11, wherein the corrected linear acceleration data is transmitted to a Bluetooth module for delivery to a second computer.

Images