System and method for correlating navigation data and sensor data

Abstract

A system and method for correlating navigation data, temporal data, and sensor data in real time including receiving sensor frame data at a first time, accessing navigation data entries stored in a retrievable location at second times, selecting a subset of the navigation data entries stored before the first time, selecting the navigation data entry from the subset that has a second time value closest to the first time, and interleaving execution of operations when a processor executing instructions the perform the operations is idle. The operations include storing the selected navigation data entry with the sensor frame data as correlated data, creating a file to save the correlated data when one or more parameters has reached a pre-selected threshold, retrieving the correlated data, and saving the correlated data locally in the file.

Description

SYSTEM AND METHOD FOR CORRELATING NAVIGATION DATA AND SENSORDATACROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 674,459, filed on July 23, 2024, which is hereby incorporated by reference in its entirety.FIELD OF THE DISCLOSURE

[0002] The present disclosure is directed to correlating sensor, temporal, and navigation data in real time.BACKGROUND

[0003] Recording, processing, and displaying aircraft data in real-time is a monumental task especially when correlating video data with timing data. For a flight testing or production system, recording good data without gaps is crucial for certification, event recreation, and trade-study.SUMMARY

[0004] Systems and methods in accordance with embodiments of the present disclosure provides a data-saving mechanism that correlates temporal data with aircraft navigation data and with data collected in association with the navigation data, such as, for example, but not limited to, sensor data. In some configurations, the navigation data are correlated to video data from multiple cameras of varying frame rate and size. The collected data can be provided by, for example, but not limited to, universal serial bus (USB) devices, gigabit Ethernet communications protocol (GigE) devices, and Camera Link protocol devices. In some configurations, the aircraft navigation data include, but are not limited to including, inertial measurement unit (IMU) data, global positioning system (GPS) data, and embedded GPS / IMU (EGI) data. Applications that record large quantities of data in real time, for example, flight test applications, vehicle applications in which multiple cameras are used, automated aerial and refueling simulations, and applications processing satellite data can be augmented by a system and method in accordance with the present disclosure. An asynchronous data caching mechanism, a dynamic queue data structure, multi-data source filtering, and an atomic write call to the caching mechanism combine to store the correlated data. A file snippet manager accommodates power failures. The collected data can bereceived in any format, for example, but not limited to, long-wave infrared (LWIR) data, light detection and ranging (LiDAR) data, laser detection and ranging (LADAR), and visible data. In some configurations, to accommodate any data format, the frame length is specified. In some configurations, the frame length is specified in bytes. In some configurations, the output data can include, but are not limited to including, GPS and time-synchronized data from snippets.

[0005] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions. One general aspect includes a method for correlating and saving sensor frame data with navigation data and temporal data in real time. The method includes receiving the sensor frame data at a first time. The method also includes accessing, from at least one retrievable location, at least one navigation data entry associated with at least one second time. The method also includes selecting a subset of the at least one navigation data entry associated with the at least one second time that occurred before the first time. The method also includes selecting one entry from the subset based on a proximity of the second time associated with the selected one to the first time. The method also includes interleaving execution on a processor of instructions performing operations including associating the selected one entry with the sensor frame data as correlated data, creating a file to save the correlated data when one or more parameters has reached a preselected threshold, and retrieving the file storing the correlated data. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0006] Implementations may include one or more of the following features. The method may include interleaving the execution on the processor of the instructions performing the operations at times when a processor is idle. The sensor frame data are processed in one or more sensors threads simultaneously. The sensor frame data may include data from one or more cameras, where the cameras provide the sensor frame data at different frame rates and in different frame sizes. Receiving the sensor frame data may include interfacing with one or more of a universal serial bus (USB) device, a gigabit ethemet communications protocol (GigE) device, or a camera link protocol device. The at least one navigation data entry mayinclude one or more of global positioning system (GPS), inertial measurement unit (IMU), or embedded GPS / IMU (EGI) data in any format. The sensor frame data may include one or more of long-wave infrared (LWIR), light detection and ranging (LiDAR), laser detection and ranging (LADAR), or visible data. The retrievable location may include one or more of a queue, a network stream, or a disk. The method may include saving the correlated data locally in the file. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0007] One general aspect includes a computer system for correlating and saving sensor frame data with navigation data and temporal data in real time. The computer system also includes a hardware processor, and a non-volatile storage medium storing instructions that when executed by the hardware processor perform operations may include receiving the sensor frame data at a first time, accessing, from at least one retrievable location, at least one navigation data entry associated with at least one a second time, selecting a subset of the at least one navigation data entry associated with the at least one second time that occurred before the first time, selecting one entry from the subset based on a proximity of the second time associated with the selected one entry to the first time, and interleaving execution on the hardware processor of the operations including associating the selected one entry with the sensor frame data as correlated data, creating a file to save the correlated data when one or more parameters has reached a pre-selected threshold, and retrieving the file storing the correlated data. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0008] Implementations may include one or more of the following features. The computer system where the operations further may include interleaving the execution on the hardware processor of the instructions performing the operations at times when the hardware processor is idle. The sensor frame data are processed in one or more sensors threads simultaneously. The sensor frame data may include data from one or more cameras, where the cameras provide the sensor frame data at different frame rates and in different frame sizes. Receiving the sensor frame data may include interfacing with one or more of a USB device, a GigE device, or a camera link protocol device. The at least one navigation data entry may include one or more of GPS, IMU, or EGI data in any format. The sensor frame data may include one or more of LWIR, LiDAR, LADAR, or visible data. The retrievable location may include one or more of a queue, a network stream, or a disk. The computer system may include saving thecorrelated data locally in the file. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0009] One general aspect includes a computer program product for correlating and saving sensor frame data with navigation data and temporal data in real time. The computer program product also includes receiving the sensor frame data at a first time. The product also includes accessing, from at least one retrievable location, at least one navigation data entry associated with at least one second time. The product also includes selecting a subset of the at least one navigation data entry associated with the at least one second time that occurred before the first time. The product also includes selecting one entry from the subset based on a proximity of the at least one second time associated with the one entry to the first time. The product also includes interleaving execution on the computing device of the instructions performing the operations including associating the selected one entry with the sensor frame data as correlated data, creating a file to save the correlated data when one or more parameters has reached a pre-selected threshold, and retrieving the file storing the correlated data. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0010] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present teachings, as claimed.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate aspects of the present teachings and together with the description, serve to explain the principles of the present teachings.

[0012] FIG. 1 is a flowchart of a sensor management in accordance with embodiments of the present disclosure;

[0013] FIG. 2A-2B is a flowchart of sensor data processing in accordance with embodiments of the present disclosure;

[0014] FIG. 3A-3B is a flowchart of data processing of multiple sensor types in accordance with embodiments of the present disclosure;

[0015] FIG. 4 is a flowchart of the file data storage process in accordance with embodiments of the present disclosure;

[0016] FIG. 5A-5B is a flowchart of the reception, correlation, and storage of data in accordance with embodiments of the present disclosure;

[0017] FIG. 6 is a flowchart of the correlation process in accordance with embodiments of the present disclosure; and

[0018] FIG. 7 is a flowchart of a method for correlating sensor data with navigation data in accordance with embodiments of the present disclosure.

[0019] It should be noted that some details of the figures have been simplified and are drawn to facilitate understanding rather than to maintain strict structural accuracy, detail, and scale.DESCRIPTION

[0020] Reference will now be made in detail to the present teachings, examples of which are illustrated in the accompanying drawings. In the drawings, like reference numerals have been used throughout to designate identical elements. In the following description, reference is made to the accompanying drawings that form a part thereof, and in which is shown by way of illustration specific examples of practicing the present teachings. The following description is, therefore, merely exemplary.

[0021] The present disclosure is directed to a system and method for collecting and correlating time, sensor, and navigation data, synchronizing the data, and saving the data in real time. The system and method receive navigation, time, and sensor data as input and process the data as follows. A write object is instantiated with, for example, but not limited to, a snippet size, a navigation provider shared object, and other tunable parameters. When a desired data frame is ready to be saved, a byte array of the data frame is saved to a queue. When data are available on the queue, the data are dequeued and saved, for example, locally. When data-related parameters have met pre-selected thresholds, such as, for example, but not limited to, a threshold data frame count has been reached, the data are saved into a file and a new file is created in between saving the byte array frame on the queue. In some configurations, when there is no gap between queue saves, the queue saves are interrupted and the data are saved.

[0022] Referring now to FIG. 1, one step in correlating temporal, navigation, and sensor data is managing sensors. In some configurations, when cameras are the exemplary sensors, camera parameters are initialized 107, and when 109 the camera status and configurationindicate that the camera is ready receive data, frame acquisition handlers 111 create camera threads 101. The camera thread 101 receives camera frame data 103 from the configured camera and provides the frame data 103 to a camera frame object 105. In general, sensors are initialized and their status monitored with software / hardware interfaces that are dedicated to sensor interface management, thus freeing other data processing functions, such as, for example, but not limited to, recording data, streaming data, and processing data.

[0023] Referring now to FIGs. 2A-2B, another step in correlating time, navigation, and sensor data is processing the sensor data, specifically error checking and command / control of the sensor. The exemplary sensor shown in FIG. 2 A is an optical sensor 201. Other sensor data can be received and processed in approximately the same way as the optical sensor 201. The system of the present teachings is not limited to receiving optical sensor data. In some configurations, pulse per second (PPS) data are received by an optical sensor 201 from a PPS provider 203. Raw frame data are acquired 207 from the optical sensor 201 along with telemetry data, and the frame is timestamped 209 and moved to a frame data queue 205. Processing of the frame continues by determining when the horizontal pixel count is incremented 211, the vertical pixel count is incremented 213, and the frame count is incremented 215, when the unhandled frame meets pre-selected criteria, the frame is ready 221 for further processing. When horizontal pixel count is not incremented 211, vertical pixel count is not incremented 213, or the frame count is not incremented 215, a warning state is raised in which a false update 217 of the frame has occurred. When the unhandled frame does not pass the frame check 222, a warning state is raised in which a frame overrun 219 has occurred. When 225 the optical sensor is in an error state, an error state alert 223 is raised. When 229 a frame grabber is in an error state, an error state alert 227 is raised. When 233 there are optical sensor commands from the optical sensor command queue 231 to be executed, information is provided to the optical sensor by means of, for example, but not limited to, a sensor applications programming interface (API). When 235 the frame passes pre-selected criteria, another frame is acquired 207 and frame processing continues with timestamping 209 the frame, as described herein.

[0024] Referring now to FIGs. 3A-3B, broadening of the correlating process includes processing data from multiple sensor types. Whenever sensor data are ready, the next sensor data available can be queried through the sensor management system of the present disclosure. The sensor management system cycles through 315 a sensor list 317 and, if 319 the sensor is ready, identifies 321 the newest data frame available and stores 311 the latest frame. The entity querying for data is provided with data from the sensor that has availabledata. When new sensors are registered, the sensor and sensor data objects are initialized based on parameters from, for example, but not limited to, a configuration file 302. The health and status of the sensor are monitored, providing either feedback to a sensor health monitoring system and / or re-initializing the sensor and sensor processes whenever there is a fault or bad data. In some configurations, a memory buffer for the newest sensor data is provided. Sensor objects are tracked to determine when data are available from a sensor, and this information is kept in a list that is queried for data that are available. The data are accessed using a query of the sensor list 317 to check for new and unhandled data. A copy of data can be accessed, along with the health and status of the sensor by, for example, multiple callers when there is, for example, a callback function to check if the frame has been handled by the entity accessing the data. In some configurations, file snippets are created to ensure data integrity because the snippets can be tested for whether or not a sensor is progressively getting fresh data. If snippets are created, and if there is an issue with the Possible sensors include, but are not limited to, LWIRs 301 / 303, visible sensors 305, and barometric pressure sensors 307. A sensor data manager 309 can receive data from the sensors.

[0025] Referring now to FIG. 4, sensor data entering the system through various threads are stored in files that are managed by a file manager that executes in a separate thread from the data receiver threads. In some configurations, a file manager is initialized 401 , a file manager handler is created 403, and navigation and frame file objects 405 are created with parameters based on a configuration file. The file manager checks that a specified directory exists and is available to write to. If so, the file manager launches a thread for each sensor as described herein. In each thread, the file manager checks the disk utilization of the directory to ensure that there is space available. If there is no space available, the file manager dynamically points the thread to a contingent directory, that had been specified as a parameter, to resume save operations. The file manager checks the directory structure and builds a new directory structure if a suitable one does not exist. In each thread, the file manager checks 411 for sensor data from a frame data queue 417 in shared memory. The file manager queries data in a navigation data queue 419 to correlate the navigation data with the frame data. The frame data to be saved are retrieved, saved 413, and cleared from the frame data queue 417. The correlated data 409 are written 415 to the file 407. In each thread, if there are no data to write from the frame data queue, the file manager checks 421 that the current number of data writes is less than or equal to a threshold specified by a configuration file. If this is true, the existing file is closed 423, the navigation and frame file object 427 is stored 431, a new file is opened 425 with a new filename, a new navigation and frame file object 429, and a new storagedevice 433, and all of the counters are reset. The file manager checks for file errors and checks the target disk to ensure that there is enough storage space. If the disk and all of the contingency disks are full, an error is raised and the frame data queue 417 is cleared periodically. The file manager periodically checks the navigation data queue 419 to clear data that are associated with the sensor frames but are no longer needed.

[0026] Referring now to FIGs. 5A-5B, a block diagram 500 of a process flow for correlating sensor data recording with navigation data is shown. Components of a system in accordance with the present disclosure can include, but are not limited to including, a navigation data manager 502, a sensor manager 504, and a file write manager 506. A navigation device 510 is discovered and configured, and, when 512 alignment of the device is reached (i.e. when the vertical axis is found for example by aligning the device’s vertical ring laser gyro with the earth’s gravity vector, and by finding inertial north), data are received by the navigation data manager 502 from a navigation device 514 such as, for example, but not limited to, Falcn EGI (UDP-IP bus). Data are received from the navigation device through a user datagram protocol / intemet protocol (UDP / IP) to Ethernet interface onto a plurality of Ethernet channels. Data output from navigation data manager 502 is provided to SQL database 516 to ensure that the data are compliant with an SQL-accessible format. Data output from navigation data manager 502 is also provided to a shared navigation pointer 518 with a mutually exclusive flag (mutex set) which protects the navigation data from concurrent access, as the navigation data objects 520 are stored within a shared navigation pointer 518 area. The navigation data objects 520 are correlated 522 with sensor data objects, each of which can include any data type, and the correlated data are provided to the file write manager 506. With regard to the sensor manager 504, when a new sensor is initiated at 524, a connection attempt between the sensor 526 and the sensor manager 504. If 528 the sensor 526 is successfully connected, and if 530 sensor data are available, the sensor data are stored in a sensor data object 532. If no sensor data are available, then the process goes back to 528 to wait for a successful sensor connection to be established. The sensor data object 532 is then correlated with the navigation data object 520.

[0001] With regard to file write manager 506, when a new storage means 534 such as a file is discovered, a file path is created 536 and a sensor data queue 540 is initialized 538. Correlated data are stored on the sensor data queue 540 as they arrive from the correlation function. When 542 data are available on the sensor data queue 540, the data are moved to a binary frame 544, which includes the navigation data object 546 and the data 548 from thesensor data queue. The data from the binary frame 544 are stored in a sensor binary file 550. The file write manager 506 maintains a sensor counter.

[0027] Referring now to FIG. 6, further with respect to correlating sensor data recording with navigation data, shared data structures, such as the shared sensor frame object and the shared navigation data object discussed herein, protect the data from improper access. The sensors update the shared memory objects, and the correlation process accesses the shared memory objects when the sensor process is complete. In some configurations, a semaphore ensures that multiple threads can access the shared memory by managing access by threads to the data in the shared memory. The data correlation process can thus takes place without data corruption. The correlation process starts when a new sensor frame 601 at a sensor frame time and data from the navigation data queue 603 are available to the correlator. The correlator accesses 605 data from the navigation data queue 603 that are collected before the sensor frame time, and then selects 607 the navigation queue data from the accessed subset that are queued closest in time to the first sensor frame time. The sensor frame and the selected navigation queue data are stacked 609 together and stored.

[0028] Referring now to FIG. 7, method 700 for correlating and saving sensor frame data with navigation data and temporal data in real time can include, but is not limited to including, receiving 702 the sensor frame data at a first time, accessing 704 navigation data entries associated with second times, selecting 706 a subset of the navigation data entries associated with the second times that occurred before the first time, selecting 708 one navigation data entry from the subset based on a proximity of the second time associated with the one of the subset to the first time, and at times when a processor is idle, interleaving 710 execution on the processor of instructions performing operations including associating the selected navigation data entry with the sensor frame data as correlated data, creating a file to save the correlated data when one or more parameters has reached a pre-selected threshold, and retrieving the file storing the correlated data.

[0029] Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the disclosure are approximations, the numerical values set forth in the specific examples are reported as precisely as possible. Any numerical value, however, inherently contains certain errors necessarily resulting from the standard deviation found in their respective testing measurements. Moreover, all ranges disclosed herein are to be understood to encompass any and all sub-ranges subsumed therein.

[0030] While the present teachings have been illustrated with respect to one or more implementations, alterations and / or modifications can be made to the illustrated exampleswithout departing from the spirit and scope of the appended claims. In addition, while a particular feature of the present teachings may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for any given or particular function. As used herein, the terms “a”, “an”, and “the” may refer to one or more elements or parts of elements. As used herein, the terms “first” and “second” may refer to two different elements or parts of elements. As used herein, the term “at least one of A and B” with respect to a listing of items such as, for example, A and B, means A alone, B alone, or A and B. Those skilled in the art will recognize that these and other variations are possible. Furthermore, to the extent that the terms “including,” “includes,” “having,” “has,” “with,” or variants thereof are used in either the detailed description and the claims, such terms are intended to be inclusive in a manner similar to the term “comprising.” Further, in the discussion and claims herein, the term “about” indicates that the value listed may be somewhat altered, as long as the alteration does not result in nonconformance of the process or structure to the intended purpose described herein. Finally, “exemplary” indicates the description is used as an example, rather than implying that it is an ideal.

[0031] It will be appreciated that variants of the above-disclosed and other features and functions, or alternatives thereof, may be combined into many other different systems or applications. Various presently unforeseen or unanticipated alternatives, modifications, variations, or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompasses by the following claims.

[0002] The examples set forth herein represent the necessary information to enable those skilled in the art to practice the embodiments and illustrate the best mode of practicing the embodiments. Upon reading the description in light of the accompanying drawing figures, those skilled in the art will understand the concepts of the disclosure and will recognize applications of these concepts not particularly addressed herein. It should be understood that these concepts and applications fall within the scope of the disclosure and the accompanying claims.

[0003] It will be understood that when an element is referred to as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being “directly connected” or “directly coupled” to another element, there are no intervening elements present.

[0004] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Claims

CLAIMS1. A method for correlating and saving sensor frame data with navigation data and temporal data in real time comprising: receiving the sensor frame data at a first time; accessing, from at least one retrievable location, at least one navigation data entry associated with at least one second time; selecting a subset of the at least one navigation data entry associated with the at least one second time that occurred before the first time; selecting one entry from the subset based on a proximity of the second time associated with the selected one to the first time; and interleaving execution on a processor of instructions performing operations including: associating the selected one entry with the sensor frame data as correlated data; creating a file to save the correlated data when one or more parameters has reached a pre-selected threshold; and retrieving the file storing the correlated data.

2. The method of claim 1, further comprising: interleaving the execution on the processor of the instructions performing the operations at times when a processor is idle.

3. The method of claim 1, further comprising: receiving the sensor frame data from one or more sensors, wherein the sensor frame data are processed in one or more sensors threads simultaneously.

4. The method of claim 1 , wherein the sensor frame data comprise: data from one or more cameras, wherein the cameras provide the sensor frame data at different frame rates and in different frame sizes.

5. The method of claim 1, wherein receiving the sensor frame data comprises:interfacing with one or more of a universal serial bus (USB) device, a gigabit Ethernet communications protocol (GigE) device, or a Camera Link protocol device.

6. The method of claim 1, wherein the at least one navigation data entry comprises: one or more of global positioning system (GPS), inertial measurement unit (IMU), or embedded GPS / IMU (EGI) data in any format.

7. The method of claim 1 , wherein the sensor frame data comprise: one or more of long-wave infrared (LWIR), light detection and ranging (LiDAR), laser detection and ranging (LADAR), or visible data.

8. The method of claim 1, wherein the retrievable location comprises: one or more of a queue, a network stream, or a disk.

9. The method of claim 1, further comprising: saving the correlated data locally in the file.

10. A computer system for correlating and saving sensor frame data with navigation data and temporal data in real time comprising: a hardware processor; and a non-volatile storage medium storing instructions that when executed by the hardware processor perform operations comprising: receiving the sensor frame data at a first time; accessing, from at least one retrievable location, at least one navigation data entry associated with at least one a second time; selecting a subset of the at least one navigation data entry associated with the at least one second time that occurred before the first time; selecting one entry from the subset based on a proximity of the second time associated with the selected one entry to the first time; and interleaving execution on the hardware processor of the operations including: associating the selected one entry with the sensor frame data as correlated data; creating a file to save the correlated data when one or more parameters has reached a pre-selected threshold; andretrieving the file storing the correlated data.

11. The computer system of claim 10, wherein the operations further comprise: interleaving the execution on the hardware processor of the instructions performing the operations at times when the hardware processor is idle.

12. The computer system of claim 10, further comprising: receiving the sensor frame data from one or more sensors, wherein the sensor frame data are processed in one or more sensors threads simultaneously.

13. The computer system of claim 10, wherein the sensor frame data comprise: data from one or more cameras, wherein the cameras provide the sensor frame data at different frame rates and in different frame sizes.

14. The computer system of claim 10, wherein receiving the sensor frame data comprises: interfacing with one or more of a USB device, a GigE device, or a Camera Link protocol device.

15. The computer system of claim 10, wherein the at least one navigation data entry comprises: one or more of GPS, IMU, or EGI data in any format.

16. The computer system of claim 10, wherein the sensor frame data comprise: one or more of LWIR, LiDAR, LADAR, or visible data.

17. The computer system of claim 10, wherein the retrievable location comprises: one or more of a queue, a network stream, or a disk.

18. The computer system of claim 10, further comprising: saving the correlated data locally in the file.

19. A computer program product for correlating and saving sensor frame data with navigationdata and temporal data in real time, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to perform operations comprising: receiving the sensor frame data at a first time; accessing, from at least one retrievable location, at least one navigation data entry associated with at least one second time; selecting a subset of the at least one navigation data entry associated with the at least one second time that occurred before the first time; selecting one entry from the subset based on a proximity of the at least one second time associated with the one entry to the first time; and interleaving execution on the computing device of the instructions performing the operations including: associating the selected one entry with the sensor frame data as correlated data; creating a file to save the correlated data when one or more parameters has reached a pre-selected threshold; and retrieving the file storing the correlated data.

20. The computer program product of claim 19, wherein the operations further comprise: interleaving the execution on the computing device of the instructions performing the operations at times when the computing device is idle.

Images