Safety critical data recorder

Abstract

A method to record low-latency data in real-time with a single tread onto a storge medium is disclosed. The method includes instantiating, by a single thread of a scheduler of a real-time operating system executed by a hardware processor, a write command to write data from a data queue to a storage medium; determining, by the single thread, that the data is in the data queue; obtaining, by the single thread, the data from the data queue; and saving, by the single thread and the write command, the data that is obtained from the data queue to the storage medium as an output file.

Description

Attorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP Safety Critical Data Recorder Cross Reference to Related Applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 674,538 filed on July 23, 2024, the contents of which are hereby incorporated by reference in its entirety. Field

[0002] The present disclosure relates to a safety critical data recorder. Background

[0003] Probe and drogue aerial refueling currently relies on a skilled human pilot to perform the complicated maneuvering needed to plug the drogue and maintain formation flight with a tanker. However, future autonomous air vehicles make this level of high-fidelity pilot- control difficult: an autonomous aerial refueling receive (A2R2) capability will be required for these future aircraft to take advantage of the extended endurance that aerial refueling allows.

[0004] Recording, processing, and displaying aircraft data in real-time is a monumental task especially whenever you add the complexity of video data that is correlated with timing data. For a flight testing or production system, recording good data without gaps is crucial for certification, event re-creation, and trade-study. Summary

[0005] According to examples of the present disclosure, systems, devices, and methods for recording low-latency data in real-time with a single thread that writes the data to disk in snippets ensuring data integrity is disclosed. In some examples, the recording mechanism is software based and is not reliant on any specific hardware to function. The disclosed systems, devices, and methods can regulate processing over a single frame and keep up where conventional data recorders will use multiple threads to perform the same functionality. This provides for high efficiency in its processing and parsing of the data to be saved. The disclosed systems, devices, and methods can be used to record data locally on a real-time operating system (RTOS) and / or used by those seeking to increase the performance of their current system. Existing solutions do not exist for a real-time operating system. This data is generally re-published to another computer to handle the recording. According to examples of the present disclosure, the disclosed systems, devices, and method work without another computer handing that process. Input from the present systems, devices, and methods can comprise data to beAttorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP saved and output from the present systems, devices, and methods can comprise recorded snippets of data spanning cycles of a single thread for maximum utilization and load balancing.

[0006] According to examples of the present disclosure, the disclosed systems, devices, and methods can be configured to perform a method that comprises the following features: 1) instantiate writeObject with snippet size and tunable parameters that include maximum frame period; 2) Whenever a desired data frame is ready to be saved, transfer the byte array to a queue; 3) whenever write byte data is available on the queue, the caching mechanism will pull it off the queue, save to local disk; 4) if parameters of the output file such as data frame count have been reached, the cacher will save the file and create a new one in between saving frames (if queue lags behind, the cacher will prioritize catching up without waiting if specified); and 5) the cacher will have an added function to determine if there is enough time to perform specified tasks such as creating a new file or appending an existing one, if there is not enough time between the cacher and other tasks, the cacher will wait for the next interval to fan out processing whenever there is less work being accomplished.

[0007] According to examples of the present disclosure, a method to record low-latency data in real-time with a single tread onto a storge medium is disclosed. The method comprises instantiating, by a single thread of a scheduler of a real-time operating system executed by a hardware processor, a write command to write data from a data queue to a storage medium; determining, by the single thread, that the data is in the data queue; obtaining, by the single thread, the data from the data queue; and saving, by the single thread and the write command, the data that is obtained from the data queue to the storage medium as an output file.

[0008] Various additional features can be included in the method including one or more of the following features. The data comprises time-synchronized image data and / or video data and associated geographic data and / or navigation-related data. The image data and / or the video data comprises infrared image or infrared video obtained from one or more infrared spectrum capable cameras and / or one or more visible spectrum capable cameras. The geographic data and / or navigation-related data are obtained from an embedded Global Positioning System (GPS) + Inertial Navigation System (INS) Embedded GPS (EGI) (“embedded GPS + INS (EGI)”) system. The write command is configured to define a memory size for the data. The write command is configured to define one or more tunable parameters. The one or more tunable parameters comprise a maximum frame period. The data is configured as a data frame. The method further comprises determining that the data formatted as a data frame is ready to be saved and transferring the data frame as a data byte array to the data queue. The method further comprises prior to saving the data, determining that a data frame count in the output fileAttorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP has met a predetermined threshold, wherein the data is saved based on the data frame count meeting the predetermined threshold. The method further comprises creating a new output file once the data is saved.

[0009] According to examples of the present disclosure, a computer system is disclosed that comprises a hardware processor and a non-volatile storage medium storing instructions that when executed by the hardware processor performs a method to record low-latency data in real-time with a single tread onto a storge medium, the method comprising: instantiating, by a single thread of a scheduler of a real-time operating system executed by a hardware processor, a write command to write data from a data queue to a storage medium; determining, by the single thread, that the data is in the data queue; obtaining, by the single thread, the data from the data queue; and saving, by the single thread and the write command, the data that is obtained from the data queue to the storage medium as an output file.

[0010] Various additional features can be included in the computer system including one or more of the following features. The data comprises time-synchronized image data and / or video data and associated geographic data and / or navigation-related data. The image data and / or the video data comprises infrared image or infrared video obtained from one or more infrared spectrum capable cameras and / or one or more visible spectrum capable cameras. The geographic data and / or navigation-related data are obtained from an embedded Global Positioning System (GPS) + Inertial Navigation System (INS) Embedded GPS (EGI) (“embedded GPS + INS (EGI)”) system. The write command is configured to define a memory size for the data. The write command is configured to define one or more tunable parameters. The one or more tunable parameters comprise a maximum frame period. The data is configured as a data frame. The method further comprises determining that the data formatted as a data frame is ready to be saved and transferring the data frame as a data byte array to the data queue. The method further comprises prior to saving the data, determining that a data frame count in the output file has met a predetermined threshold, wherein the data is saved based on the data frame count meeting the predetermined threshold. The method further comprises creating a new output file once the data is saved. Brief Description of the Figures

[0011] FIG. 1 shows a block diagram of a hardware configuration of a system for data recording according to examples of the present disclosure.

[0012] FIG.2 shows a block diagram of a process flow between a sensor factory and a data recorder for the system of FIG.1 according to examples of the present disclosure.Attorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP

[0013] FIG. 3 shows a block diagram of a process flow between a sensor thread from a sensor factory and a sensor recorder thread from a data recorder for the system of FIG. 1 according to examples of the present disclosure.

[0014] FIG.4A and FIG.4B show a block diagram of a process flow between a navigation data manager, a sensor manager, and a file write manager for the system of FIG. 1 according to examples of the present disclosure.

[0015] FIG.5 shows a software stack for a central processing unit (CPU) of a data recorder according to examples of the present disclosure.

[0016] FIG.6 illustrates an example of such a computing system, in accordance with some examples.

[0017] FIG.7 shows a flowchart depicting a method to record low-latency data in real-time with a single tread onto a storge medium according to examples of the present disclosure. Detailed Description

[0018] FIG. 1 shows block diagram 100 of a hardware configuration of a system for data recording according to examples of the present disclosure. Data recorder 102 can be configured to interface with and collect data from one or more sensors. The one or more sensors can take a variety of forms including video cameras and position / telemetry-type sensors.

[0019] As shown in FIG. 1, camera link card 104 is configured to communicate with one or more cameras, such as first camera 106 and second camera 108. For example, first camera 106 and second camera 108 can be different types of infrared cameras. In one non-limiting example, first camera 106 can be an ATOM®1024 camera core, which is a high-resolution uncooled thermal imager for long-wave infrared (LWIR) applications. Second camera 108 can be a Teledyne forward looking infrared (FLIR) Boson 640 LWIR thermal camera core. Camera link card 104 can communicate with first camera 106 over a physical connector using a CL port on first camera 106 and a CLA port on camera link card 104. Camera link card 104 can communicate with second camera 108 over a physical connector using a CL port on second camera 108 and a CLB port on camera link card 104. In some examples, more than two cameras can be coupled with camera link card 104.

[0020] Data recorder 102 communicates with camera link card 104 over a communication interface, such as a PCIe interface on camera link card 104 and PCIe2 interface on data recorder 102. Data recorder 102 can also communicate directly with one or more cameras over an appropriate video-capable interface. As shown in FIG. 1, third camera 110 can be a Boson 640+ LWIR thermal camera core and can communicate with data recorder 102 over a universalAttorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP serial bus (USB). Fourth camera 112 can be a visible camera produced by Imprex or a similarly suitable camera and can communicate over with data recorder 102 over an ethernet communication link.

[0021] Data recorder 102 can also communicate with an embedded Global Positioning System (GPS) + Inertial Navigation System (INS) Embedded GPS (EGI) (“embedded GPS + INS (EGI)”) 114. Embedded GPS + INS (EGI) 114 can provide one or more the following outputs: linear and angular acceleration, linear and angular velocity, position, attitude (roll, pitch), platform azimuth, magnetic and true heading, altitude, body angular rates, time tags, and coordinated universal time (UTC) synchronized time. EGIs provide a robust civil certification to DO-178 and DO-254, enabling much easier certification at the aircraft level for features such as automatic dependence surveillance-broadcast (ADS-B), required navigation performance (RNP) / area navigation (RNAV), wide area application services (WAAS) and more. The GPS gives an absolute drift-free position value that can be used to reset the INS solution or can be blended with it by use of a mathematical algorithm, such as a Kalman filter. The angular orientation of the unit can be inferred from the series of position updates from the GPS. The change in the error in position relative to the GPS can be used to estimate the unknown angle error. The benefits of using GPS with an INS are that the INS may be calibrated by the GPS signals and that the INS can provide position and angle updates at a quicker rate than GPS. For high dynamic vehicles, such as missiles and aircraft, INS fills in the gaps between GPS positions. Additionally, GPS may lose its signal and the INS can continue to compute the position and angle during the period of lost GPS signal. The two systems are complementary and are often employed together. Using GPS / INS allows for smoother position and velocity estimates that can be provided at a sampling rate faster than the GPS receiver. This also allows for accurate estimation of the aircraft attitude (roll, pitch, and yaw) angles. In general, GPS / INS sensor fusion is a nonlinear filtering problem, which is commonly approached using the extended Kalman filter (EKF) or the unscented Kalman filter (UKF). The EKF uses an analytical linearization approach using Jacobian matrices to linearize the system, while the UKF uses a statistical linearization approach called the unscented transform which uses a set of deterministically selected points to handle the nonlinearity. Embedded GPS + INS (EGI) 114 can communicate with data recorder 102 through interface card 116, such as a 1553 PCIe card. Using a 1553 PCLe card as an example of interface card 116, embedded GPS + INS (EGI) 114 can communicate with 1553 PCIe card over a 1553 interface port and 1555 PCIe card can communication with data recorder 1082 over a PCIe interface port. Additionally orAttorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP alternatively, embedded GPS + INS (EGI) 114 can communicate directly with data recorder 102 through an Ethernet port connection.

[0022] Data recorder 102 can output data from one or more cameras (e.g., first camera 106, second camera 108, third camera 110, fourth camera 112, or other camera(s)) and embedded GPS + INS (EGI) 114) to one or more storge devices, such as first storage device 118 and / or second storage device 120. For example, first storage device 118 and / or second storage device 120 can be a solid-state drive (SSD) that can communicate over a Serial Advanced Technology Attachment (SATA) or serial-attached SCSI (SAS). Data recorder 102 can also output data from one or more cameras (first camera 106, second camera 108, third camera 110, fourth camera 112, or other camera(s)) and embedded GPS + INS (EGI) 114 to video stream router 122 over an Ethernet communication link.

[0023] FIG.2 shows block diagram 200 of a process flow between sensor factory 202 and data recorder 102 for the system of FIG. 1 according to examples of the present disclosure. Sensor factory (or sensor bus) 202 comprises one or more sensor interfaces, such as sensor interface 1204, sensor interface 2206, sensor interface 3208, and sensor interface 4210. For example and with reference to FIG. 1, senser interface 1204 can represent the interface for LWIR 1 sensor 212 (e.g., interface for first camera 106 and camera link card 104), senser interface 2206 can represent the interface for LWIR 2 sensor 214 (e.g., interface for second camera 108 and camera link card 104), senser interface 3208 can represent the interface for visible sensor 216 (e.g., interface for fourth camera 112 and data recorder 102), and senser interface 4210 can represent the interface for pressure sensor 218 (e.g., interface for embedded GPS + INS (EGI) 114 and data recorder 102). Senser interface 1204 can configure and read data from LWIR 1 sensor 212, senser interface 2206 can configure and read data from LWIR 2 sensor 214, senser interface 3208 can configure and read data from visible sensor 216, and senser interface 4210 can configure and read data from pressure sensor 218. Data from senser interface 1204, senser interface 2206, senser interface 3208, and senser interface 4210 can be read from and / or pushed to sensor data manager 220. Sensor data manager 220 provides or is queried for sensor data by data recorder 102.

[0024] Data recorder 102 comprises LWIR 1 data queue 222, LWIR 2 data queue 224, visible data queue 226, and barometer data queue 228. Sensor data from LWIR 1 data queue 222 is provided to LWIR 1 file operations 230 and LWIR 1 stream data 232. Sensor data from LWIR 2 data queue 224 is provided to LWIR 2 file operations 234 and LWIR 2 stream data 236. Sensor data from visible data queue 226 is provided to visible file operations 238 andAttorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP visible stream data 240. Sensor data from barometer data queue 228 is provided to barometer file operations 242 and barometer stream data 244.

[0025] Navigation interface 246 is configured to configure and read data from navigation provider 248. Navigation interface 246 is further configured to query LWIR 1 data queue 222, LWIR 2 data queue 224, visible data queue 226, and barometer data queue 228.

[0026] The output of data recorder 102 is provided to a storage device 250 (e.g., first storage device 118 and / or second storage device 120).

[0027] FIG.3 shows block diagram 300 of a process flow between sensor thread 302 from sensor factory 202 and sensor recorder thread 304 from data recorder 102 for the system of FIG.1 according to examples of the present disclosure. Although FIG.3 shows only one sensor and sensor thread for simplicity, each sensor (e.g., first camera 106, second camera 108, third camera 110, fourth camera 112, or other camera(s)) and embedded GPS + INS (EGI) 114) can be associated with its own sensor thread.

[0028] As shown in FIG. 3, sensor thread 302 comprises sensor 306 and navigation provider 248. Sensor 306 produces sensor object accessor 308 and navigation provider 248 produces sensor object accessor 310. Sensor object accessor 308 from sensor 306 is provided to data queue 312 in sensor recorder thread 304. Sensor data from data queue 312, if available, is provided to grabdata() 314. Sensor object accessor 310 from navigation provider 248 is also provided to grabdata() 314. Data from grabdata() 314 is provided to isfilefule decision box 316. If the file is full, then the process proceeds to close the file at closefile() 318. If the file is not full, then the process proceeds to savedata() 320 where the data is saved. After the file is closed at 318, a new file is created at createnewfile() 322. Data from savedata() 320 is then provided to isqueueddata 324, which then is provided to grabdata() 314.

[0001] FIG.4A and FIG.4B show block diagram 400 of a process flow between navigation data manager 402, sensor manager 404, and file write manager 406 for the system of FIG. 1 according to examples of the present disclosure. A navigation device 410 is discovered and configured 408, and, when alignment of the device is reached (i.e. when the vertical axis is found for example by aligning the device’s vertical ring laser gyro with the earth’s gravity vector, and by finding inertial north), data are received by the navigation data manager 402 from a navigation device 414 such as, for example, but not limited to, Falcn EGI (UDP-IP bus). Data are received from the navigation device through a user datagram protocol / internet protocol (UDP / IP) to Ethernet interface onto a plurality of Ethernet channels. As shown, new data, at 408, is queried from Falcn EGI (1553 bus) 410 and is provided to box 412 where a determination that alignment is greater than or equal to a predetermined threshold. Data fromAttorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP box 412 is provided to navigation data manager 402. Falcn EGI (UDP-IP bus) 414 is provided to navigation data manager 402 at ETH-OUT-13, ETH-OUT-28, ETH-OUT-10003, ETH- OUT-20004, and ETH-OUT-10008. Data output from navigation data manager 402 is provided to SQL database 416 to ensure that the data are compliant with an SQL-accessible format. Data output from navigation data manager 402 is also provided to a shared navigation pointer 418 with a mutually exclusive flag (mutex set) which protects the navigation data from concurrent access, as the navigation data objects 420 are stored within a shared navigation pointer 418 area. As shown, data output from navigation data manager 402 is provided to SQL database 416. Data output from navigation data manager 402 is also provided to a shared navigation pointer 418 with a mutually exclusive flag (mutex set). Shared navigation pointer 418 comprises a navigation data object 420. Mutex get is provided from shared navigation pointer 418 to decision box, set(void*data) 422, where navigation data object 420 is copied. The navigation data objects 420 are correlated 422 with sensor data objects, each of which can include any data type, and the correlated data are provided to the file write manager 406. With regard to the sensor manager 404, when a new sensor is initiated at 424, a connection attempt between the sensor 426 and the sensor manager 404. If 428 the sensor 426 is successfully connected, and if 430 sensor data are available, the sensor data are stored in a sensor data object 432. If no sensor data are available, then the process goes back to 428 to wait for a successful sensor connection to be established. The sensor data object 432 is then correlated with the navigation data object 420.

[0029] With regard to file write manager 406, when a new storage means 434 such as a file is discovered, a file path is created 436 and a sensor data queue 440 is initialized 438. Correlated data are stored on the sensor data queue 440 as they arrive from the correlation function. When 442 data are available on the sensor data queue 440, the data are moved to a binary frame 444, which includes the navigation data object 446 and the data 448 from the sensor data queue. The data from the binary frame 444 are stored in a sensor binary file 450. The file write manager 406 maintains a sensor counter. As shown with regard to sensor manager 404, when a new sensor is initiated at 424, sensor data from sensor 426 is provided to sensor manager 404. A determination is made at 428 if sensor 426 is connected. If sensor 426 is connected, a determination is made at 430 as to whether sensor data is available. If sensor data is available, the sensor data is provided to sensor data object 432. If no sensor data is available, then the process goes back to 428 to wait for a sensor to be connected. Sensor data provided by sensor data object 432 is provided by sensor manager 404 to decision box set (void*data) 422.Attorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP

[0030] With regard to file write manager 406, data from decision box set (void*data) 422 is provided to file write manager 406. New data, from box 434, is provided to decision box 436 (clear / create file path), is then provided to decision box 438 (initiate sensor data queue). Data from decision box 438 (initiate sensor data queue) and data from decision box set (void*data) 422 is provided to first-in-first out (FIFO) sensor data queue 440 of file write manager 406. If data is available in FIFO sensor data queue 440 as determined by data available decision box 442, the data is written to binary frame 444, which comprises navigation data object 446 and sensordataqueue(0) 448. Binary frame 444 is then stored in sensor binary file 450. If no data is available as determined by data available decision box 442, then the process goes back to FIFO sensor data queue 440 to await the next data to save.

[0031] FIG.5 shows software stack 500 for a central processing unit (CPU) of data recorder 102 according to examples of the present disclosure. The first (bottom) layer of software stack 500 is CPU layer 502. The second layer above CPU layer 502 comprises operating system layer 504 and system calls layer 506. The third layer above operating system layer 504 comprises kernel layer 508 and file input / output (I / O) layer 510. The fourth layer above kernel layer 508 comprises 1553 layer 512, user datagram protocol (UDP)-internet protocol (IP) layer 514, universal serial bus (USB) controller layer 516, cameralink layer 518, Gigabit Ethernet (GigE) layer 520 and above file input / output (I / O) layer 510 comprises recorder layer 522, and logging layer 524. The fifth layer above 1553 layer 512 comprises command and control (C2) EGI layer 526. The fifth layer above UDP-IP layer 514 comprises video streamer layer 528 and EGI layer 530. The fifth layer above the USB controller layer 516 comprises FLIP application protocol interface (API) layer 532 and Video4Linux2 (V4L2) layer 534. The fifth layer above cameralink layer 518 comprises Video4Linux1 (V4L1) layer 536. The fifth layer above the GigE layer 520 comprises internetwork packet exchange (IPX) layer 538.

[0032] In one or more embodiments, the functions described can be implemented in hardware, software, firmware, or any combination thereof. For a software implementation, the techniques described herein can be implemented with modules (e.g., procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, and so on) that perform the functions described herein. A module can be coupled to another module or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, or the like can be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, and the like. The software codes can be stored in memory units and executed by processors. The memory unit can be implemented within theAttorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP processor or external to the processor, in which case it can be communicatively coupled to the processor via various means as is known in the art.

[0033] In some embodiments, any of the methods of the present disclosure may be executed by a computing system. FIG. 6 illustrates an example of such a computing system 600, in accordance with some examples. The computing system 600 may include a computer or computer system 601A, which may be an individual computer system 601A or an arrangement of distributed computer systems. The computer system 601A includes one or more analysis module(s) 602 configured to perform various tasks according to some embodiments, such as one or more methods disclosed herein. To perform these various tasks, the analysis module 602 executes independently, or in coordination with, one or more processors 604, which is (or are) connected to one or more storage media 606. The processor(s) 604 is (or are) also connected to a network interface 607 to allow the computer system 601A to communicate over a data network 609 with one or more additional computer systems and / or computing systems, such as 601B, 601C, and / or 601D (note that computer systems 601B, 601C and / or 601D may or may not share the same architecture as computer system 601A, and may be located in different physical locations, e.g., computer systems 601A and 601B may be located in a processing facility, while in communication with one or more computer systems such as 601C and / or 601D that are located in one or more data centers, and / or located in varying countries on different continents). A processor can include a microprocessor, microcontroller, processor module or subsystem, programmable integrated circuit, programmable gate array, or another control or computing device.

[0034] The storage media 606 can be implemented as one or more computer-readable or machine-readable storage media. In some examples, the storage media 606 is a non-volatile storage media / medium that can store instructions that when executed by one or more hardware processors, such as processor(s) 604, performs a method to record low-latency data in real-time with a single tread onto a storge medium as described herein. The storage media 606 can include a real-time operating system 610 that is configured to include a scheduler 611 that is configured to schedule one or more operations to be performed by the processor(s) 604 and / or the real-time operating system 610. The scheduler 611, the real-time operating system 610, and / or the processors(s) 604 can initiate one or more thread(s) 612 to perform one or more operations, function, or actions described herein.

[0035] Note that while in the example embodiment of FIG.6 storage media 606 is depicted as within computer system 601A, in some embodiments, storage media 606 may be distributed within and / or across multiple internal and / or external enclosures of computing system 601AAttorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP and / or additional computing systems. Storage media 606 may include one or more different forms of memory including semiconductor memory devices such as dynamic or static random access memories (DRAMs or SRAMs), erasable and programmable read-only memories (EPROMs), electrically erasable and programmable read-only memories (EEPROMs) and flash memories, magnetic disks such as fixed, floppy and removable disks, other magnetic media including tape, optical media such as compact disks (CDs) or digital video disks (DVDs), BLURAY®disks, or other types of optical storage, or other types of storage devices. Note that the instructions discussed above can be provided on one computer-readable or machine-readable storage medium, or alternatively, can be provided on multiple computer- readable or machine-readable storage media distributed in a large system having possibly plural nodes. Such computer-readable or machine-readable storage medium or media is (are) considered to be part of an article (or article of manufacture). An article or article of manufacture can refer to any manufactured single component or multiple components. The storage medium or media can be located either in the machine running the machine-readable instructions, or located at a remote site from which machine-readable instructions can be downloaded over a network for execution.

[0036] In some embodiments, computing system 600 contains one or more machine-learning module(s) 608. In the example of computing system 600, computer system 601A includes the machine-learning module 608. In some embodiments, a single machine-learning module may be used to perform some or all aspects of one or more embodiments of the methods. In alternate embodiments, a plurality of machine-learning modules may be used to perform some or all aspects of methods.

[0037] It should be appreciated that computing system 600 is only one example of a computing system, and that computing system 600 may have more or fewer components than shown, may combine additional components not depicted in the example embodiment of FIG. 6, and / or computing system 600 may have a different configuration or arrangement of the components depicted in FIG.6. The various components shown in FIG.6 may be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0038] Further, the steps in the processing methods described herein may be implemented by running one or more functional modules in information processing apparatus such as general purpose processors or application specific chips, such as ASICs, FPGAs, PLDs, or other appropriate devices. These modules, combinations of these modules, and / or their combination with general hardware are all included within the scope of protection of the invention.Attorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP

[0039] Interpretations, models and / or other interpretation aids may be refined in an iterative fashion; this concept is applicable to embodiments of the present methods discussed herein. This can include use of feedback loops executed on an algorithmic basis, such as at a computing device (e.g., computing system 600, FIG. 6), and / or through manual control by a user who may make determinations regarding whether a given step, action, template, model, or set of curves has become sufficiently accurate for the evaluation of the conditions under consideration.

[0040] FIG. 7 shows a flowchart 700 depicting a method to record low-latency data in real- time with a single tread onto a storge medium according to examples of the present disclosure. The method begins by instantiating, by a single thread of a scheduler of a real-time operating system executed by a hardware processor, a write command to write data from a data queue to a storage medium, as in 702. Turning to the figures described above, real-time operating system 610, as executed by the processor(s) 604, initiates the scheduler 611 to schedule the write command using a thread of the thread(s) 612. The write command writes data from the data queue, such as first-in-first out (FIFO) sensor data queue 440, to the storage medium, such as the binary frame 444 that is then stored in sensor binary file 450, on storage media 606, for example.

[0041] In some examples, the data comprises time-synchronized image data and / or video data and associated geographic data and / or navigation-related data. In some examples, the image data and / or the video data comprises infrared image or infrared video obtained from one or more infrared spectrum capable cameras and / or one or more visible spectrum capable cameras. In some examples, the geographic data and / or navigation-related data are obtained from an embedded Global Positioning System (GPS) + Inertial Navigation System (INS) Embedded GPS (EGI) (“embedded GPS + INS (EGI)”) system. In some examples, the write command is configured to define a memory size for the data. In some examples, the write command is configured to define one or more tunable parameters. In some examples, the one or more tunable parameters comprise a maximum frame period. In some examples, the data is configured as a data frame.

[0042] The method continues by determining, by the single thread, that the data is in the data queue, as in 704. Returning to the figures described above, the thread(s) 612 can determine that the data is in the data queue, such as such as first-in-first out (FIFO) sensor data queue 440, at the decision box set (void*data) 422.

[0043] The method continues by obtaining, by the single thread, the data from the data queue, as in 706. Returning to the figures described above, the thread(s) 612 can obtain the data in theAttorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP data queue, such as such as first-in-first out (FIFO) sensor data queue 440, when the decision box set (void*data) 422 indicates that data is available to be retrieved.

[0044] The method continues by saving, by the single thread and the write command, the data that is obtained from the data queue to the storage medium as an output file, as in 708. Returning to the figures described above, the thread(s) 612 can save the data in the data queue, such as such as the first-in-first out (FIFO) sensor data queue 440, to the storage medium, such as the binary frame 444 that is then stored in sensor binary file 450, on storage media 606, for example.

[0045] In some examples, the method includes determining that the data formatted as a data frame is ready to be saved and transferring the data frame as a data byte array to the data queue, as in 710. Returning to the figures described above, the thread(s) 612 can determine that the data formatted as a data frame is ready to be saved and can transfer the data frame as a data byte array to the data queue, such as such as from the first-in-first out (FIFO) sensor data queue 440 to the storage medium, such as the binary frame 444 that is then stored in sensor binary file 450, on storage media 606, for example.

[0046] In some examples, the method includes prior to saving the data, determining that a data frame count in the output file has met a predetermined threshold, wherein the data is saved based on the data frame count meeting the predetermined threshold, as in 712. Returning to the figures described above, the thread(s) 612 can determine that that a data frame count in the output file has met a predetermined threshold, wherein the data is saved based on the data frame count meeting the predetermined threshold.

[0047] In some examples, the method includes creating a new output file once the data is saved, as in 714. Returning to the figures described above, the thread(s) 612 can create a new output file once the data is saved.

[0048] The examples set forth herein represent the necessary information to enable those skilled in the art to practice the embodiments and illustrate the best mode of practicing the embodiments. Upon reading the description in light of the accompanying drawing figures, those skilled in the art will understand the concepts of the disclosure and will recognize applications of these concepts not particularly addressed herein. It should be understood that these concepts and applications fall within the scope of the disclosure and the accompanying claims.

[0049] It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could beAttorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present disclosure. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0050] It will be understood that when an element such as a layer, region, or substrate is referred to as being “on” or extending “onto” another element, it can be directly on or extend directly onto the other element or intervening elements may also be present. In contrast, when an element is referred to as being “directly on” or extending “directly onto” another element, there are no intervening elements present. Likewise, it will be understood that when an element such as a layer, region, or substrate is referred to as being “over” or extending “over” another element, it can be directly over or extend directly over the other element or intervening elements may also be present. In contrast, when an element is referred to as being “directly over” or extending “directly over” another element, there are no intervening elements present. It will also be understood that when an element is referred to as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being “directly connected” or “directly coupled” to another element, there are no intervening elements present.

[0051] Relative terms such as “below” or “above” or “upper” or “lower” or “horizontal” or “vertical” may be used herein to describe a relationship of one element, layer, or region to another element, layer, or region as illustrated in the Figures. It will be understood that these terms and those discussed above are intended to encompass different orientations of the device in addition to the orientation depicted in the Figures.

[0052] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including” when used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0053] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of thisAttorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Claims

Attorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP What is Claimed is:

1. A method to record low-latency data in real-time with a single tread onto a storge medium, the method comprising: instantiating, by a single thread of a scheduler of a real-time operating system executed by a hardware processor, a write command to write data from a data queue to a storage media; determining, by the single thread, that the data is in the data queue; obtaining, by the single thread, the data from the data queue; and saving, by the single thread and the write command, the data that is obtained from the data queue to the storage media as an output file.

2. The method of claim 1, wherein the data comprises time-synchronized image data and / or video data and associated geographic data and / or navigation-related data.

3. The method of claim 2, wherein the image data and / or the video data comprises infrared image or infrared video obtained from one or more infrared spectrum capable cameras and / or one or more visible spectrum capable cameras.

4. The method of claim 2, wherein the geographic data and / or navigation-related data are obtained from an embedded Global Positioning System (GPS) + Inertial Navigation System (INS) Embedded GPS (EGI) (“embedded GPS + INS (EGI)”) system.

5. The method of claim 1, wherein the write command is configured to define a memory size for the data.

6. The method of claim 1, wherein the write command is configured to define one or more tunable parameters.

7. The method of claim 6, wherein the one or more tunable parameters comprise a maximum frame period.

8. The method of claim 1, wherein the data is configured as a data frame.Attorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP 9. The method of claim 1, further comprising: determining that the data formatted as a data frame is ready to be saved; and transferring the data frame as a data byte array to the data queue.

10. The method of claim 1, further comprising: prior to saving the data, determining that a data frame count in the output file has met a predetermined threshold, wherein the data is saved based on the data frame count meeting the predetermined threshold.

11. The method of claim 10, further comprising creating a new output file once the data is saved.

12. A computer system comprising: a hardware processor; and a storage media storing instructions that when executed by the hardware processor performs a method to record low-latency data in real-time with a single tread onto the storage media, the method comprising: instantiating, by a single thread of a scheduler of a real-time operating system executed by the hardware processor, a write command to write data from a data queue to the storage media; determining, by the single thread, that the data is in the data queue; obtaining, by the single thread, the data from the data queue; and saving, by the single thread and the write command, the data that is obtained from the data queue to the storage media as an output file.

13. The computer system of claim 12, wherein the data comprises time-synchronized image data and / or video data and associated geographic data and / or navigation-related data.

14. The computer system of claim 13, wherein the image data and / or the video data comprises infrared image or infrared video obtained from one or more infrared spectrum capable cameras and / or one or more visible spectrum capable cameras.Attorney Docket No.0192.0445-PCT Client Ref. No.24-0548-US-PSP 15. The computer system of claim 14, wherein the geographic data and / or navigation- related data are obtained from an embedded Global Positioning System (GPS) + Inertial Navigation System (INS) Embedded GPS (EGI) (“embedded GPS + INS (EGI)”) system.

16. The computer system of claim 12, wherein the write command is configured to define a memory size for the data.

17. The computer system of claim 12, wherein the write command is configured to define one or more tunable parameters.

18. The computer system of claim 17, wherein the one or more tunable parameters comprise a maximum frame period.

19. The computer system of claim 12, wherein the method further comprising: determining that the data formatted as a data frame is ready to be saved; and transferring the data frame as a data byte array to the data queue.

20. The computer system of claim 12, wherein the method further comprising: prior to saving the data, determining that a data frame count in the output file has met a predetermined threshold, wherein the data is saved based on the data frame count meeting the predetermined threshold.

Images