Camera data manager

Abstract

A method to method for managing a software interface for one or more cameras is disclosed. The method includes instantiating, by a single thread of a scheduler of a real-time operating system executed by a hardware processor, a sensor object with a configuration file; launching, by the single thread, an asynchronous pipeline; configuring, by the single thread, each of the one or more camera; requesting, by the single thread, frame data from each of the one or more cameras; and providing, by the single thread, a semaphore to the frame data to protect the frame data.

Description

Attorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP Camera Data Manager Cross Reference to Related Applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 674,538 filed on July 23, 2024, the contents of which are hereby incorporated by reference in its entirety. Field

[0002] The present disclosure relates to a camera data manager. Background

[0003] Probe and drogue aerial refueling currently relies on a skilled human pilot to perform the complicated maneuvering needed to plug the drogue and maintain formation flight with a tanker. However, future autonomous air vehicles make this level of high-fidelity pilot- control difficult: an autonomous aerial refueling receive (A2R2) capability will be required for these future aircraft to take advantage of the extended endurance that aerial refueling allows.

[0004] Recording, processing, and displaying aircraft data in real-time is a monumental task especially whenever you add the complexity of video data that is correlated with timing data. For a flight testing or production system, recording good data without gaps is crucial for certification, event re-creation, and trade-study. Summary

[0005] According to examples of the present disclosure, systems, devices, and methods for creating easy to manage software interfaces for camera data are disclosed by providing an abstract camera factory interface for sensors that run at different frame rates and collecting data in different ways. Sensor data is published to a shared broker construct. A caller or subscriber is then permitted to simply request sensor data. The disclosed systems, devices, and methods are not limited to a particular number of cameras and supports, among other things, USB camera, GigE cameras, and a variety of different camera interfaces, including Camera Link. The disclosed systems, devices, and methods correlate navigation data to inertial measurement unit (IMU) data and global position system (GPS) data present in embedded GPS-aided inertial navigation system (INS) (EGI) devices. The disclosed systems, devices, and methods can be used with a variety of camera types and data format types.

[0006] According to examples of the present disclosure, the disclosed systems, devices, and methods can be configured to perform a method that comprises the following features: 1)Attorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP Instantiate sensor object with configuration file; 2) asynchronous pipeline is launched and begins configuring the camera and requesting frame data; 3) whenever frame data comes in, it is safe-guarded with a semaphore, cache data object is overwritten with new data; 4) the caller may request data at system clock speed or at a down-sampled rate without failure. Linux provides v4l2 or kernel level driver (Video For Linux) control of sensors. The disclosed systems, devices, and methods expands upon that functionality to provide software that does not need to block or wait for a response from the sensor, can be queried for error state without querying the sensor directly, smart memory utilization built in so that the caller may not assume the burden for itself as assumed in the Linux implementation. The disclosed systems, devices, and methods provide for asynchronous data caching mechanism, a dynamic queue data structure, smart clock restriction (processing is slowed down to a factor of the sensor's frame rate), atomic read call to caching. The disclosed systems, devices, and methods can be applied to any system that needs to acquire highly accurate optical data for certification or recreation later.

[0007] According to examples of the present disclosure, a method for managing a software interface for one or more cameras is disclosed. The method comprises instantiating, by a single thread of a scheduler of a real-time operating system executed by a hardware processor, a sensor object with a configuration file; launching, by the single thread, an asynchronous pipeline; configuring, by the single thread, each of the one or more cameras; requesting, by the single thread, frame data from each of the one or more cameras; and providing, by the single thread, a semaphore to the frame data to protect the frame data.

[0008] Various additional features can be included in the method including one or more of the following features. The method further comprises storing the frame data with the semaphore in a data cache. The method further comprises overwriting the frame data with the semaphore that is stored in the data cache with new data acquired from one camera of the one or more cameras. The scheduler is configured to request data from each of the one or more cameras at system clock speed or at a down-sampled rate. The frame data comprises time- synchronized image data and / or video data and associated geographic data and / or kinematic- related data. The image data and / or the video data comprises infrared image or infrared video obtained from one or more infrared spectrum capable cameras and / or one or more visible spectrum capable cameras. The geographic data and / or kinematic-related data are obtained from an embedded Global Positioning System (GPS) + Inertial Navigation System (INS) Embedded GPS (EGI) (“embedded GPS + INS (EGI)”) system.Attorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP

[0009] According to examples of the present disclosure, a computer system is disclosed that comprises a hardware processor and a storage media storing instructions that when executed by the hardware processor performs a method for managing a software interface for one or more cameras, the method comprising: instantiating, by a single thread of a scheduler of a real-time operating system executed by a hardware processor, a sensor object with a configuration file; launching, by the single thread, an asynchronous pipeline; configuring, by the single thread, each of the one or more cameras; requesting, by the single thread, frame data from each of the one or more cameras; and providing, by the single thread, a semaphore to the frame data to protect the frame data.

[0010] Various additional features can be included in the computer system including one or more of the following features. The method further comprises storing the frame data with the semaphore in a data cache. The method further comprises overwriting the frame data with the semaphore that is stored in the data cache with new data acquired from one camera of the one or more cameras. The scheduler is configured to request data from each of the one or more cameras at system clock speed or at a down-sampled rate. The frame data comprises time- synchronized image data and / or video data and associated geographic data and / or kinematic- related data. The image data and / or the video data comprises infrared image or infrared video obtained from one or more infrared spectrum capable cameras and / or one or more visible spectrum capable cameras. The geographic data and / or kinematic-related data are obtained from an embedded Global Positioning System (GPS) + Inertial Navigation System (INS) Embedded GPS (EGI) (“embedded GPS + INS (EGI)”) system.

[0011] According to examples of the present disclosure, a non-volatile storage media is disclosed that stores instructions that when executed by a hardware processor performs a method for managing a software interface for one or more cameras, the method comprising: instantiating, by a single thread of a scheduler of a real-time operating system executed by a hardware processor, a sensor object with a configuration file; launching, by the single thread, an asynchronous pipeline; configuring, by the single thread, each of the one or more cameras; requesting, by the single thread, frame data from each of the one or more cameras; and providing, by the single thread, a semaphore to the frame data to protect the frame data.

[0012] Various additional features can be included in the non-volatile storage media including one or more of the following features. The method further comprises storing the frame data with the semaphore in a data cache. The method further comprises overwriting the frame data with the semaphore that is stored in the data cache with new data acquired from one camera of the one or more cameras. The scheduler is configured to request data from each ofAttorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP the one or more cameras at system clock speed or at a down-sampled rate. The frame data comprises time-synchronized image data and / or video data and associated geographic data and / or kinematic-related data. The image data and / or the video data comprises infrared image or infrared video obtained from one or more infrared spectrum capable cameras and / or one or more visible spectrum capable cameras. The geographic data and / or kinematic-related data are obtained from an embedded Global Positioning System (GPS) + Inertial Navigation System (INS) Embedded GPS (EGI) (“embedded GPS + INS (EGI)”) system. Brief Description of the Figures

[0013] FIG. 1 shows a block diagram of a hardware configuration of a system for data recording according to examples of the present disclosure.

[0014] FIG.2 shows a block diagram of a process flow between a sensor factory and a data recorder for the system of FIG.1 according to examples of the present disclosure.

[0015] FIG. 3 shows a block diagram of a process flow between a sensor thread from a sensor factory and a sensor recorder thread from a data recorder for the system of FIG. 1 according to examples of the present disclosure.

[0016] FIG.4A and FIG.4B show a block diagram of a process flow between a navigation data manager, a sensor manager, and a file write manager for the system of FIG. 1 according to examples of the present disclosure.

[0017] FIG.5 shows a software stack for a central processing unit (CPU) of a data recorder according to examples of the present disclosure.

[0018] FIG.6 illustrates an example of such a computing system, in accordance with some examples.

[0019] FIG.7 shows a flowchart depicting a method for managing a software interface for one or more cameras according to examples of the present disclosure. Detailed Description

[0020] FIG. 1 shows block diagram 100 of a hardware configuration of a system for data recording according to examples of the present disclosure. Data recorder 102 can be configured to interface with and collect data from one or more sensors. The one or more sensors can take a variety of forms including video cameras and position / telemetry-type sensors.

[0021] As shown in FIG. 1, camera link card 104 is configured to communicate with one or more cameras, such as first camera 106 and second camera 108. For example, first camera 106 and second camera 108 can be different types of infrared cameras. In one non-limitingAttorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP example, first camera 106 can be an ATOM®1024 camera core, which is a high-resolution uncooled thermal imager for long-wave infrared (LWIR) applications. Second camera 108 can be a Teledyne forward looking infrared (FLIR) Boson 640 LWIR thermal camera core. Camera link card 104 can communicate with first camera 106 over a physical connector using a CL port on first camera 106 and a CLA port on camera link card 104. Camera link card 104 can communicate with second camera 108 over a physical connector using a CL port on second camera 108 and a CLB port on camera link card 104. In some examples, more than two cameras can be coupled with camera link card 104.

[0022] Data recorder 102 communicates with camera link card 104 over a communication interface, such as a PCIe interface on camera link card 104 and PCIe2 interface on data recorder 102. Data recorder 102 can also communicate directly with one or more cameras over an appropriate video-capable interface. As shown in FIG. 1, third camera 110 can be a Boson 640+ LWIR thermal camera core and can communicate with data recorder 102 over a universal serial bus (USB). Fourth camera 112 can be a visible camera produced by Imprex or a similarly suitable camera and can communicate over with data recorder 102 over an ethernet communication link.

[0023] Data recorder 102 can also communicate with an embedded Global Positioning System (GPS) + Inertial Navigation System (INS) Embedded GPS (EGI) (“embedded GPS + INS (EGI)”) 114. Embedded GPS + INS (EGI) 114 can provide one or more the following outputs: linear and angular acceleration, linear and angular velocity, position, attitude (roll, pitch), platform azimuth, magnetic and true heading, altitude, body angular rates, time tags, and coordinated universal time (UTC) synchronized time. EGIs provide a robust civil certification to DO-178 and DO-254, enabling much easier certification at the aircraft level for features such as automatic dependence surveillance-broadcast (ADS-B), required navigation performance (RNP) / area navigation (RNAV), wide area application services (WAAS) and more. The GPS gives an absolute drift-free position value that can be used to reset the INS solution or can be blended with it by use of a mathematical algorithm, such as a Kalman filter. The angular orientation of the unit can be inferred from the series of position updates from the GPS. The change in the error in position relative to the GPS can be used to estimate the unknown angle error. The benefits of using GPS with an INS are that the INS may be calibrated by the GPS signals and that the INS can provide position and angle updates at a quicker rate than GPS. For high dynamic vehicles, such as missiles and aircraft, INS fills in the gaps between GPS positions. Additionally, GPS may lose its signal and the INS can continue to compute the position and angle during the period of lost GPS signal. The two systems areAttorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP complementary and are often employed together. Using GPS / INS allows for smoother position and velocity estimates that can be provided at a sampling rate faster than the GPS receiver. This also allows for accurate estimation of the aircraft attitude (roll, pitch, and yaw) angles. In general, GPS / INS sensor fusion is a nonlinear filtering problem, which is commonly approached using the extended Kalman filter (EKF) or the unscented Kalman filter (UKF). The EKF uses an analytical linearization approach using Jacobian matrices to linearize the system, while the UKF uses a statistical linearization approach called the unscented transform which uses a set of deterministically selected points to handle the nonlinearity. Embedded GPS + INS (EGI) 114 can communicate with data recorder 102 through interface card 116, such as a 1553 PCIe card. Using a 1553 PCLe card as an example of interface card 116, embedded GPS + INS (EGI) 114 can communicate with 1553 PCIe card over a 1553 interface port and 1555 PCIe card can communication with data recorder 1082 over a PCIe interface port. Additionally or alternatively, embedded GPS + INS (EGI) 114 can communicate directly with data recorder 102 through an Ethernet port connection.

[0024] Data recorder 102 can output data from one or more cameras (e.g., first camera 106, second camera 108, third camera 110, fourth camera 112, or other camera(s)) and embedded GPS + INS (EGI) 114) to one or more storge devices, such as first storage device 118 and / or second storage device 120. For example, first storage device 118 and / or second storage device 120 can be a solid-state drive (SSD) that can communicate over a Serial Advanced Technology Attachment (SATA) or serial-attached SCSI (SAS). Data recorder 102 can also output data from one or more cameras (first camera 106, second camera 108, third camera 110, fourth camera 112, or other camera(s)) and embedded GPS + INS (EGI) 114 to video stream router 122 over an Ethernet communication link.

[0025] FIG.2 shows block diagram 200 of a process flow between sensor factory 202 and data recorder 102 for the system of FIG. 1 according to examples of the present disclosure. Sensor factory (or sensor bus) 202 comprises one or more sensor interfaces, such as sensor interface 1204, sensor interface 2206, sensor interface 3208, and sensor interface 4210. For example and with reference to FIG. 1, senser interface 1204 can represent the interface for LWIR 1 sensor 212 (e.g., interface for first camera 106 and camera link card 104), senser interface 2206 can represent the interface for LWIR 2 sensor 214 (e.g., interface for second camera 108 and camera link card 104), senser interface 3208 can represent the interface for visible sensor 216 (e.g., interface for fourth camera 112 and data recorder 102), and senser interface 4210 can represent the interface for pressure sensor 218 (e.g., interface for embedded GPS + INS (EGI) 114 and data recorder 102). Senser interface 1204 can configure and readAttorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP data from LWIR 1 sensor 212, senser interface 2206 can configure and read data from LWIR 2 sensor 214, senser interface 3208 can configure and read data from visible sensor 216, and senser interface 4210 can configure and read data from pressure sensor 218. Data from senser interface 1204, senser interface 2206, senser interface 3208, and senser interface 4210 can be read from and / or pushed to sensor data manager 220. Sensor data manager 220 provides or is queried for sensor data by data recorder 102.

[0026] Data recorder 102 comprises LWIR 1 data queue 222, LWIR 2 data queue 224, visible data queue 226, and barometer data queue 228. Sensor data from LWIR 1 data queue 222 is provided to LWIR 1 file operations 230 and LWIR 1 stream data 232. Sensor data from LWIR 2 data queue 224 is provided to LWIR 2 file operations 234 and LWIR 2 stream data 236. Sensor data from visible data queue 226 is provided to visible file operations 238 and visible stream data 240. Sensor data from barometer data queue 228 is provided to barometer file operations 242 and barometer stream data 244.

[0027] Navigation interface 246 is configured to configure and read data from navigation provider 248. Navigation interface 246 is further configured to query LWIR 1 data queue 222, LWIR 2 data queue 224, visible data queue 226, and barometer data queue 228.

[0028] The output of data recorder 102 is provided to a storage device 250 (e.g., first storage device 118 and / or second storage device 120).

[0029] FIG.3 shows block diagram 300 of a process flow between sensor thread 302 from sensor factory 202 and sensor recorder thread 304 from data recorder 102 for the system of FIG.1 according to examples of the present disclosure. Although FIG.3 shows only one sensor and sensor thread for simplicity, each sensor (e.g., first camera 106, second camera 108, third camera 110, fourth camera 112, or other camera(s)) and embedded GPS + INS (EGI) 114) can be associated with its own sensor thread.

[0030] As shown in FIG. 3, sensor thread 302 comprises sensor 306 and navigation provider 248. Sensor 306 produces sensor object accessor 308 and navigation provider 248 produces sensor object accessor 310. Sensor object accessor 308 from sensor 306 is provided to data queue 312 in sensor recorder thread 304. Sensor data from data queue 312, if available, is provided to grabdata() 314. Sensor object accessor 310 from navigation provider 248 is also provided to grabdata() 314. Data from grabdata() 314 is provided to isfilefule decision box 316. If the file is full, then the process proceeds to close the file at closefile() 318. If the file is not full, then the process proceeds to savedata() 320 where the data is saved. After the file is closed at 318, a new file is created at createnewfile() 322. Data from savedata() 320 is then provided to isqueueddata 324, which then is provided to grabdata() 314.Attorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP

[0001] FIG.4A and FIG.4B show block diagram 400 of a process flow between navigation data manager 402, sensor manager 404, and file write manager 406 for the system of FIG. 1 according to examples of the present disclosure. A navigation device 410 is discovered and configured 408, and, when alignment of the device is reached (i.e. when the vertical axis is found for example by aligning the device’s vertical ring laser gyro with the earth’s gravity vector, and by finding inertial north), data are received by the navigation data manager 402 from a navigation device 414 such as, for example, but not limited to, Falcn EGI (UDP-IP bus). Data are received from the navigation device through a user datagram protocol / internet protocol (UDP / IP) to Ethernet interface onto a plurality of Ethernet channels. As shown, new data, at 408, is queried from Falcn EGI (1553 bus) 410 and is provided to box 412 where a determination that alignment is greater than or equal to a predetermined threshold. Data from box 412 is provided to navigation data manager 402. Falcn EGI (UDP-IP bus) 414 is provided to navigation data manager 402 at ETH-OUT-13, ETH-OUT-28, ETH-OUT-10003, ETH- OUT-20004, and ETH-OUT-10008. Data output from navigation data manager 402 is provided to SQL database 416 to ensure that the data are compliant with an SQL-accessible format. Data output from navigation data manager 402 is also provided to a shared navigation pointer 418 with a mutually exclusive flag (mutex set) which protects the navigation data from concurrent access, as the navigation data objects 420 are stored within a shared navigation pointer 418 area. As shown, data output from navigation data manager 402 is provided to SQL database 416. Data output from navigation data manager 402 is also provided to a shared navigation pointer 418 with a mutually exclusive flag (mutex set). Shared navigation pointer 418 comprises a navigation data object 420. Mutex get is provided from shared navigation pointer 418 to decision box, set(void*data) 422, where navigation data object 420 is copied. The navigation data objects 420 are correlated 422 with sensor data objects, each of which can include any data type, and the correlated data are provided to the file write manager 406. With regard to the sensor manager 404, when a new sensor is initiated at 424, a connection attempt between the sensor 426 and the sensor manager 404. If 428 the sensor 426 is successfully connected, and if 430 sensor data are available, the sensor data are stored in a sensor data object 432. If no sensor data are available, then the process goes back to 428 to wait for a successful sensor connection to be established. The sensor data object 432 is then correlated with the navigation data object 420.

[0031] With regard to file write manager 406, when a new storage means 434 such as a file is discovered, a file path is created 436 and a sensor data queue 440 is initialized 438. Correlated data are stored on the sensor data queue 440 as they arrive from the correlationAttorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP function. When 442 data are available on the sensor data queue 440, the data are moved to a binary frame 444, which includes the navigation data object 446 and the data 448 from the sensor data queue. The data from the binary frame 444 are stored in a sensor binary file 450. The file write manager 406 maintains a sensor counter. As shown with regard to sensor manager 404, when a new sensor is initiated at 424, sensor data from sensor 426 is provided to sensor manager 404. A determination is made at 428 if sensor 426 is connected. If sensor 426 is connected, a determination is made at 430 as to whether sensor data is available. If sensor data is available, the sensor data is provided to sensor data object 432. If no sensor data is available, then the process goes back to 428 to wait for a sensor to be connected. Sensor data provided by sensor data object 432 is provided by sensor manager 404 to decision box set (void*data) 422.

[0032] With regard to file write manager 406, data from decision box set (void*data) 422 is provided to file write manager 406. New data, from box 434, is provided to decision box 436 (clear / create file path), is then provided to decision box 438 (initiate sensor data queue). Data from decision box 438 (initiate sensor data queue) and data from decision box set (void*data) 422 is provided to first-in-first out (FIFO) sensor data queue 440 of file write manager 406. If data is available in FIFO sensor data queue 440 as determined by data available decision box 442, the data is written to binary frame 444, which comprises navigation data object 446 and sensordataqueue(0) 448. Binary frame 444 is then stored in sensor binary file 450. If no data is available as determined by data available decision box 442, then the process goes back to FIFO sensor data queue 440 to await the next data to save.

[0033] FIG.5 shows software stack 500 for a central processing unit (CPU) of data recorder 102 according to examples of the present disclosure. The first (bottom) layer of software stack 500 is CPU layer 502. The second layer above CPU layer 502 comprises operating system layer 504 and system calls layer 506. The third layer above operating system layer 504 comprises kernel layer 508 and file input / output (I / O) layer 510. The fourth layer above kernel layer 508 comprises 1553 layer 512, user datagram protocol (UDP)-internet protocol (IP) layer 514, universal serial bus (USB) controller layer 516, cameralink layer 518, Gigabit Ethernet (GigE) layer 520 and above file input / output (I / O) layer 510 comprises recorder layer 522, and logging layer 524. The fifth layer above 1553 layer 512 comprises command and control (C2) EGI layer 526. The fifth layer above UDP-IP layer 514 comprises video streamer layer 528 and EGI layer 530. The fifth layer above the USB controller layer 516 comprises FLIP application protocol interface (API) layer 532 and Video4Linux2 (V4L2) layer 534. The fifth layer aboveAttorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP cameralink layer 518 comprises Video4Linux1 (V4L1) layer 536. The fifth layer above the GigE layer 520 comprises internetwork packet exchange (IPX) layer 538.

[0034] In one or more embodiments, the functions described can be implemented in hardware, software, firmware, or any combination thereof. For a software implementation, the techniques described herein can be implemented with modules (e.g., procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, and so on) that perform the functions described herein. A module can be coupled to another module or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, or the like can be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, and the like. The software codes can be stored in memory units and executed by processors. The memory unit can be implemented within the processor or external to the processor, in which case it can be communicatively coupled to the processor via various means as is known in the art.

[0035] In some embodiments, any of the methods of the present disclosure may be executed by a computing system. FIG. 6 illustrates an example of such a computing system 600, in accordance with some examples. The computing system 600 may include a computer or computer system 601A, which may be an individual computer system 601A or an arrangement of distributed computer systems. The computer system 601A includes one or more analysis module(s) 602 configured to perform various tasks according to some embodiments, such as one or more methods disclosed herein. To perform these various tasks, the analysis module 602 executes independently, or in coordination with, one or more processors 604, which is (or are) connected to one or more storage media 606. The processor(s) 604 is (or are) also connected to a network interface 607 to allow the computer system 601A to communicate over a data network 609 with one or more additional computer systems and / or computing systems, such as 601B, 601C, and / or 601D (note that computer systems 601B, 601C and / or 601D may or may not share the same architecture as computer system 601A, and may be located in different physical locations, e.g., computer systems 601A and 601B may be located in a processing facility, while in communication with one or more computer systems such as 601C and / or 601D that are located in one or more data centers, and / or located in varying countries on different continents). A processor can include a microprocessor, microcontroller, processor module or subsystem, programmable integrated circuit, programmable gate array, or another control or computing device.Attorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP

[0036] The storage media 606 can be implemented as one or more computer-readable or machine-readable storage media. In some examples, the storage media 606 is a non-volatile storage media / medium that can store instructions that when executed by one or more hardware processors, such as processor(s) 604, performs a method to record low-latency data in real-time with a single tread onto a storge medium as described herein. The storage media 606 can include a real-time operating system 610 that is configured to include a scheduler 611 that is configured to schedule one or more operations to be performed by the processor(s) 604 and / or the real-time operating system 610. The scheduler 611, the real-time operating system 610, and / or the processors(s) 604 can initiate one or more thread(s) 612 to perform one or more operations, function, or actions described herein.

[0037] Note that while in the example embodiment of FIG.6 storage media 606 is depicted as within computer system 601A, in some embodiments, storage media 606 may be distributed within and / or across multiple internal and / or external enclosures of computing system 601A and / or additional computing systems. Storage media 606 may include one or more different forms of memory including semiconductor memory devices such as dynamic or static random access memories (DRAMs or SRAMs), erasable and programmable read-only memories (EPROMs), electrically erasable and programmable read-only memories (EEPROMs) and flash memories, magnetic disks such as fixed, floppy and removable disks, other magnetic media including tape, optical media such as compact disks (CDs) or digital video disks (DVDs), BLURAY®disks, or other types of optical storage, or other types of storage devices. Note that the instructions discussed above can be provided on one computer-readable or machine-readable storage medium, or alternatively, can be provided on multiple computer- readable or machine-readable storage media distributed in a large system having possibly plural nodes. Such computer-readable or machine-readable storage medium or media is (are) considered to be part of an article (or article of manufacture). An article or article of manufacture can refer to any manufactured single component or multiple components. The storage medium or media can be located either in the machine running the machine-readable instructions, or located at a remote site from which machine-readable instructions can be downloaded over a network for execution.

[0038] In some embodiments, computing system 600 contains one or more machine-learning module(s) 608. In the example of computing system 600, computer system 601A includes the machine-learning module 608. In some embodiments, a single machine-learning module may be used to perform some or all aspects of one or more embodiments of the methods. In alternateAttorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP embodiments, a plurality of machine-learning modules may be used to perform some or all aspects of methods.

[0039] It should be appreciated that computing system 600 is only one example of a computing system, and that computing system 600 may have more or fewer components than shown, may combine additional components not depicted in the example embodiment of FIG. 6, and / or computing system 600 may have a different configuration or arrangement of the components depicted in FIG.6. The various components shown in FIG.6 may be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0040] Further, the steps in the processing methods described herein may be implemented by running one or more functional modules in information processing apparatus such as general purpose processors or application specific chips, such as ASICs, FPGAs, PLDs, or other appropriate devices. These modules, combinations of these modules, and / or their combination with general hardware are all included within the scope of protection of the invention.

[0041] Interpretations, models and / or other interpretation aids may be refined in an iterative fashion; this concept is applicable to embodiments of the present methods discussed herein. This can include use of feedback loops executed on an algorithmic basis, such as at a computing device (e.g., computing system 600, FIG. 6), and / or through manual control by a user who may make determinations regarding whether a given step, action, template, model, or set of curves has become sufficiently accurate for the evaluation of the conditions under consideration.

[0042] FIG.7 shows a flowchart 700 depicting a method for managing a software interface for one or more cameras according to examples of the present disclosure. The method comprises instantiating, by a single thread of a scheduler of a real-time operating system executed by a hardware processor, a sensor object with a configuration file, as in 702. Turning to the figures described above, real-time operating system 610, as executed by the processor(s) 604, initiates the scheduler 611 to instantiate a sensor object with a configuration file using a single thread from the thread(s) 612.

[0043] The method continues by launching, by the single thread, an asynchronous pipeline, as in 704. Returning to the figures described above, real-time operating system 610, as executed by the processor(s) 604, initiates the scheduler 611 to launch an asynchronous pipeline using the single thread from the thread(s) 612.

[0044] The method continues by configuring, by the single thread, each of the one or more cameras, as in 706. Returning to the figures described above, real-time operating system 610,Attorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP as executed by the processor(s) 604, initiates the scheduler 611 to configure each of the one or more cameras using the single thread from the thread(s) 612.

[0045] The method continues by requesting, by the single thread, frame data from each of the one or more cameras, as in 708. Returning to the figures described above, real-time operating system 610, as executed by the processor(s) 604, initiates the scheduler 611 to request, by the single thread, frame data from each of the one or more cameras (e.g., first camera 106, second camera 108, third camera 110, fourth camera 112, or other camera(s)) and embedded GPS + INS (EGI) 114).

[0046] The method continues by providing, by the single thread, a semaphore to the frame data to protect the frame data, as in 710. Returning to the figures described above, real-time operating system 610, as executed by the processor(s) 604, initiates the scheduler 611 to provide, by the single thread, a semaphore to the frame data to protect the frame data.

[0047] In some examples, the method comprises storing the frame data with the semaphore in a data cache, as in 712. Returning to the figures described above, real-time operating system 610, as executed by the processor(s) 604, initiates the scheduler 611 to store the frame data with the semaphore in a data cache, such as storage media 606.

[0048] In some examples, the method comprises overwriting the frame data with the semaphore that is stored in the data cache with new data acquired from one camera of the one or more cameras, as in 714. Returning to the figures described above, real-time operating system 610, as executed by the processor(s) 604, initiates the scheduler 611 to overwrite the frame data with the semaphore that is stored in the data cache with new data acquired from one camera of the one or more cameras (e.g., first camera 106, second camera 108, third camera 110, fourth camera 112, or other camera(s)) and embedded GPS + INS (EGI) 114).

[0049] In some examples, the scheduler is configured to request data from each of the one or more cameras at system clock speed or at a down-sampled rate. The frame data comprises time-synchronized image data and / or video data and associated geographic data and / or kinematic-related data. The image data and / or the video data comprises infrared image or infrared video obtained from one or more infrared spectrum capable cameras and / or one or more visible spectrum capable cameras. The geographic data and / or kinematic-related data are obtained from an embedded Global Positioning System (GPS) + Inertial Navigation System (INS) Embedded GPS (EGI) (“embedded GPS + INS (EGI)”) system.

[0050] The examples set forth herein represent the necessary information to enable those skilled in the art to practice the embodiments and illustrate the best mode of practicing the embodiments. Upon reading the description in light of the accompanying drawing figures,Attorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP those skilled in the art will understand the concepts of the disclosure and will recognize applications of these concepts not particularly addressed herein. It should be understood that these concepts and applications fall within the scope of the disclosure and the accompanying claims.

[0051] It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present disclosure. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0052] It will be understood that when an element such as a layer, region, or substrate is referred to as being “on” or extending “onto” another element, it can be directly on or extend directly onto the other element or intervening elements may also be present. In contrast, when an element is referred to as being “directly on” or extending “directly onto” another element, there are no intervening elements present. Likewise, it will be understood that when an element such as a layer, region, or substrate is referred to as being “over” or extending “over” another element, it can be directly over or extend directly over the other element or intervening elements may also be present. In contrast, when an element is referred to as being “directly over” or extending “directly over” another element, there are no intervening elements present. It will also be understood that when an element is referred to as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being “directly connected” or “directly coupled” to another element, there are no intervening elements present.

[0053] Relative terms such as “below” or “above” or “upper” or “lower” or “horizontal” or “vertical” may be used herein to describe a relationship of one element, layer, or region to another element, layer, or region as illustrated in the Figures. It will be understood that these terms and those discussed above are intended to encompass different orientations of the device in addition to the orientation depicted in the Figures.

[0054] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including” when used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude theAttorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0055] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Claims

Attorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP What is Claimed is:

1. A method for managing a software interface for one or more cameras, the method comprising: instantiating, by a single thread of a scheduler of a real-time operating system executed by a hardware processor, a sensor object with a configuration file; launching, by the single thread, an asynchronous pipeline; configuring, by the single thread, each of the one or more cameras; requesting, by the single thread, frame data from each of the one or more cameras; and providing, by the single thread, a semaphore to the frame data to protect the frame data.

2. The method of claim 1, further comprising storing the frame data with the semaphore in a data cache.

3. The method of claim 2, further comprising overwriting the frame data with the semaphore that is stored in the data cache with new data acquired from one camera of the one or more cameras.

4. The method of claim 1, wherein the scheduler is configured to request data from each of the one or more cameras at system clock speed or at a down-sampled rate.

5. The method of claim 1, wherein the frame data comprises time-synchronized image data and / or video data and associated geographic data and / or kinematic-related data.

6. The method of claim 5, wherein the image data and / or the video data comprises infrared image or infrared video obtained from one or more infrared spectrum capable cameras and / or one or more visible spectrum capable cameras.

7. The method of claim 5, wherein the geographic data and / or kinematic-related data are obtained from an embedded Global Positioning System (GPS) + Inertial Navigation System (INS) Embedded GPS (EGI) (“embedded GPS + INS (EGI)”) system.

8. A computer system comprising: a hardware processor; andAttorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP a storage media storing instructions that when executed by the hardware processor performs a method for managing a software interface for one or more cameras, the method comprising: instantiating, by a single thread of a scheduler of a real-time operating system executed by a hardware processor, a sensor object with a configuration file; launching, by the single thread, an asynchronous pipeline; configuring, by the single thread, each of the one or more cameras; requesting, by the single thread, frame data from each of the one or more cameras; and providing, by the single thread, a semaphore to the frame data to protect the frame data.

9. The computer system of claim 8, wherein the method further comprising storing the frame data with the semaphore in a data cache.

10. The computer system of claim 9, the method further comprising overwriting the frame data with the semaphore that is stored in the data cache with new data acquired from one camera of the one or more cameras.

11. The computer system of claim 8, wherein the scheduler is configured to request data from each of the one or more cameras at system clock speed or at a down-sampled rate.

12. The computer system of claim 8, wherein the frame data comprises time-synchronized image data and / or video data and associated geographic data and / or kinematic-related data.

13. The computer system of claim 12, wherein the image data and / or the video data comprises infrared image or infrared video obtained from one or more infrared spectrum capable cameras and / or one or more visible spectrum capable cameras.

14. The computer system of claim 12, wherein the geographic data and / or kinematic-related data are obtained from an embedded Global Positioning System (GPS) + Inertial Navigation System (INS) Embedded GPS (EGI) (“embedded GPS + INS (EGI)”) system.

15. A non-volatile storage media that stores instructions that when executed by a hardware processor performs a method for managing a software interface for one or more cameras, the method comprising:Attorney Docket No.0192.0446-PCT Client Ref. No.24-0547-US-PSP instantiating, by a single thread of a scheduler of a real-time operating system executed by a hardware processor, a sensor object with a configuration file; launching, by the single thread, an asynchronous pipeline; configuring, by the single thread, each of the one or more cameras; requesting, by the single thread, frame data from each of the one or more cameras; and providing, by the single thread, a semaphore to the frame data to protect the frame data.

16. The non-volatile storage media of claim 15, wherein the method further comprises storing the frame data with the semaphore in a data cache.

17. The non-volatile storage media of claim 16, wherein the method further comprises overwriting the frame data with the semaphore that is stored in the data cache with new data acquired from one camera of the one or more cameras.

18. The non-volatile storage media of claim 15, wherein the scheduler is configured to request data from each of the one or more cameras at system clock speed or at a down-sampled rate.

19. The non-volatile storage media of claim 15, wherein the frame data comprises time- synchronized image data and / or video data and associated geographic data and / or kinematic- related data.

20. The non-volatile storage media of claim 19, wherein the image data and / or the video data comprises infrared image or infrared video obtained from one or more infrared spectrum capable cameras and / or one or more visible spectrum capable cameras.

Images