Operation threat detection system and method

Abstract

The invention relates to an operation threat detection system and method for mobile assets, and the system comprises a mobile asset airborne data collection and recording system, the system includes an event detector, a gyroscope, a global positioning system, and a digital video recorder adapted to receive data obtained from at least one of onboard at least one mobile asset and off-board of the at least one mobile asset; and a processing component adapted to obtain at least one media file from the at least one mobile asset and process the at least one media file to determine threat detection, critical event detection, and non-compliance applicable mobile asset regulations.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 316,343, filed on March 3, 2022, and claims priority to U.S. Non-Provisional Application No. 18 / 117,260, filed on March 3, 2023, to the extent allowed by law, and the contents of said applications are incorporated herein by reference in their entirety.

[0003] The entire disclosure of each of the above applications and the following applications is incorporated herein by reference in its entirety: U.S. Provisional Application No. 62 / 680,907, filed on June 5, 2018; U.S. Non-Provisional Application No. 16 / 431,466, filed on June 4, 2019; U.S. Provisional Application No. 62 / 825,943, filed on March 29, 2019; U.S. Provisional Application No. 62 / 829,730, filed on April 5, 2019; U.S. Non-Provisional Application No. 16 / 833,590, filed on March 28, 2020; U.S. Provisional Application No. 63 / 061,548, filed on August 5, 2020; U.S. Provisional Application No. 63 / 061,548, filed on August 5, 2021 U.S. Non-Provisional Application No. 17 / 394,135, filed on April 4, 2011; U.S. Provisional Application No. 63 / 110,628, filed on November 6, 2020; U.S. Non-Provisional Application No. 17 / 518,868, filed on November 4, 2021; U.S. Non-Provisional Application No. 11 / 464,095, filed on August 11, 2006, now U.S. Patent No. 7,953,425, issued on May 31, 2011; U.S. Provisional Application No. 61 / 624,142, filed on April 13, 2012; U.S. Non-Provisional Application No. 13 / 861,826, filed on April 12, 2013, now U.S. Patent No. 7,953,425, issued on March 15, 2016 U.S. Patent No. 9,285,294 filed on January 29, 2015; U.S. Nonprovisional Application No. 14 / 608,423 filed on January 29, 2015, now U.S. Patent No. 9,285,295 issued on March 15, 2016; U.S. Nonprovisional Application No. 14 / 996,925 filed on January 15, 2016, now U.S. Patent No. 9,915,535 issued on March 13, 2018; PCT Application No. PCT / US2013 / 36408 filed on April 12, 2013; U.S. Provisional Application No. 61 / 875,737 filed on September 10, 2013; U.S. Provisional Application No. 14 / 481,2 filed on September 9, 2014 U.S. Nonprovisional Application No. 90, now U.S. Patent No. 9,648,456, issued on May 9, 2017; U.S. Nonprovisional Application No. 15 / 473,870, filed on March 30, 2017, now U.S. Patent No. 9,913,087, issued on March 6, 2018; PCT Application No. PCT / US2014 / 54768, filed on September 9, 2014; U.S. Provisional Application No. 62 / 337,227, filed on May 16, 2016; U.S. Nonprovisional Application No. 15 / 595,650, filed on May 15, 2017, now U.S. Patent No. 9,934,623, issued on April 3, 2018;U.S. Non-Provisional Application No. 15 / 907,486, filed February 28, 2018, now U.S. Patent No. 10,445,951, issued October 15, 2019; PCT Application No. PCT / US2017 / 32961, filed May 16, 2017; U.S. Provisional Application No. 62 / 337,225, filed May 16, 2016; U.S. Non-Provisional Application No. 15 / 595,689 filed on September 10, 2019, now U.S. Patent No. 10,410,441 issued on September 10, 2019; U.S. Non-Provisional Application No. 16 / 385,745 filed on April 16, 2019, now U.S. Patent No. 11,055,935 issued on July 6, 2021; PCT / US2017 / 329 filed on May 16, 2017 PCT Application No. 68; U.S. Provisional Application No. 62 / 337,228, filed May 16, 2016; U.S. Non-Provisional Application No. 15 / 595,712, filed May 15, 2017, now U.S. Patent No. 10,392.038, issued August 27, 2019; PCT Application No. PCT / US2017 / 32971, filed May 16, 2017; PCT Application No. PCT / US2019 / 35606, filed on June 5, 2019; PCT Application No. PCT / US2020 / 25609, filed on March 29, 2020; PCT Application No. PCT / US2021 / 44733, filed on August 5, 2021; and PCT Application No. PCT / US2021 / 58177, filed on November 5, 2021. All patent applications, patents, and printed publications cited herein are incorporated herein by reference in their entirety, except for any definitions, subject matter disclaimers, or disclaimers, and unless the incorporated material is inconsistent with the explicit disclosure herein, in which case the language of the disclosure controls. ; Technical Field

[0004] The present disclosure relates to equipment for use in high-value assets, and in particular, to operational threat detection systems and methods for use in high-value mobile assets. Background Art

[0005] High-value mobile assets (such as locomotives, aircraft, mass transit systems, mining equipment, transportable medical equipment, cargo, marine vessels, and military vessels) often utilize onboard data acquisition and recording "black box" systems and / or "event recorder" systems. These data acquisition and recording systems (such as event data recorders or flight data recorders) record various system parameters for accident investigations, crew performance evaluations, fuel efficiency analysis, maintenance planning, and predictive diagnostics. Typical data acquisition and recording systems include digital and analog inputs as well as pressure switches and pressure transducers that record data from various onboard sensor devices. The recorded data may include parameters such as speed, distance traveled, location, fuel quantity, engine revolutions per minute (RPM), fluid level, operator control, pressure, current and forecast weather conditions, and environmental conditions. In addition to basic event and operational data, video and audio event / data recording capabilities are also deployed on many of these same mobile assets. Typically, after an accident involving an asset occurs and an investigation is required, data is extracted from the data recorder once it has been recovered. Certain situations may arise where the data recorder cannot be recovered or the data is otherwise unavailable. In these situations, data collected by the data collection and recording system, such as event and operational data, video data, and audio data, may be immediately needed, regardless of whether physical access to the data collection and recording system or the data is available. Data associated with the mobile asset, such as data collected by the data collection and recording system, may be used to determine compliance with domestic federal regulations and / or foreign regulations for the mobile asset. Summary of the invention

[0006] The present disclosure generally relates to real-time data collection and recording systems and operational threat detection systems for use in high-value mobile assets. The teachings herein can provide real-time or near real-time access to data (e.g., event and operational data, video data, and audio data) recorded by a real-time data collection and recording system on a high-value mobile asset. One embodiment of a method for identifying at least one of a configurable predefined threat, a configurable predefined event, and non-compliance with applicable mobile asset regulations involving at least one mobile asset includes: receiving data using a data collection and recording system onboard the mobile asset based on at least one data signal from at least one data source onboard the mobile asset and at least one data source remote from the mobile asset, the data being collected during a configurable predetermined time frame; identifying at least one of a threat, a critical event, and non-compliance with applicable mobile asset regulations using an artificial intelligence model; correlating the threat, the critical event, and at least one of the non-compliance with applicable mobile asset regulations within and before and after at least one of a configurable predetermined amount of time before the time frame, a configurable predetermined amount of time after the time frame, and a configurable predetermined amount of time during the time frame; and displaying audit results based on the data on a dashboard of the mobile asset.

[0007] One embodiment of a system for identifying at least one of threat detection, critical event detection, and non-compliance with applicable mobile asset regulations involving at least one mobile asset comprises: a data acquisition and recording system onboard the at least one mobile asset, the data acquisition and recording system comprising an event detector, a gyroscope, a global positioning system, and a digital video record adapted to receive data obtained from at least one of onboard the at least one mobile asset and outside the at least one mobile asset; and a processing component adapted to obtain at least one multimedia file from the at least one mobile asset and process the at least one multimedia file to determine at least one of threat detection, critical event detection, and non-compliance with the applicable mobile asset regulations.

[0008] These and other variations of the present disclosure are described in greater detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The description herein refers to the accompanying drawings, wherein like reference numerals refer to like parts throughout the several views, and wherein:

[0010] Figure 1 A field implementation of a first embodiment of an exemplary real-time data acquisition and recording system according to an embodiment of the present disclosure is shown;

[0011] Figure 2 shows a field implementation of a second embodiment of an exemplary real-time data acquisition and recording system according to an embodiment of the present disclosure;

[0012] Figure 3 is a flow chart of a process for recording data and / or information from a mobile asset according to an embodiment of the present disclosure;

[0013] Figure 4 is a flow chart of a process for appending data and / or information from a mobile asset after a power outage according to an embodiment of the present disclosure;

[0014] Figure 5 is a diagram showing exemplary temporary record blocks and complete record blocks saved to a crash-hardened memory module according to an embodiment of the present disclosure;

[0015] Figure 6 is a diagram illustrating an exemplary temporary recording block in a crash-hardened memory module before power interruption and after power restoration according to an embodiment of the present disclosure;

[0016] Figure 7 is a diagram showing exemplary recorded segments in a crash-hardened memory module after power has been restored according to an embodiment of the present disclosure;

[0017] Figure 8 A field implementation of a first embodiment of a real-time data acquisition and recording system viewer according to an embodiment of the present disclosure is shown;

[0018] Fig. 9 is a flow chart of a process for recording video data, audio data and / or information from a mobile asset according to an embodiment of the present disclosure;

[0019] Fig.10 is a flow chart of a process for recording video data, audio data and / or information from a mobile asset according to an embodiment of the present disclosure;

[0020] Fig.11 is a flow chart illustrating an exemplary fisheye view of a 360 degree camera of a real-time data acquisition and recording system viewer according to an embodiment of the present disclosure;

[0021] Fig.12 is a diagram showing an exemplary panoramic view of a 360-degree camera of a real-time data acquisition and recording system viewer according to an embodiment of the present disclosure;

[0022] Fig.13 is a diagram showing exemplary four views of a 360-degree camera of a real-time data acquisition and recording system viewer according to an embodiment of the present disclosure;

[0023] Fig.14is a diagram showing an exemplary rectified view of a 360 degree camera of a real-time data acquisition and recording system viewer according to an embodiment of the present disclosure;

[0024] Fig.15 A field implementation scheme of a first embodiment of a data acquisition and recording system video content analysis system according to an embodiment of the present disclosure is shown;

[0025] Fig.16A is a diagram illustrating exemplary track detection according to an embodiment of the present disclosure;

[0026] Fig. 16B is a diagram showing exemplary track detection and turnout detection according to an embodiment of the present disclosure;

[0027] Fig. 16C is a diagram illustrating exemplary track detection, track number counting, and signal detection according to an embodiment of the present disclosure;

[0028] Fig.16D is a diagram showing exemplary intersection and track detection according to an embodiment of the present disclosure;

[0029] Fig.16E is a diagram illustrating exemplary dual overhead signal detection according to an embodiment of the present disclosure;

[0030] Fig.16F is a diagram illustrating exemplary multi-track detection according to an embodiment of the present disclosure;

[0031] Figure 16G is a diagram showing an exemplary turnout and track detection according to an embodiment of the present disclosure;

[0032] Fig.16H is a diagram illustrating exemplary turnout detection according to an embodiment of the present disclosure;

[0033] Fig.17 is a flow chart of a process for determining an internal state of a mobile asset according to an embodiment of the present disclosure;

[0034] Fig.18 is a flow chart of a process for determining object detection and obstacle detection occurring external to a mobile asset according to an embodiment of the present disclosure;

[0035] Fig.19 is a flow chart of a process for sharing data and / or information from a mobile asset through a web browser or a virtual reality device according to an embodiment of the present disclosure;

[0036] Fig. 20 shows a field implementation of a seventh embodiment of an exemplary real-time data acquisition and recording system according to an embodiment of the present disclosure;

[0037] Fig.21 is a diagram illustrating exemplary signal detection of an automated signal compliance monitoring and alerting system according to an embodiment of the present disclosure;

[0038] Fig. 22 is a flow chart of a first example of a process for determining signal compliance according to an embodiment of the present disclosure;

[0039] FIG23A is a diagram showing a prior art asset setup showing how components are connected via a vehicle bus;

[0040] FIG23B is a diagram showing a prior art monitoring and diagnostic system (MDS) installed on an asset;

[0041] FIG23C is a diagram of a prior art train operator display (TOD) installed in an asset;

[0042] FIG23D is a diagram showing information relay between a mobile asset and a control center in the prior art;

[0043] Fig.24 is a diagram of a first embodiment of a connection diagnostic system according to an embodiment of the present disclosure, showing information relay between a mobile asset, a control center, and a remote user;

[0044] Fig.25A is a diagram of a first embodiment of a connection diagnostic system according to an embodiment of the present disclosure, showing a configuration of multiple devices on a mobile asset;

[0045] Fig.25B is a diagram of a first embodiment of a connected diagnostic system according to an embodiment of the present disclosure, showing a configuration of multiple devices on an asset, showing an example of an onboard display, and an example of a remote view of the onboard display;

[0046] Fig.26A is a diagram of a first embodiment of a connection diagnostic system according to an embodiment of the present disclosure, showing a plurality of assets connected in a train;

[0047] Fig.26B is a diagram of a first embodiment of a connected diagnostic system according to an embodiment of the present disclosure, showing an example of a plurality of assets connected in a train and an onboard display and / or human machine interface (HMI) of one of the assets;

[0048] Fig.27A is a perspective view of an example of an onboard display and / or human machine interface (HMI) of a first embodiment of a connection diagnostic system according to an embodiment of the present disclosure;

[0049] Fig.27Bis a perspective view of an example of an onboard display and / or human machine interface (HMI) of a first embodiment of a connection diagnostic system according to an embodiment of the present disclosure;

[0050] Fig.27C is a perspective view of an example of a remote view of an onboard display and / or a human machine interface (HMI) of a first embodiment of a connected diagnostic system according to an embodiment of the present disclosure;

[0051] Fig.28A is a diagram of a first illustrative exemplary embodiment of an asset setup including a first embodiment of a connected diagnostic system according to an embodiment of the present disclosure, showing how components are connected via an onboard asset communication channel and an offboard communication channel;

[0052] Fig.28B is a diagram of a second illustrative exemplary embodiment of an asset setup including a first embodiment of a connection diagnostic system and a catenary camera according to an embodiment of the present disclosure, showing how the components are connected via an onboard asset communication channel and an offboard communication channel;

[0053] Fig.28C is a diagram of a third illustrative exemplary embodiment of an asset setup including the first embodiment of a connected diagnostic system without a catenary camera according to an embodiment of the present disclosure, showing how the components are connected via an onboard asset communication channel and an offboard communication channel;

[0054] Fig.28D is a diagram of an example of a remote view of an onboard display and / or a human machine interface (HMI) of a first embodiment of a connection diagnostic system according to an embodiment of the present disclosure;

[0055] FIG29 is a diagram of an existing workflow of a prior art vehicle operation;

[0056] Fig.30 is a diagram of a workflow of vehicle operation of a first embodiment of a connected diagnostic system according to an embodiment of the present disclosure;

[0057] Fig.31A The onboard display and / or human-machine interface (HMI) of the first embodiment of the connection diagnosis system according to the embodiment of the present disclosure is Fig.28A A diagram of an example of an operational screen display of an asset showing real-time signals;

[0058] Fig.31B A remote platform of a first embodiment of a connection diagnosis system according to an embodiment of the present disclosure Fig.28A Figure 1 shows an example of a download screen display of an error log for an asset;

[0059] Fig.32AA first embodiment of a connection diagnosis system according to an embodiment of the present disclosure is provided. Fig.28A Figures of examples of operational screen displays of an asset that may be displayed on an onboard display and / or a human machine interface (HMI) and / or a remote view of an onboard display and / or HMI on a remote network platform;

[0060] Fig.32B A first embodiment of a connection diagnosis system according to an embodiment of the present disclosure is provided. Fig.28A Figure 1 shows an example of a fault screen display of an asset that may be displayed on an onboard display and / or a human machine interface (HMI) and / or a remote view of an onboard display and / or HMI on a remote network platform;

[0061] Fig.32C A first embodiment of a connection diagnosis system according to an embodiment of the present disclosure is provided. Fig.28A Figure 1 shows an example of a fault screen display of an asset that may be displayed on an onboard display and / or a human machine interface (HMI) and / or a remote view of an onboard display and / or HMI on a remote network platform;

[0062] Fig.32D A first embodiment of a connection diagnosis system according to an embodiment of the present disclosure is provided. Fig.28B Figures of examples of maintenance screen displays for assets showing access screens that may be displayed on an onboard display and / or a human machine interface (HMI) and / or a remote view of an onboard display and / or HMI on a remote network platform;

[0063] Fig.32E A first embodiment of a connection diagnosis system according to an embodiment of the present disclosure is provided. Fig.28A Figure 1 is an example of a maintenance screen display for an asset showing a system status screen that may be displayed on an onboard display and / or a human machine interface (HMI) and / or a remote view of an onboard display and / or HMI on a remote network platform;

[0064] Fig.32F A first embodiment of a connection diagnosis system according to an embodiment of the present disclosure is provided. Fig.28A Figure 1 is an example of a maintenance screen display for an asset showing a subsystem status screen that may be displayed on an onboard display and / or a human machine interface (HMI) and / or a remote view of an onboard display and / or HMI on a remote network platform;

[0065] Figure 32G A first embodiment of a connection diagnosis system according to an embodiment of the present disclosure is provided. Fig.28AA diagram of an example of a maintenance screen display of an asset showing on-demand data from a real-time signal, which diagram may be displayed on an onboard display and / or a human machine interface (HMI) and / or a remote view of an onboard display and / or HMI on a remote network platform;

[0066] Fig.32H A first embodiment of a connection diagnosis system according to an embodiment of the present disclosure is provided. Fig.28B An illustration of an example of a speedometer screen display of an asset that may be displayed on an onboard display and / or a human machine interface (HMI) and / or a remote view of an onboard display and / or HMI on a remote network platform;

[0067] Fig.32I is a diagram of an example of a setup screen display of a fourth illustrative exemplary embodiment of an asset including the first embodiment of a connected diagnostic system according to an embodiment of the present disclosure, which may be displayed on an onboard display and / or a human machine interface (HMI) and / or a remote view of an onboard display and / or HMI on a remote network platform;

[0068] Fig.32J is a diagram of an example of a setup screen display of a fourth illustrative exemplary embodiment of an asset including the first embodiment of a connected diagnostic system showing a brightness toggle radio button, which may be displayed on an onboard display and / or a human machine interface (HMI) and / or a remote view of an onboard display and / or HMI on a remote network platform in accordance with an embodiment of the present disclosure;

[0069] Figure 32K is a diagram of an example of an operation screen display of a fourth illustrative exemplary embodiment of an asset including the first embodiment of a connected diagnostic system according to an embodiment of the present disclosure, which may be displayed on an onboard display and / or a human machine interface (HMI) and / or a remote view of an onboard display and / or HMI on a remote network platform;

[0070] Figure 32L is a diagram of an example of an operation screen display of a fourth illustrative exemplary embodiment of an asset including the first embodiment of a connected diagnostic system according to an embodiment of the present disclosure, which may be displayed on an onboard display and / or a human machine interface (HMI) and / or a remote view of an onboard display and / or HMI on a remote network platform;

[0071] Figure 32M A first embodiment of a connection diagnosis system according to an embodiment of the present disclosure is provided. Fig.28A Figures of examples of operational screen displays of an asset that may be displayed on an onboard display and / or a human machine interface (HMI) and / or a remote view of an onboard display and / or HMI on a remote network platform;

[0072] Figure 32N A first embodiment of a connection diagnosis system according to an embodiment of the present disclosure is provided. Fig.28A Figures of examples of downloaded screen displays of assets showing real-time data that may be displayed on an onboard display and / or human machine interface (HMI) and / or a remote view of an onboard display and / or HMI on a remote network platform;

[0073] FIG33 is a diagram of existing technology in a prior art vehicle;

[0074] Fig.34 is a flow chart of a process for transmitting data between an onboard system and an offboard system in real time according to an embodiment of the present disclosure;

[0075] Fig.35 is a flow chart of a process for communicating historical data between an onboard system and an offboard system according to an embodiment of the present disclosure;

[0076] Fig.36 is a flow chart of a process for executing a user request from a remote platform to an onboard system according to an embodiment of the present disclosure;

[0077] Fig.37 is a flow chart of a process for integrating data with a third-party device according to an embodiment of the present disclosure;

[0078] Fig.38 is a flow chart of a process for controlling a mobile asset to remotely randomly download at least one media file from at least one selected service according to an embodiment of the present disclosure;

[0079] Fig.39 is a flow chart of a process for remotely narrowing down a list of controlled mobile assets in at least one selected service according to an embodiment of the present disclosure;

[0080] Fig.40 is a flow chart of a process for remotely screening a list of controlled mobile assets in at least one selected service according to an embodiment of the present disclosure;

[0081] Fig.41 is a flow chart of a process for controlling the transfer of at least one media file from each selected service to a cloud storage device according to an embodiment of the present disclosure;

[0082] Fig.42 is a flow chart of a process for detecting threats and / or critical events in at least one downloaded media file in a cloud storage device according to an embodiment of the present disclosure;

[0083] Fig.43is a flow chart of a process for detecting mobile phone usage in at least one downloaded media file in a cloud storage device according to an embodiment of the present disclosure;

[0084] Fig.44 is a flow chart of a process for detecting camera obstructions in at least one downloaded media file in a cloud storage device according to an embodiment of the present disclosure;

[0085] Fig.45 is a flow chart of a process for detecting an intersection fault in at least one downloaded media file in a cloud storage device according to an embodiment of the present disclosure; and

[0086] Fig.46 is a flow chart of a process for associating a critical event detected in at least one downloaded media file in a cloud storage device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0087] A first embodiment of a real-time data acquisition and recording system described herein provides remotely located users (e.g., asset owners, operators, and investigators) with real-time or near real-time access to a wide range of data related to high-value assets (e.g., event and operational data, video data, and audio data). The data acquisition and recording system records data related to the asset via a data recorder and streams the data to a remote data repository and remotely located users before, during, and after an incident. The data is streamed to the remote data repository in real-time or near real-time, making the information available at least at the time of an incident or emergency, thereby virtually eliminating the need to locate and download a "black box" to investigate an incident involving the asset, and eliminating the need to interact with the data recorder on the asset to request downloads of specific data, locate and transfer files, and use custom applications to view the data. The system of the present disclosure retains typical recording capabilities and adds the ability to stream data to a remote data repository and remote terminal users before, during, and after an incident. In the vast majority of cases, the information recorded in the data recorder is redundant and unnecessary because the data has already been collected and stored in the remote data repository.

[0088] Prior to the system of the present disclosure, data was extracted from a "black box" or "event recorder" after an accident had occurred and an investigation was required. Data files containing time segments recorded by the "black box" must be downloaded and retrieved from the "black box" and then viewed by the user with dedicated software. The user must obtain physical or remote access to the asset, select the desired data to be downloaded from the "black box", download the file containing the desired information to a computing device, and use a custom application operating on the computing device to locate the appropriate file with the desired data. The system of the present disclosure has eliminated the need for the user to perform these steps, requiring only the user to use a general-purpose web browser to navigate to the desired data. Remotely located users can access a general-purpose web browser to navigate to the desired data associated with a selected asset or multiple assets to view and analyze the operational efficiency and safety of the assets in real time or near real time.

[0089] Remotely located users (e.g., asset owners, operators, and / or investigators) can access a universal web browser to navigate to real-time and / or historical desired data related to a selected asset to view and analyze the asset's operational efficiency and safety in real-time or near real-time. The ability to view operations in real-time or near real-time enables rapid assessment and adjustment of actions. During an incident, for example, real-time information and / or data can facilitate classification of the situation and provide valuable information to first responders. During normal operations, for example, near real-time information and / or data can be used to audit crew performance and aid in network-wide situational awareness.

[0090] The data may include, but is not limited to: analog and frequency parameters, such as speed, pressure, temperature, current, voltage, and acceleration originating from the asset and / or nearby assets; Boolean data, such as switch position, actuator position, warning light illumination, and actuator commands; Global Positioning System (GPS) data and / or Geographic Information System (GIS) data, such as position, speed, and altitude; internally generated information, such as the regulatory speed limit of the asset given its current location; video and image information from cameras located at various locations in, on, or near the asset; audio information from microphones located at various locations in, on, or near the asset; information sent from a data center about the asset's operating plan, such as route, schedule, and cargo manifest information; information about environmental conditions in the area in which the asset is currently operating or planned to operate, including current and forecast weather conditions; asset control status and operating data generated by systems, such as Positive Train Control (PTC) in a locomotive; and data derived from a combination of any of the above, including, but not limited to, additional data, video and audio analysis, and analytics.

[0091] Figure 1 and 2Field implementations of first and second embodiments of exemplary real-time data acquisition and recording systems (DARS) 100, 200, respectively, in which aspects of the present disclosure may be implemented are shown. DARS 100, 200 is a system that delivers real-time information from a data recording device to a remotely located end user. DARS 100, 200 includes a data recorder 154, 254 mounted on a vehicle or mobile asset 148, 248 and communicating with any number of various information sources through any combination of onboard wired and / or wireless data links 170, 270 (e.g., wireless gateways / routers), or communicating with off-board information sources via a data center 150, 250 of DARS 100, 200 via a data link (e.g., wireless data link 146). Data recorder 154, 254 includes an onboard data manager 120, 220, a data encoder 122, 222, a vehicle event detector 156, 256, a queue repository 158, 258, and a wireless gateway / router 172, 272. Additionally, in this implementation, the data recorder 154, 254 may include a crash-hardened memory module 118, 218 and / or an Ethernet switch 162, 262 with or without Power over Ethernet (POE). Exemplary hardened memory modules 118, 218 may be, for example, a crash-resistant event recorder memory module that complies with federal regulations and / or Federal Railroad Administration regulations; a crash-survivable memory unit that complies with federal regulations and / or Federal Aviation Administration regulations; a crash-hardened memory module that complies with any applicable federal regulations; or any other suitable hardened memory device as known in the art. Figure 2 In the second embodiment shown in , the data logger 254 may further include an optional non-crash-hardened removable storage device 219 .

[0092] The wired and / or wireless data links 170, 270 may include any or a combination of discrete signal inputs, standard or proprietary Ethernet, serial connections, and wireless connections. The Ethernet connection may utilize the Ethernet switch 162, 262 of the data logger 154, 254 and may utilize POE. The Ethernet switch 162, 262 may be internal or external and may support POE. Additionally, data from a remote data source (e.g., Figure 1 and 2 Data of the map component 164, 264, route / crew list component 124, 224 and weather component 126, 226 in the implementation scheme can be used from the data center 150, 250 to the onboard data manager 120, 220 and the vehicle event detector 156, 256 via the wireless data link 146, 246 and the wireless gateway / router 172, 272.

[0093] The data recorder 154, 254 collects data or information from a variety of sources (which can vary greatly based on the configuration of the asset) through the onboard data link 170, 270. The data encoder 122, 222 encodes at least a minimum data set, typically defined by a regulatory agency. In this embodiment, the data encoder 122, 222 receives data from a variety of asset 148, 248 sources and data center 150, 250 sources. The information sources may include any number of components in the assets 148, 248, such as any of the following: analog inputs 102, 202, digital inputs 104, 204, I / O modules 106, 206, vehicle controllers 108, 208, engine controllers 110, 210, inertial sensors 112, 212, global positioning systems (GPS) 114, 214, cameras 116, 216, active train control (PTC) / signal data 166, 266, fuel data 168, 268, cellular transmission detectors (not shown), internal drive data, and any additional data signals, and any number of components in the data centers 150, 250, such as any of the route / crew list components 124, 224, weather components 126, 226, map components 164, 264, and any additional data signals. The data encoder 122, 222 compresses or encodes the data and time synchronizes the data to facilitate efficient real-time transmission and replication to the remote data repository 130, 230. The data encoder 122, 222 transmits the encoded data to the onboard data manager 120, 220, which then stores the encoded data in the collision-hardened memory module 118, 218 and the queue repository 158, 258 for replication to the remote data repository 130, 230 via the remote data manager 132, 232 located in the data center 150, 250. Optionally, the onboard data manager 120, 220 may also store a third copy of the encoded data in Figure 2 The onboard data manager 120, 220 and the remote data manager 132, 232 work together to manage the data replication process. A single remote data manager 132, 232 in a data center 150, 250 can manage the replication of data from multiple assets 148, 248.

[0094] Data from the various input components and data from the in-cab audio / graphical user interface (GUI) 160, 260 is sent to the vehicle event detector 156, 256. The vehicle event detector 156, 256 processes the data to determine whether an event, accident, or other predefined situation has occurred involving the asset 148, 248. When the vehicle event detector 156, 256 detects a signal indicating the occurrence of a predefined event, the vehicle event detector 156, 256 sends the processed data of the occurrence of the predefined event to the onboard data manager 120, 220 along with supporting data surrounding the predefined event. The vehicle event detector 156, 256 detects events based on data from a variety of sources (e.g., analog inputs 102, 202, digital inputs 104, 204, I / O modules 106, 206, vehicle controllers 108, 208, engine controllers 110, 210, inertial sensors 112, 212, GPS 114, 214, cameras 116, 216, route / crew inventory components 124, 224, weather components 126, 226, map components 164, 264, PTC / signal data 166, 266, and fuel data 168, 268, which may vary based on the configuration of the asset). When the vehicle event detector 156, 256 detects an event, the detected asset event information is stored in a queuing repository 158, 258 and may be optionally presented to the crew of the asset 148, 248 via an in-cab audio / graphical user interface (GUI) 160, 260.

[0095] The onboard data manager 120, 220 also sends the data to the queue repository 158. In near real-time mode, the onboard data manager 120, 220 stores the encoded data and any event information received from the data encoder 122, 222 in the collision hardened memory module 118, 218 and in the queue repository 158, 258. Figure 2 In a second embodiment of the invention, the onboard data manager 220 may optionally store the encoded data in the non-crash hardened removable storage device 219. After accumulating five minutes of encoded data in the queue repository 158, 258, the onboard data manager 120, 220 stores the five minutes of encoded data to the remote data repository 130, 230 via the remote data manager 132, 232 in the data center 150, 250 via the wireless data link 146, 246 accessed through the wireless gateway / router 172, 272. In real-time mode, the onboard data manager 120, 220 stores the encoded data and any event information received from the data encoder 122, 222 to the crash hardened memory module 118, 218, and optionally stores the encoded data in the remote data repository 130, 230. Figure 2The data is stored in a non-crash-hardened removable storage device 219 and stored to a remote data repository 130, 230 via a remote data manager 132, 232 in a data center 150, 250 via a wireless data link 146, 246 accessed through a wireless gateway / router 172, 272. The onboard data manager 120, 220 and the remote data manager 132, 232 may communicate via various wireless communication links (e.g., Wi-Fi, cellular, satellite, and private wireless systems) utilizing the wireless gateway / router 172, 272. The wireless data link 146, 246 may be, for example, a wireless local area network (WLAN), a wireless metropolitan area network (WMAN), a wireless wide area network (WWAN), a private wireless system, a cellular telephone network, or any other means of transmitting data from the data logger 154, 254 of the DARS 100, 200 to (in this example) the remote data manager 130, 230 of the DARS 100, 200. When the wireless data connection is unavailable, data is stored in memory and queued in the queue repository 158, 258 until wireless connectivity is restored and the data copy process can continue.

[0096] In parallel with data logging, the data logger 154, 254 continuously and autonomously copies data to the remote data repository 130, 230. The copying process has two modes: real-time mode and near real-time mode. In real-time mode, data is copied to the remote data repository 130, 230 every second. In near real-time mode, data is copied to the remote data repository 130, 230 every five minutes. The rates for near real-time mode and real-time mode are configurable, and the rate for real-time mode can be adjusted to support high-resolution data by copying data to the remote data repository 130, 230 every 0.10 seconds. When the DARS 100, 200 is in near real-time mode, the onboard data manager 120, 220 queues the data in the queue repository 158, 258 before copying the data to the remote data manager 132, 232. The onboard data manager 120, 220 also replicates the vehicle event detector information queued in the queue repository 158, 258 to the remote data manager 132, 232. The near real-time mode is used during normal operation under most conditions in order to improve the efficiency of the data replication process.

[0097] The real-time mode may be initiated based on an event that occurs and is detected by a vehicle event detector 156, 256 onboard the asset 148, 248 or by a request initiated from the data center 150, 250. A typical data center 150, 250 initiated request for real-time mode is initiated when a remotely located user 152, 252 has requested real-time information from a network client 142, 242. A typical reason for the origin of real-time mode onboard an asset 148, 248 is the detection of an event or incident by the vehicle event detector 156, 256, such as an operator initiated emergency stop request, emergency braking activity, rapid acceleration or deceleration on any axis, or loss of input power to the data recorder 154, 254. When transitioning from near real-time mode to real-time mode, all data that has not yet been copied to the remote data repository 130, 230 is copied and stored in the remote data repository 130, 230, and then real-time replication is initiated. The transition between near real-time mode and real-time mode typically occurs in less than five seconds. The data logger 154, 254 reverts to near real-time mode after a predetermined amount of time has passed since the event or incident, a predetermined amount of inactivity, or when the user 152, 252 no longer desires real-time information from the asset 148, 248. The predetermined amount of time required to initiate the transition is configurable and is typically set to ten minutes.

[0098] When the data recorder 154, 254 is in real-time mode, the onboard data manager 120, 220 attempts to continuously empty its queue to the remote data manager 132, 232, thereby storing data to the crash-hardened memory module 118, 218 and optionally to the remote data manager 132, 232. Figure 2 The onboard data manager 120, 220 also sends the detected vehicle information queued in the queue repository 158, 258 to the remote data manager 132, 232.

[0099] Upon receiving data to be copied from the data logger 154, 254 and data from the map component 164, 264, the route / crew list component 124, 224, and the weather component 126, 226, the remote data manager 132, 232 stores the compressed data to the remote data repository 130, 230 in the data center 150, 250 of the DARS 100, 200. The remote data repository 130, 230 may be, for example, a cloud-based data storage device or any other suitable remote data storage device. When the data is received, a process is initiated that causes the data decoder 136, 236 to decode the most recently copied data for / from the remote data repository 130, 230 and send the decoded data to the remote event detector 134, 234. The remote data manager 132, 232 stores the vehicle event information in the remote data repository 130, 230. When the remote event detector 134, 234 receives the decoded data, it processes the decoded data to determine whether an event of interest is found in the decoded data. The decoded information is then used by the remote event detector 134, 234 to detect an event, incident, or other predefined condition occurring in the data with respect to the asset 148, 248. Upon detecting an event of interest from the decoded data, the remote event detector 134, 234 stores the event information and supporting data in the remote data repository 130, 230. When the remote data manager 132, 232 receives the remote event detector 134, 234 information, the remote data manager 132, 232 stores the information in the remote data repository 130, 230.

[0100] A remotely located user 152, 252 can access information related to a particular asset 148, 248 or multiple assets, including vehicle event detector information, using a standard network client 142, 242 (e.g., a web browser) or a virtual reality device (not shown), which in this embodiment can display thumbnail images from a selected camera. The network client 142, 242 communicates the user's 152, 252 request for information to the network server 140, 240 via the network 144, 244 using common network standards, protocols, and technologies. The network 144, 244 can be, for example, the Internet. The network 144, 244 can also be a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network, or any other means of transmitting data from the network server 140, 240 to (in this example) the network client 142, 242. The network server 140, 240 requests the desired data from the data decoder 136, 236. The data decoder 136, 236 obtains the requested data related to a particular asset 148, 248 or multiple assets from the remote data repository 130, 230 upon request to the network server 140, 240. The data decoder 136, 236 decodes the requested data and sends the decoded data to the localizer 138, 238. Localization is the process of converting data into a format desired by an end user, such as converting the data into a user's preferred language and measurement units. The localizer 138, 238 identifies the profile settings set by the user 152, 252 by accessing the network client 142, 242, and uses the profile settings to prepare the information sent to the network client 142, 242 for presentation to the user 152, 252 in the user's preferred language and measurement units, because the original encoded data and detected event information are saved to the remote data repository 130, 230 using Coordinated Universal Time (UTC) and the International System of Units (SI units). The localizer 138, 238 converts the decoded data into a format desired by the user 152, 252, such as a language and measurement unit preferred by the user 152, 252. The localizer 138, 238 sends the localized data in the format preferred by the user 152, 252 to the network server 140, 240 upon request. The network server 140, 240 then sends the localized data of the asset or multiple assets to the network client 142, 242 for viewing and analysis, providing playback and real-time display of standard video and 360-degree video. The network client 142, 242 can display and the user 152, 252 can view the data, video and audio of a single asset or the data, video and audio of multiple assets simultaneously. The network client 142, 242 can also provide synchronized playback and real-time display of data and multiple video and audio data from both standard and 360-degree video sources on, in or near an asset, nearby assets and / or a remotely located site.

[0101] Figure 3 300 for recording data and / or information from an asset 148, 248 according to an embodiment of the present disclosure. The data recorder 154, 254 receives data signals from various input components, which include physical or calculated data elements from the asset 148, 248 and the data center 150, 250, such as speed, latitude coordinates, longitude coordinates, horn detection, throttle position, weather data, map data and / or route and / or crew data 302. The data encoder 122, 222 creates a record 304 containing a series of structured bits for configuring and recording the data signal information. The encoded record is then sent to the onboard data manager 120, 220, which sequentially combines the series of records into record blocks 306 containing up to five minutes of data in chronological order. A temporary record block contains less than five minutes of data, while a full record block contains a full five minutes of data. Each record block contains all the data required to fully decode the contained signal, including data integrity checks. A record block must at least begin with a start record and end with an end record.

[0102] To ensure that all encoded signal data is stored in the collision-hardened memory module 118 and optionally in Figure 2 If the data recorder 154, 254 loses power or is subjected to extreme temperature or mechanical stress due to an impact or other catastrophic event, the onboard data manager 120, 220 stores temporary recorded blocks in the crash-hardened memory module 118 308 at a predetermined rate, and optionally in the Figure 2 In a non-crash-hardened removable storage device 219, wherein the predetermined rate is configurable and / or variable, such as Figure 5 The temporary record block is saved at least once per second, but may also be saved as frequently as once every tenth of a second. The rate at which the temporary record blocks are saved depends on the sampling rate of each signal. Each temporary record block contains a complete set of records since the last complete record block. When recording each temporary record block, the data recorder 154, 254 may store the data in the collision-hardened memory module 118, 218 and optionally in Figure 2 The non-crash-resistant removable storage device 219 alternates between two temporary storage locations to prevent the data logger 154, 254 from storing data in the Figure 2The data recorder 254 is protected from corruption or loss of data for more than one second in the event of a loss of power to the crash-hardened memory modules 118, 218 and / or the optional non-crash-hardened removable storage device 219. Each time a new temporary record block is saved to a temporary crash-hardened memory location, it will overwrite an existing temporary record block previously stored in that location.

[0103] In this embodiment, every five minutes, when the data recorder 154, 254 is in near real-time mode, the onboard data manager 120, 220 stores a complete record block containing the last five minutes of encoded signal data into Figure 7 218, and sends a copy of the complete record block to the remote data manager 132, 232 for storage in the remote data repository 130, 230 for a predetermined retention period, such as two years 310. Figure 2 The crash-hardened memory module 118, 218 and / or the optional non-crash-hardened removable storage device 219 of the data recorder 254 stores recorded fragments of the most recent recorded blocks for a specified storage duration, which in this embodiment is the federally specified duration that the data recorder 154, 254 must store operational and / or video data in the crash-hardened memory module 118, 218 with an additional 24 hour buffer and then be overwritten.

[0104] Figure 4 4 is a flow chart of a process 400 for appending data and / or information from an asset 148, 248 after a power outage according to an embodiment of the present disclosure. Once power is restored, the data logger 154, 254 identifies the last temporary record block 402 stored in one of the two temporary crash-hardened memory locations and verifies the last temporary record block 404 using the 32-bit cyclic redundancy check included in the end record of each record block. The verified temporary record block is then appended to the crash-hardened memory record segment, and the record segment (which may contain up to five minutes of data before power was lost) is sent to the remote data manager 132, 232 to be stored for a retention period 406. The encoded signal data is stored in Figure 2 The data recorder 254 of the crash-hardened memory module 118, 218 and / or the optional non-crash-hardened removable storage device 219 is stored in the circular buffer for a specified storage duration. Since the crash-hardened memory recording segments are split into multiple recording blocks, each time a complete recording block is saved to Figure 2 When the data logger 254's crash-hardened memory modules 118 , 218 and / or the optional non-crash-hardened removable storage device 219 are updated, the data logger 154 , 254 removes older record blocks when necessary to free up memory space.

[0105] Figure 6 6 is a diagram showing exemplary temporary recording blocks before power is lost to the data recorder 154, 254 and after power is restored. When the temporary recording block stored in temporary location 2 is valid at (2 / 1 / 2016 10:10:08AM) 602, that temporary recording block is appended to Figure 2 The recorded segments 702 in the crash-hardened memory module 118, 218 and / or the optional non-crash-hardened removable storage device 219 of the data recorder 254 ( Figure 7 ),like Figure 7 When the temporary record block stored in temporary location 2 is invalid at (2 / 1 / 2016 10:10:08AM), the temporary record block in temporary location 1 is verified at (2 / 1 / 2016 10:10:07AM) and if valid, it is appended to Figure 2 The recorded segments in the crash-hardened memory module 118 , 218 and / or the optional non-crash-hardened removable storage device 219 of the data recorder 254 .

[0106] Whenever any record block needs to be saved in Figure 2 When the data recorder 254 stores data in the crash-hardened memory module 118, 218 and / or the optional non-crash-hardened removable storage device 219, the recorded segment is immediately flushed to disk. Since the data recorder 154, 254 alternates between two different temporary storage locations when saving temporary record blocks, there is always one temporary storage location that will not be modified or flushed to the crash-hardened memory or non-crash-hardened removable storage device, thereby ensuring that at least one of the two temporary record blocks stored in the temporary storage location is valid and that the data recorder 154, 254 will not lose more than one second of data whenever the data recorder 154, 254 loses power. Similarly, when the data recorder 154, 254 is writing data to Figure 2 When the data logger 254's crash-hardened memory module 118, 218 and / or optional non-crash-hardened removable storage device 219 is disconnected, the data logger 154, 254 will not lose more than one tenth of a second of data every time the data logger 154, 254 loses power.

[0107] To simplify explanation, process 300 and process 400 are depicted and described as a series of steps. However, the steps according to the present disclosure may occur in various orders and / or concurrently. In addition, the steps according to the present disclosure may occur with other steps not presented and described herein. In addition, all steps shown may not be required to implement the method according to the disclosed subject matter.

[0108] A third embodiment of a real-time data acquisition and recording system and viewer described herein provides remotely located users (e.g., asset owners, operators, and investigators) with real-time or near real-time access to a wide range of data (e.g., event and operational data, video data, and audio data) for high-value assets. The data acquisition and recording system records data related to the asset via a data recorder and streams the data to a remote data repository and remotely located users before, during, and after an incident. The data is streamed to the remote data repository in real-time or near real-time, making the information available at least at the time of an incident or emergency, thereby virtually eliminating the need to locate and download a "black box" in order to investigate an incident involving the asset, and eliminating the need to interact with the data recorder on the asset to request downloads of specific data, locate and transfer files, and use custom applications to view the data. The system of the present disclosure retains typical recording capabilities and adds the ability to stream data to a remote data repository and remote terminal users before, during, and after an incident. In the vast majority of cases, the information recorded in the data recorder is redundant and unnecessary because the data has already been collected and stored in the remote data repository.

[0109] Prior to the system of the present disclosure, data was extracted from a "black box" or "event recorder" after an accident had occurred and an investigation was required. Data files containing time segments recorded by the "black box" must be downloaded and retrieved from the "black box" and then viewed by the user with dedicated software. The user must obtain physical or remote access to the asset, select the desired data to be downloaded from the "black box", download the file containing the desired information to a computing device, and use a custom application operating on the computing device to locate the appropriate file with the desired data. The system of the present disclosure has eliminated the need for the user to perform these steps, requiring only the user to use a general-purpose web browser to navigate to the desired data. Remotely located users can access a general-purpose web browser to navigate to the desired data associated with a selected asset or multiple assets to view and analyze the operational efficiency and safety of the assets in real time or near real time.

[0110] Remotely located users (e.g., asset owners, operators, and / or investigators) can access a universal web browser to navigate to real-time and / or historical desired data related to a selected asset to view and analyze the asset's operational efficiency and safety in real-time or near real-time. The ability to view operations in real-time or near real-time enables rapid assessment and adjustment of actions. During an incident, for example, real-time information and / or data can facilitate classification of the situation and provide valuable information to first responders. During normal operations, for example, near real-time information and / or data can be used to audit crew performance and aid in network-wide situational awareness.

[0111] The real-time data acquisition and recording system of the third embodiment uses at least one or any combination of an image measuring device, a video measuring device, and a range measuring device in, on, or near a mobile asset as part of the data acquisition and recording system. The image measuring device and / or the video measuring device include (but are not limited to) a long-range camera (some of which have a viewing range of up to 4,000 feet), a 360-degree camera, a fixed camera, a narrow-angle camera, a wide-angle camera, a 360-degree fisheye camera, and / or other cameras. The range measuring device includes (but is not limited to) radar and light detection and ranging ("LIDAR"). LIDAR is a survey method that measures the distance to a target by illuminating the target with a pulsed laser and measuring the reflected pulse with a sensor. Prior to the system of the present disclosure, the "black box" and / or "event recorder" did not include a 360-degree camera or other camera in, on, or near a mobile asset. The system of the present disclosure uses 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye cameras, radar, LIDAR and / or other cameras as part of the data acquisition and recording system to add the ability to use and record video, thereby providing 360-degree views, narrow views, wide views, fisheye views and / or other views in, on or near mobile assets to remote data repositories, remotely located users and investigators before, during and after an incident involving a mobile asset. The ability to view operations, 360-degree videos and / or other videos in real time or near real time enables rapid assessment and adjustment of crew behavior. Owners, operators and investigators can view and analyze the operational efficiency, safety of people, vehicles and infrastructure and can investigate or verify accidents. The ability to view 360-degree videos and / or other videos from mobile assets enables rapid assessment and adjustment of crew behavior. During an incident, for example, 360-degree videos and / or other videos can facilitate classification of situations and provide valuable information to emergency personnel and investigators. During normal operations, for example, 360-degree video and / or other video can be used to audit crew performance and aid in network-wide situational awareness. 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye cameras, radar, LIDAR and / or other cameras provide a complete situational picture to provide surveillance video for law enforcement and / or rail police, inspect critical infrastructure, monitor railroad crossings, view track work progress, crew audits and real-time remote monitoring in both the cab and the marshaling yard.

[0112] Existing systems require users to download video files containing time segments in order to view the video files using dedicated software applications or other external video playback applications. The data acquisition and recording system of the present disclosure provides 360-degree video, other video, image and audio information, and range measurement information that can be displayed to a remote user through the use of a virtual reality device and / or through a standard network client, thereby eliminating the need to download and use external applications to view the video. In addition, the remotely located user can view the 360-degree video and / or other video in various modes through the use of a virtual reality device or through a standard network client (e.g., a web browser), thereby eliminating the need to download and use external applications to view the video. Existing video systems require users to download video files containing time segments of data that can only be viewed using dedicated application software or other external video playback applications that the user must purchase separately.

[0113] The data may include, but is not limited to, video and image information from cameras positioned at various locations in, on, or near an asset and audio information from microphones positioned at various locations in, on, or near an asset. A 360 degree camera is a camera that provides a 360 degree spherical field of view, a 360 degree hemispherical field of view, and / or a 360 degree fisheye field of view. Using a 360 degree camera, fixed camera, narrow angle camera, wide angle camera, 360 degree fisheye view camera, and / or other camera in, on, or near an asset provides the ability to use and record video using a 360 degree camera, fixed camera, narrow angle camera, wide angle camera, 360 degree fisheye view camera, and / or other camera as part of a DARS, thereby making 360 degree views and / or other views in, on, or near an asset available to remote data repositories, remotely located users, and investigators before, during, and after an incident.

[0114] Figure 8A field implementation of a third embodiment of an exemplary real-time data acquisition and recording system (DARS) 800 in which aspects of the present disclosure may be implemented is shown. The DARS 800 is a system that delivers real-time information, video information, and audio information from a data recorder 808 on a mobile asset 830 to a remotely located end user via a data center 832. The data recorder 808 is mounted on a vehicle or mobile asset 830 and communicates with any number of various information sources through any combination of wired and / or wireless data links, such as a wireless gateway / router (not shown). The data recorder 808 includes a crash-hardened memory module 810, an onboard data manager 812, and a data encoder 814. In a fourth embodiment, the data recorder 808 may also include a non-crash-hardened removable storage device (not shown). An exemplary ruggedized memory module 810 may be, for example, a crash-resistant event recorder memory module that complies with federal regulations and / or Federal Railroad Administration regulations; a crash-survivable memory unit that complies with federal regulations and / or Federal Aviation Administration regulations; a crash-resistant ruggedized memory module that complies with any applicable federal regulations; or any other suitable ruggedized memory device as known in the art. The wired and / or wireless data links may include any or a combination of discrete signal inputs, standard or proprietary Ethernet, serial connections, and wireless connections.

[0115] The data recorder 808 collects video data, audio data, and other data and / or information from a variety of sources (which may vary based on the configuration of the asset) via an onboard data link. In this embodiment, the data recorder 808 receives data from a video management system 804, which continuously records video data and audio data from 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye cameras, radars, LIDARs, and / or other cameras 802 and fixed cameras 806 placed in, on, or near the asset 830, and the video management system 804 stores the video and audio data to a collision-hardened memory module 810, and may also store the video and audio data in a non-collision-hardened removable storage device of the fourth embodiment. Different versions of the video data are created using different bit rates or spatial resolutions, and the versions are separated into segments of variable lengths, such as thumbnails, five-minute low-resolution segments, and five-minute high-resolution segments.

[0116] The data encoder 814 encodes at least a minimum data set, typically defined by a regulatory agency. The data encoder 814 receives video and audio data from the video management system 804 and compresses or encodes the data and time-synchronizes the data to facilitate efficient real-time transmission and replication to a remote data repository 820. The data encoder 814 transmits the encoded data to an onboard data manager 812, which then sends the encoded video and audio data to the remote data repository 820 via a remote data manager 818 located in a data center 830 in response to an on-demand request made by a remotely located user 834 or in response to observing certain operating conditions onboard the asset 830. The onboard data manager 812 and the remote data manager 818 work in conjunction to manage the data replication process. The remote data manager 818 in the data center 832 can manage the replication of data from multiple assets. The video and audio data stored in the remote data repository 820 is available to a network server 822 for access by a remotely located user 834.

[0117] The onboard data manager 812 also sends the data to a queue repository (not shown). The onboard data manager 812 monitors the video and audio data stored in the crash-hardened memory module 810 and / or the optional non-crash-hardened removable storage device of the fourth embodiment through the video management system 804 and determines whether it is in near real-time mode or real-time mode. In near real-time mode, the onboard data manager 812 stores the encoded data (including video data, audio data, and any other data or information) and any event information received from the data encoder 814 in the crash-hardened memory module 810 and / or the optional non-crash-hardened removable storage device of the fourth embodiment and in the queue repository. After five minutes of encoded data have been accumulated in the queue repository, the onboard data manager 812 stores the five minutes of encoded data to the remote data repository 820 via the remote data manager 818 in the data center 832 through the wireless data link 816. In real-time mode, the onboard data manager 812 stores the encoded data (including video data, audio data, and any other data or information) received from the data encoder 814 and any event information to the remote data repository 820 via the remote data manager 818 in the data center 832 via the wireless data link 816 at every configurable predetermined time period (such as, for example, every second or every 0.10 seconds). The onboard data manager 812 and the remote data manager 818 can communicate via various wireless communication links. The wireless data link 816 can be, for example, a wireless local area network (WLAN), a wireless metropolitan area network (WMAN), a wireless wide area network (WWAN), a private wireless system, a cellular telephone network, or any other means that transmits data from the data recorder 808 to (in this example) the remote data manager 818. The process of remotely sending and retrieving video data and audio data from the asset 830 requires a wireless data connection between the asset 830 and the data center 832. When wireless data connection is unavailable, data is stored and queued in the crash-hardened memory module 810 and / or the optional non-crash-hardened removable storage device of the fourth embodiment until wireless connectivity is restored. As soon as wireless connectivity is restored, the video, audio and any other additional data retrieval process continues.

[0118] In parallel with data logging, the data logger 808 continuously and autonomously copies data to the remote data repository 820. The copy process has two modes: real-time mode and near real-time mode. In real-time mode, data is copied to the remote data repository 820 every second. In near real-time mode, data is copied to the remote data repository 820 every five minutes. The rates for near real-time mode and real-time mode are configurable, and the rate for real-time mode can be adjusted to support high-resolution data by copying data to the remote data repository 820 every 0.10 seconds. Near real-time mode is used during normal operation under most conditions in order to improve the efficiency of the data copy process.

[0119] The real-time mode may be initiated based on an event occurring onboard the asset 830 or by a request initiated from the data center 832. The request for real-time mode initiated by the typical data center 832 is initiated when a remotely located user 834 has requested real-time information from the network client 826. The typical reason for the origin of the real-time mode onboard the asset 830 is the detection of an event or accident, such as an operator-initiated emergency stop request, emergency braking activity, rapid acceleration or deceleration on any axis, or loss of input power to the data recorder 808. When transitioning from near real-time mode to real-time mode, all data that has not yet been copied to the remote data repository 820 is copied and stored in the remote data repository 820, and then real-time replication is initiated. The transition between near real-time mode and real-time mode typically occurs in less than five seconds. After a predetermined amount of time has passed since the event or accident, a predetermined amount of inactivity, or when the user 834 no longer expects real-time information from the asset 830, the data recorder 808 reverts to near real-time mode. The predetermined amount of time required to initiate the transition is configurable and is typically set to ten minutes.

[0120] When the data recorder 808 is in real-time mode, the onboard data manager 812 attempts to continuously empty its queue to the remote data manager 818, thereby storing data to the anti-crash hardened memory module 810 and / or the optional non-anti-crash hardened removable storage device of the fourth embodiment, and sending data to the remote data manager 818 at the same time.

[0121] Upon receiving video data, audio data, and any other data or information to be copied from the data recorder 808, the remote data manager 818 stores the data to a remote data repository 820 in the data center 830. The remote data repository 820 may be, for example, a cloud-based data storage device or any other suitable remote data storage device. When the data is received, a process is initiated that causes a data decoder (not shown) to decode the most recently copied data from the remote data repository 820 and send the decoded data to a remote event detector (not shown). The remote data manager 818 also stores vehicle event information in the remote data repository 820. When the remote event detector receives the decoded data, it processes the decoded data to determine whether an event of interest is found in the decoded data. The decoded information is then used by the remote event detector to detect an event, accident, or other predefined situation that occurred in the data regarding the asset 830. When an event of interest is detected from the decoded data previously stored in the remote data repository 820, the remote event detector stores the event information and supporting data in the remote data repository 820.

[0122] The video data, audio data, and any other data or information are made available to the user 834 in response to an on-demand request made by the user 834 and / or sent by the onboard data manager 812 to the remote data repository 820 in response to certain operating conditions observed onboard the asset 830. The video data, audio data, and any other data or information stored in the remote data repository 820 are available from the network server 822 for access by the user 834. The remotely located user 834 can access the video data, audio data, and any other data or information stored in the remote data repository 820 related to a particular asset 830 or multiple assets using a standard network client 826 (e.g., a web browser) or a virtual reality device 828 (which, in this embodiment, can display thumbnail images of a selected camera). The network client 826 communicates the user 834's request for video, audio, and / or other information to the network server 822 over the network 824 using common network standard protocols and technologies. The network 824 can be, for example, the Internet. The network 824 may also be a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network, or any other means for transmitting data from the network server 822 to (in this example) the network client 826. The network server 822 requests the desired data from the remote data repository 820. Then, the network server 822 sends the requested data to the network client 826, which provides playback and real-time display of standard video, 360-degree video, and / or other video. The network client 826 plays video data, audio data, and any other data or information for viewing and analysis for the user 834 who can interact with the 360-degree video data and / or other video data and / or still image data. The user 834 may also use the network client 826 to download video data, audio data, and any other data or information, and then, may use the virtual reality device 828 to interact with the 360-degree video data for viewing and analysis.

[0123] The network client 826 may be enhanced with software applications that provide playback of 360-degree video and / or other video in a variety of different modes. The user 834 may select a mode in which the software application presents video playback (e.g., as an example). Fig.11 The fisheye view shown in Fig.12 The panoramic view shown in the figure, the double panoramic view (not shown), Fig.13 The four views shown in Fig.14 mode (corrected view shown in ).

[0124] Fig. 9840 is a flow chart of a process 840 for recording video data, audio data, and / or information from an asset 830 according to an embodiment of the present disclosure. The video management system 804 receives data signals 842 from various input components such as 360 degree cameras, fixed cameras, narrow angle cameras, wide angle cameras, 360 degree fisheye cameras, radars, LIDARs, and / or other cameras 802 and fixed cameras 806 on, in, or near the asset 830. The video management system 804 then stores the video data, audio data, and / or information in the crash-hardened memory module 810 and / or the optional non-crash-hardened removable storage device 844 of the fourth embodiment using any combination of industry standard formats such as, for example, still images, thumbnails, still image sequences, or compressed video formats. The data encoder 814 creates a record 846 containing a series of structured bits for configuring and recording the data signal information. In near real-time mode, the video management system 804 stores video data in the collision-hardened memory module 810 and / or the optional non-collision-hardened removable storage device of the fourth embodiment, while only sending limited video data (such as thumbnails or very short low-resolution video clips) off-board to the remote data repository 820 848.

[0125] In another embodiment, the encoded record is then sent to the onboard data manager 812, which sequentially combines a series of records into record blocks containing up to five minutes of data in chronological order. A temporary record block contains less than five minutes of data, while a complete record block contains the full five minutes of data. Each record block contains all the data required to fully decode the contained signal, including data integrity checks. A record block must at least begin with a start record and end with an end record.

[0126] To ensure that all encoded signal data is saved to the crash-hardened memory module 810 and / or the optional non-crash-hardened removable storage device of the fourth embodiment, if the data recorder 808 loses power, the onboard data manager 812 stores temporary record blocks in the crash-hardened memory module 810 and / or the optional non-crash-hardened removable storage device of the fourth embodiment at a predetermined rate, wherein the predetermined rate is configurable and / or variable. The temporary record blocks are saved at least once per second, but can also be saved as frequently as once every tenth of a second. The rate at which the temporary record blocks are saved depends on the sampling rate of each signal. Each temporary record block contains a complete set of records since the last complete record block. When recording each temporary record block, the data recorder 808 can alternate between two temporary storage locations in the crash-hardened memory module 810 to prevent corruption or loss of more than one second of data if the data recorder 808 loses power while storing data to the crash-hardened memory module 810 and / or the optional non-crash-hardened removable storage device of the fourth embodiment. Whenever a new temporary record block is saved to a temporary anti-collision hardened memory location, it will overwrite the existing temporary record block previously stored in that location.

[0127] In this embodiment, every five minutes, when the data recorder 808 is in near real-time mode, the onboard data manager 812 stores a complete record block containing the last five minutes of encoded signal data into a record segment in the crash-hardened memory module 810 and / or the optional non-crash-hardened removable storage device of the fourth embodiment, and sends a copy of the complete record block (including five minutes of video data, audio data, and / or information) to the remote data manager 818 for storage in the remote data repository 820 for a predetermined retention period, such as two years. The crash-hardened memory module 810 and / or the optional non-crash-hardened removable storage device of the fourth embodiment stores the record segment of the most recent record block for a specified storage duration, which in this embodiment is the federally specified duration that the data recorder 808 must store operational or video data in the crash-hardened memory module 810 with an additional 24-hour buffer and then be overwritten.

[0128] Fig.10826 or a virtual reality device 828. When an event occurs or when a remotely located authorized user 834 requests a video data segment stored in the collision hardened memory module 810 via the web client 826, the onboard data manager 812 will send the video data off-board in real time at the best resolution available given the bandwidth of the wireless data link 816, depending on the start of the event. The remotely located user 834 initiates a request 852 for specific video and / or audio data in a specific view mode via the web client 826, which communicates the request to the web server 822 via the network 824. The web server 822 requests the specific video and / or audio data from the remote data repository 820 and sends the requested video and / or audio data to the web client 826 854 via the network 824. The web client 826 displays the video and / or audio data 856 in the view mode specified by the user 834. The user 834 may then download specific video and / or audio data for viewing on the virtual reality device 828. In another embodiment, in real-time mode, thumbnails are first sent at one-second intervals, followed by short clips of lower-resolution video, and then short clips of higher-resolution video.

[0129] To simplify explanation, process 840 and process 850 are depicted and described as a series of steps. However, the steps according to the present disclosure may occur in various orders and / or concurrently. In addition, the steps according to the present disclosure may occur with other steps not presented and described herein. In addition, all steps shown may not be required to implement the method according to the disclosed subject matter.

[0130] A fifth embodiment of a real-time data acquisition and recording system and video analytics system described herein provides remotely located users with real-time or near real-time access to a wide range of data (e.g., event and operational data, video data, and audio data) of high-value assets. The data acquisition and recording system records data related to the asset and streams the data to a remote data repository and remotely located users before, during, and after an incident. The data is streamed to the remote data repository in real-time or near real-time, making the information available at least in the event of an incident or emergency, thereby virtually eliminating the need to locate and download a "black box" by streaming the information to a remote data repository in real-time or near real-time for investigating an incident involving the asset, and making the information available at least in the event of an incident or catastrophic event. The DARS performs video analysis of the recorded video data of the mobile asset to determine, for example, cab occupancy, track detection, and object detection near the track. Remotely located users can use a general web browser to navigate to and view desired data related to a selected asset or multiple assets, and do not need to interact with the data acquisition and recording system on the asset to request downloading of specific data, locate or transfer files, and use custom applications to view data.

[0131] DARS provides remotely located users with access to video data and video analytics performed by the video analytics system by streaming the data to a remote data repository and remotely located users before, during, and after an incident, eliminating the need for users to manually download, extract, and playback video to review video data to determine cab occupancy, the presence of crew members or unauthorized personnel during an incident, track detection, object detection near the track, during an investigation, or at any other time of interest. In addition, the video analytics system provides cab occupancy status determination, track detection, object detection near the track, and lead and trailing unit determination by processing images and video data in real time, thereby ensuring that the correct data is always available to users. For example, real-time image processing ensures that a locomotive designated as a trailing locomotive is not in lead service to enhance rail safety. Existing systems provide locomotive locations within a train by using train marshaling functionality in a dispatching system. Sometimes, dispatching system information can be outdated because the information is not updated in real time, and the crew can change locomotives when necessary.

[0132] Prior to the system of the present disclosure, the inspection crew and / or asset personnel had to manually inspect the track conditions, manually check whether the vehicle was in the leading or trailing position, manually survey the location of each individual object of interest, manually create a database of the geographic locations of all objects of interest, periodically perform manual field surveys on each object of interest to verify its location and identify any changes in geographic location from the original survey, manually update the database when the object of interest changes location due to repairs or additional infrastructure development since the original database was created, select and download the desired data from the digital video recorder and / or data recorder and offline inspect the downloaded data and / or video and check the track for any obstructions, and the vehicle operator must physically check for any obstructions and / or turnout changes. The system of the present disclosure has eliminated the need for the user to perform these steps, requiring only the user to use a general web browser to navigate to the desired data. Asset owners and operators can automate and improve the efficiency and safety of mobile assets in real time, and can actively monitor track conditions and obtain warning information in real time. The system of the present disclosure also eliminates the need for asset owners and operators to download data from data recorders in order to monitor track conditions and investigate accidents. As an active safety system, DARS can help operators check for any obstacles, send alerts in real time, and / or save information offline and send alert information for remote monitoring and storage. Both current and past track detection information and / or information related to object detection near the track can be stored in real time in a remote data repository to help users view the information when needed. Remotely located users can access a universal web browser to navigate to desired data related to a selected asset or multiple assets to view and analyze the operating efficiency and safety of the assets in real time or near real time.

[0133] The real-time data acquisition and recording system of the fifth embodiment can be used to continuously monitor objects of interest and identify in real time when they are moved or damaged, become covered by leaves and / or fall into disrepair and need maintenance. DARS uses video, image and / or audio information to detect and identify various infrastructure objects (such as railroad tracks) in the video, has the ability to follow the track as the mobile asset moves forward, and has the ability to create a database of objects of interest with geographical locations, audit against the database and periodically update the database. The real-time data acquisition and recording system of the fifth embodiment uses at least one or any combination of image measurement devices, video measurement devices and range measurement devices in, on or near the mobile asset as part of the data acquisition and recording system. Image measurement devices and / or video measurement devices include (but are not limited to) 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye cameras and / or other cameras. Range measurement devices include (but are not limited to) radar and light detection and ranging ("LIDAR"). LIDAR is a survey method that measures the distance to a target by illuminating the target with a pulsed laser and measuring the reflected pulse with a sensor.

[0134] DARS can automatically check track conditions, such as counting the number of existing tracks, identifying the current track on which the mobile asset is traveling, and detecting any obstacles or defects that exist, such as ballast being washed away, track breakage, track overrun, switch misalignment, switch derailment, track flooding, snow accumulation, etc., and plan any preventive maintenance to avoid any catastrophic events. DARS can also detect rail track switches and follow track changes. DARS can further detect changes in data locations, including whether objects are missing, obscured, and / or not in the expected location. Track detection, infrastructure diagnostic information, and / or infrastructure monitoring information can be displayed to users using any standard network client (such as a web browser), thereby eliminating the need to download files from data loggers and use dedicated application software or other external applications to view information as required by existing systems. This process can be extended to automatically create, audit, and / or update databases with geographic locations of objects of interest and ensure compliance with federal regulations. Using the system of the present disclosure, cameras that were previously installed to comply with federal regulations are used to perform various tasks that previously required human interaction, specialized vehicles, and / or alternative equipment. DARS allows these tasks to be performed automatically as mobile assets travel throughout a territory as part of normal revenue service and daily operations. DARS can be used to save countless hours of manual work by utilizing the normal operation of the vehicle and previously installed cameras to complete tasks that previously required manual work. DARS can also perform tasks that were previously performed using specialized vehicles, thereby preventing the closure of track segments to inspect and locate the track and objects of interest, which often results in loss of revenue service and expensive equipment to purchase and maintain. DARS further reduces the amount of time humans are required to be within the vicinity of railroad tracks, resulting in a reduction in overall accidents and potential loss of life.

[0135] The data may include, but is not limited to: measured analog and frequency parameters, such as speed, pressure, temperature, current, voltage, and acceleration originating from the mobile asset and / or nearby mobile assets; measured Boolean data, such as switch position, actuator position, warning light illumination, and actuator commands; position, speed, and altitude information from a global positioning system (GPS) and additional data from a geographic information system (GIS), such as the latitude and longitude of various objects of interest; internally generated information, such as the regulatory speed limit of the mobile asset given its current location; train control status and operating data generated by systems such as active train control (PTC); vehicle and inertial parameters, such as speed, acceleration, and position (e.g., received from GPS); location); GIS data, such as the latitude and longitude of various objects of interest; video and image information from at least one camera positioned at various locations in, on, or near the mobile asset; audio information from at least one microphone positioned at various locations in, on, or near the mobile asset; information sent from a data center to the mobile asset regarding the mobile asset's operating plan, such as route, schedule, and cargo manifest information; information regarding environmental conditions in the area in which the mobile asset is currently operating or planned to operate, such as current and forecast weather; and data derived from a combination of any of the foregoing sources, including additional data, video and audio analysis, and analytics.

[0136] "Tracks" may include, but are not limited to, rails and sleepers of a railway used for locomotive and / or train transportation. "Objects of interest" may include, but are not limited to, various infrastructure objects installed and maintained within the vicinity of railway tracks that can be identified using artificial intelligence, such as supervised learning or reinforcement learning of asset camera images and videos. Supervised learning and / or reinforcement learning utilize previously labeled data sets defined as "training" data to allow remote and autonomous identification of objects within the field of view of a camera in, on, or near a mobile asset. Supervised learning and / or reinforcement learning trains a neural network model to recognize patterns that appear within visual images obtained from a camera. These patterns (e.g., people, intersection gates, cars, trees, signals, switches, etc.) can be found alone in a single image. Consecutive frames within a video can also be analyzed to find patterns (e.g., flashing signals, moving cars, people falling asleep, etc.). DARS may or may not require human interaction at any stage of implementation, including, but not limited to, labeling the training data sets required for supervised learning and / or reinforcement learning. Objects of interest include, but are not limited to, tracks, track centerline points, milestone signs, signals, intersection gates, switches, intersections, and text-based signs. "Video Analytics" refers to any intelligent information gathered by analyzing video and / or images recorded by an image measuring device, video measuring device, and / or range measuring device (e.g., at least one camera, such as a 360-degree camera, fixed camera, narrow-angle camera, wide-angle camera, 360-degree fisheye camera, radar, LIDAR, and / or other camera) in, on, or near a mobile asset, including, but not limited to, objects of interest, geographic locations of objects, track obstructions, distances between objects of interest and the mobile asset, track misalignment, etc. The video analytics system may also be used in any mobile asset, residential area, space, or room containing surveillance cameras to enhance video surveillance. In a mobile asset, the video analytics system provides autonomous cab occupancy event detection to remotely located users in an economical and effective manner.

[0137] Fig.15A field implementation of a fifth embodiment of an exemplary real-time data acquisition and recording system (DARS) 900 in which aspects of the present disclosure may be implemented is shown. The DARS 900 is a system that delivers real-time information, video information, and audio information from a data recorder 902 on a mobile asset 964 to a remotely located end user 968 via a data center 966. The data recorder 902 is mounted on a vehicle or mobile asset 964 and communicates with any number of various information sources through any combination of wired and / or wireless data links 942, such as a wireless gateway / router (not shown). The data recorder 902 collects video data, audio data, and other data or information from a variety of sources (which may vary based on the configuration of the asset) through the onboard data link 942. The data recorder 902 includes local memory components, such as a crash-hardened memory module 904 in the asset 964, an onboard data manager 906, and a data encoder 908. In a sixth embodiment, the data recorder 902 may also include a non-crash-hardened removable storage device (not shown). An exemplary ruggedized memory module 904 may be, for example, a crash-resistant event recorder memory module that complies with federal regulations and / or Federal Railroad Administration regulations; a crash-survivable memory unit that complies with federal regulations and / or Federal Aviation Association regulations; a crash-resistant ruggedized memory module that complies with any applicable federal regulations; or any other suitable ruggedized memory device as known in the art. The wired and / or wireless data links may include any or a combination of discrete signal inputs, standard or proprietary Ethernet, serial connections, and wireless connections.

[0138] DARS 900 further includes a video analytics system 910 that includes a track and / or object detection and infrastructure monitoring component 914. The track detection and infrastructure monitoring component 914 includes an artificial intelligence component 924 (e.g., a supervised learning and / or reinforcement learning component or other neural network or artificial intelligence component), an object detection and localization component 926, and an obstacle detection component 928 that detects obstacles on or near the track and / or camera obstacles (e.g., people blocking the camera's view). In this embodiment, real-time video data is captured by at least one camera 940 mounted in the cab of an asset 964, on the asset 964, or near the asset 964. The camera 940 is placed at an appropriate height and angle to capture video data in and around the asset 964 and to obtain a sufficient amount of view for further processing. Real-time video data and image data are captured by the camera 940 in front of and / or around the asset 964, and the real-time video data and image data are fed to the track and / or object detection and infrastructure monitoring component 914 for analysis. The track detection and infrastructure monitoring component 914 of the video analytics system 910 processes the real-time video and image data frame by frame to detect the presence of rail tracks and any objects of interest. Camera position parameters (such as height, angle, shift, focus, and field of view) can be fed to the track and / or object detection and infrastructure monitoring component 914, or the camera 940 can be configured to allow the video analytics system 910 to detect and determine the camera position and parameters.

[0139] To make state determinations, such as cab occupancy detection, the video analytics system 910 uses a supervised learning and / or reinforcement learning component 924 and / or other artificial intelligence and learning algorithms to evaluate, for example, video data from a camera 940, asset data 934 (e.g., speed, GPS data, and inertial sensor data), weather component 936 data, and route / crew list data, and GIS component data 938. Cab occupancy detection is inherently susceptible to ambient noise sources, such as light reflected from clouds and sunlight passing through buildings and trees when the asset is moving. To handle ambient noise, the supervised learning and / or reinforcement learning component 924, object detection and localization component 926, obstacle detection component, asset component 934 data (which may include speed, GPS data, and inertial sensor data), weather component 936 data, and other learning algorithms are grouped together to form internal and / or external state determinations involving the mobile asset 964. The track and / or object detection and infrastructure monitoring components 914 may also include: a facial recognition system, which in an exemplary embodiment is described here, is adapted to allow authorized access to the locomotive as part of the locomotive safety system; a fatigue detection component, which is adapted to monitor crew alertness; and an activity detection component, which is used to detect unauthorized activity, such as smoking.

[0140] Additionally, the video analytics system 910 may receive location information, including latitude and longitude coordinates, of a signal (e.g., a stop sign, a traffic sign, a speed limit sign, and / or a signal of an object near the track) from the asset owner. The video analytics system 910 then determines whether the location information received from the asset owner is correct. If the location information is correct, the video analytics system 910 stores the information and will not recheck the location information again for a predetermined amount of time, such as monthly. If the location information is incorrect, the video analytics system 910 determines the correct location information and reports the correct location information to the asset owner, stores the location information, and will not recheck the location information again for a predetermined amount of time, such as monthly. Storing the location information makes detection of signals (e.g., stop signs, traffic signs, speed limit signs, and / or signals of objects near the track) easier.

[0141] Artificial intelligence of the track, such as supervised learning and / or reinforcement learning, is performed using artificial intelligence component 924 using various information obtained from successive video and / or image frames and also using additional information received from data center 966 and vehicle data component 934, including inertial sensor data and GPS data for determining learned data. Object detection and localization component 926 utilizes learned data received from supervised learning and / or reinforcement learning component 924 and specific information about mobile assets 964 and the railway, such as, for example, track width and curvature, sleeper location, and vehicle speed to distinguish rail tracks, signs, signals, etc. from other objects to determine object detection data. Obstacle detection component 928 utilizes object detection data received from object detection and location component 926 (e.g., information about obstacles present on or near the track and / or camera obstructions (e.g., personnel blocking the camera's view)) and additional information from weather component 936, route / crew list and GIS data component 938, and vehicle data component 934 (which includes inertial sensor data and GPS data) to enhance accuracy and determine obstacle detection data. Mobile asset data from vehicle data component 934 includes, but is not limited to, speed, position, acceleration, yaw / pitch rate, and railroad track crossings. Any additional information received and utilized from data center 966 includes, but is not limited to, day and night details and geographic location of mobile asset 964.

[0142] The infrastructure objects of interest, information processed by the track and / or object detection and infrastructure monitoring component 914, and diagnostic and monitoring information are sent to the data encoder 908 of the data recorder 902 via the onboard data link 942 to encode the data. The data recorder 902 stores the encoded data in the crash-hardened memory module 904 and optionally in the optional non-crash-hardened removable storage device of the sixth embodiment, and sends the encoded information to the remote data manager 946 in the data center 966 via the wireless data link 944. The remote data manager 946 stores the encoded data in the remote data repository 948 in the data center 966.

[0143] To determine obstacle detection 928 or object detection 926, such as the presence 964 of a track in front of an asset, objects on and / or near the track, obstacles on or near the track, and / or obstacles blocking the camera's view, the vehicle analytics system 910 uses supervised learning and / or reinforcement learning components 924 or other artificial intelligence, object detection and localization components 926 and obstacle detection components 928 and other image processing algorithms to process and evaluate camera images and video data from camera 940 in real time. Track and / or object detection and infrastructure monitoring component 914 uses processed video data along with asset component 934 data (which may include speed, GPS data, and inertial sensor data), weather component 936 data, and route / crew list and GIS component 938 data to determine external state determinations, such as leading and towing mobile assets, in real time. When processing image and video data for track and / or object detection, for example, the video analytics system 910 automatically configures the camera 940 parameters required for track detection, detects travel through switches, counts the number of tracks, detects any additional tracks along the side of the asset 964, determines the track on which the asset 964 is currently traveling, detects track geometry defects, detects track washout scenarios (e.g., detects water near the track within the defined limits of the track), and detects missing slope or track scenarios. Object detection accuracy depends on the existing lighting conditions in and around the asset 964. The DARS 900 will handle different lighting conditions with the help of additional data collected from onboard the asset 964 and the data center 966. The DARS 900 is enhanced to work in various lighting conditions, work in various weather conditions, detect more objects of interest, integrate with existing database systems to automatically create, audit and update data, detect multiple tracks, work consistently with curved tracks, detect any obstacles, detect any track defects that may cause safety issues, and work in a low-cost embedded system.

[0144] Interior and / or exterior state determinations, such as cab occupancy; object detection and location, such as track detection and near-track object detection; and obstacle detection, such as obstacles on or near the track and obstacles blocking the camera, from the video analytics system 910 are provided to the data recorder 902 via the onboard data link 942, along with any data from the vehicle management system (VMS) or digital video recorder component 932. The data recorder 902 stores the interior and / or exterior state determinations, object detection and location component 926 data, and obstacle detection component 928 data in the crash-hardened memory module 904 and optionally in the non-crash-hardened removable storage device of the sixth embodiment, and in the remote data repository 948 via the remote data manager 946 located in the data center 966. The web server 958 provides the interior and / or exterior state determinations, object detection and location component 926 information, and obstacle detection component 928 information to the remotely located user 968 via the web client 962 upon request.

[0145] The data encoder 908 encodes at least a minimum data set, typically defined by a regulatory agency. The data encoder 908 receives video, image, and audio data from any of the cameras 940, the video analytics system 910, and the video management system 932, and compresses or encodes the data and synchronizes the data in time to facilitate efficient real-time transmission and replication to a remote data repository 948. The data encoder 908 transmits the encoded data to the onboard data manager 906, which then sends the encoded video, image, and audio data to the remote data repository 948 via a remote data manager 946 located in a data center 966 in response to an on-demand request made by a user 968 or in response to observing certain operating conditions onboard an asset 964. The onboard data manager 906 and the remote data manager 946 work in conjunction to manage the data replication process. The remote data manager 946 in the data center 966 can manage the replication of data from multiple assets 964.

[0146] The onboard data manager 906 determines whether the detected events, internal and / or external state determinations, object detection and location, and / or obstacle detections should be queued or sent out immediately based on the priority of the detected events. For example, in normal operating conditions, detecting an obstacle on the track is much more urgent than detecting whether a person is in the cab of the asset 964. The onboard data manager 906 also sends the data to a queue repository (not shown). In near real-time mode, the onboard data manager 906 stores the encoded data and any event information received from the data encoder 908 in the anti-collision hardened memory module 904 and in the queue repository. After five minutes of encoded data have been accumulated in the queue repository, the onboard data manager 906 stores the five minutes of encoded data to the remote data repository 948 via the remote data manager 946 in the data center 966 via the wireless data link 944. In real-time mode, the onboard data manager 906 stores the encoded data and any event information received from the data encoder 908 to the anti-collision hardened memory module 904 and to the remote data repository 948 via the wireless data link 944 via the remote data manager 946 in the data center 966 at each configurable predetermined time period (such as (for example) every second or every 0.10 seconds).

[0147] In this embodiment, the onboard data manager 906 sends video data, audio data, internal and / or external state determinations, object detection and location information, obstacle detection information, and any other data or event information to a remote data repository 948 via a remote data manager 946 in a data center 966 via a wireless data link 944. The wireless data link 944 may be, for example, a wireless local area network (WLAN), a wireless metropolitan area network (WMAN), a wireless wide area network (WWAN), a wireless virtual private network (WVPN), a cellular telephone network, or any other means of transmitting data from the data logger 902 to (in this example) the remote data manager 946. The process of remotely retrieving data from the asset 964 requires a wireless connection between the asset 964 and the data center 966. When a wireless data connection is not available, the data is stored and queued until wireless connectivity is restored.

[0148] In parallel with data logging, the data logger 902 continuously and autonomously copies data to the remote data repository 948. The copy process has two modes: real-time mode and near real-time mode. In real-time mode, data is copied to the remote data repository 948 every second. In near real-time mode, data is copied to the remote data repository 948 every five minutes. The rates for near real-time mode and real-time mode are configurable, and the rate for real-time mode can be adjusted to support high-resolution data by copying data to the remote data repository 948 every 0.10 seconds. Near real-time mode is used during normal operation under most conditions in order to improve the efficiency of the data copy process.

[0149] The real-time mode can be initiated based on an event occurring onboard the asset 964 or by a request initiated from the data center 966. The request for real-time mode initiated by the typical data center 966 is initiated when a remotely located user 968 has requested real-time information from the network client 962. The typical reason for the origin of the real-time mode onboard the asset 964 is the detection of an event or accident involving the asset 964, such as an operator-initiated emergency stop request, emergency braking activity, rapid acceleration or deceleration on any axis, or loss of input power to the data recorder 902. When the near real-time mode is changed to the real-time mode, all data that has not been copied to the remote data repository 948 is copied and stored in the remote data repository 948, and then the real-time copy is initiated. The transition between the near real-time mode and the real-time mode usually occurs in less than five seconds. After a predetermined amount of time has passed since the event or accident, a predetermined amount of inactivity, or when the user 968 no longer expects real-time information from the asset 964, the data recorder 902 reverts to the near real-time mode. The predetermined amount of time required to initiate the transition is configurable and is usually set to ten minutes.

[0150] When the data recorder 902 is in real-time mode, the onboard data manager 906 attempts to continuously empty its queue to the remote data manager 946, thereby storing data to the anti-collision hardened memory module 940 and optionally to the optional non-anti-collision hardened removable storage device of the sixth embodiment, and simultaneously sending data to the remote data manager 946.

[0151] Upon receiving video data, audio data, internal and / or external state determinations, object detection and positioning information, obstacle detection information, and any other data or information to be copied from the data recorder 902, the remote data manager 946 stores the data (e.g., encoded data and detected event data) it receives from the onboard data manager 906 to a remote data repository 948 in the data center 966. The remote data repository 948 may be, for example, a cloud-based data storage device or any other suitable remote data storage device. When the data is received, a process is initiated that causes the data decoder 954 to decode the most recently copied data from the remote data repository 948 and send the decoded data to the track / object detection / positioning information component 950 that looks at the stored data for additional 'post-processing' events. In this implementation, the track / object detection / positioning information component 950 includes an object / obstacle detection component for determining internal and / or external state determinations, object detection and positioning information, and obstacle detection information. Upon detecting interior and / or exterior information, object detection and positioning information, and / or obstacle detection information, the track / object detection / positioning information component 950 stores the information in the remote data repository 948 .

[0152] Remotely located user 968 can use a standard web client 962 (e.g., a web browser) or a virtual reality device (not shown) (e.g., Figure 8The virtual reality device 828 (which in this embodiment can display a thumbnail image of a selected camera) accesses video data, audio data, internal and / or external state determination, object detection and positioning information, obstacle detection information, and any other information stored in the remote data repository 948, including track information, asset information, and cab occupancy information, related to a specific asset 964 or multiple assets. The network client 962 communicates the user 968's request for information to the network server 958 via the network 960 using common network standards, protocols, and technologies. The network 960 can be, for example, the Internet. The network 960 can also be a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network, or any other means that transmits data from the network server 958 to (in this example) the network client 962. The network server 958 requests the desired data from the remote data repository 948, and the data decoder 954 obtains the requested data related to the specific asset 964 from the remote data repository 948 when requested to the network server 958. The data decoder 954 decodes the requested data and sends the decoded data to the localizer 956. The localizer 956 identifies the profile settings set by the user 968 by accessing the network client 962, and uses the profile settings to prepare the information sent to the network client 962 for presentation to the user 968, because the original encoded data and the detected track / object detection / positioning information are saved to the remote data repository 948 using Coordinated Universal Time (UTC) and the International System of Units (SI units). The localizer 956 converts the decoded data into the format desired by the user 968, such as the measurement units and language preferred by the user 968. The localizer 956 sends the localized data in the format preferred by the user 968 to the network server 958 upon request. The network server 958 then sends the localized data to the network client 962 for viewing and analysis, thereby providing playback and real-time display of standard video and 360-degree video, as well as internal and / or external state determination, object detection and positioning information, and obstacle detection information, such as track and / or object detection ( Fig.16A ), track and turnout inspection ( Fig. 16B ), track and / or object detection, counting the number of tracks and signal detection ( Fig. 16C ), intersection and track and / or object detection ( Fig.16D ), dual overhead signal detection ( Fig.16E ), multi-track and / or multi-object detection ( Fig.16F ), turnout and track and / or object detection ( Figure 16G ) and turnout detection ( Fig.16H ).

[0153] The network client 962 is enhanced with a software application that provides playback of 360-degree video and / or other video in a variety of different modes. The user 968 can select the mode in which the software application presents video playback, such as, for example, fisheye view, rectified view, panoramic view, dual panoramic view, and quad view.

[0154] Fig.17 970 for determining an internal state of an asset 964 according to an embodiment of the present disclosure. The video analytics system 910 receives data signals 972 from various input components such as cameras 940 (including but not limited to 360 degree cameras, fixed cameras, narrow angle cameras, wide angle cameras, 360 degree fisheye cameras, radar, LIDAR, and / or other cameras) on, in, or near the asset 964, vehicle data component 934, weather component 936, and route / inventory / GIS component 938. The video analytics system 910 processes the data signals 974 and determines an internal state 976, such as cab occupancy, using supervised learning and / or reinforcement learning components.

[0155] Fig.18 980 for determining object detection / localization and obstacle detection occurring outside and inside an asset 964 according to an embodiment of the present disclosure. The video analytics system 910 receives data signals 982 from various input components such as cameras 940 (including but not limited to 360 degree cameras, fixed cameras, narrow angle cameras, wide angle cameras, 360 degree fisheye cameras, radar, LIDAR, and / or other cameras) on, in, or near the asset 964, vehicle data component 934, weather component 936, and route / inventory / GIS component 938. The video analytics system 910 processes the data signals 984 and determines obstacle detection 986 and object detection and localization 988, such as the presence of tracks, using supervised learning and / or reinforcement learning component 924, object detection / localization component 926, and obstacle detection component 928.

[0156] To simplify explanation, process 970 and process 980 are depicted and described as a series of steps. However, the steps according to the present disclosure may occur in various orders and / or concurrently. In addition, the steps according to the present disclosure may occur with other steps not presented and described herein. In addition, all steps shown may not be required to implement the method according to the disclosed subject matter.

[0157] Real-time data collection and recording The data sharing system works in conjunction with a real-time data collection and recording system (such as, for example, DARS 100, 200) and a viewer, which provides remotely located users (such as asset owners, operators, and investigators) with real-time or near real-time access to a wide range of data (such as event and operational data, video data, and audio data) of high-value assets (such as, for example, locomotives). The data collection and recording system records data related to the asset and streams the data to a remote data repository and remotely located users before, during, and after an incident. The data is streamed to the remote data repository in real-time or near real-time, making the information available at least at the time of an incident or emergency, thereby virtually eliminating the need to locate and download a "black box" in order to investigate an incident involving the asset, and eliminating the need to interact with the data collection and recording system on the asset to request downloads of specific data, locate and transfer files, and use custom applications to view data. The real-time data collection and recording system retains typical recording capabilities and adds the ability to stream data to a remote data repository and remotely located users before, during, and after an incident.

[0158] Remotely located users (e.g., asset owners, operators, and / or investigators) can access a universal web browser to navigate to real-time and / or historical desired data related to a selected asset to view and analyze the asset's operational efficiency and safety in real-time or near real-time. The ability to view operations in real-time or near real-time enables rapid assessment and adjustment of actions. During an incident, for example, real-time information and / or data can facilitate classification of the situation and provide valuable information to first responders. During normal operations, for example, near real-time information and / or data can be used to audit crew performance and aid in network-wide situational awareness.

[0159] A remotely located user can access a general web browser to use the viewer and navigate to the desired data associated with a selected asset or multiple assets to view and analyze the operational efficiency and safety of the assets in real time or near real time. The viewer provides the ability to view operations and / or videos (including, but not limited to, 360-degree videos) in real time or near real time, which enables rapid assessment and adjustment of crew behavior. Owners, operators, and investigators can view and analyze the operational efficiency, safety of people, vehicles, and infrastructure and can investigate or inspect accidents. During an accident, for example, 360-degree videos can facilitate classification of situations and provide valuable information to emergency personnel and investigators. During normal operations, for example, 360-degree videos can be used to audit crew performance and help full network situational awareness. In addition, remotely located users can view 360-degree videos in various modes with the viewer by using a virtual reality device or through a standard network client (such as a web browser), thereby eliminating the need to download and use external applications to view videos.

[0160] The data sharing system allows users to share data obtained from data collection and recording systems with remotely located users. Users can share data in a secure, controlled, tracked and audited manner with remote recipient end users who have Internet access and modern web browsers. Users share the URL of the data, not the file. URL-based data sharing enables users to control, track and audit sensitive data. Users will be able to share data to improve the safety of the world's transportation systems without having to worry about unauthorized data dissemination. Data can be shared with remotely located users by investigators using a web client without having to locate and download the "black box".

[0161] The data may include, but is not limited to: analog and frequency parameters, such as speed, pressure, temperature, current, voltage, and acceleration originating from the asset and / or nearby assets; Boolean data, such as switch position, actuator position, warning light illumination, and actuator commands; Global Positioning System (GPS) data and / or Geographic Information System (GIS) data, such as position, speed, and altitude; internally generated information, such as the regulatory speed limit of the asset given its current location; video and image information from cameras located at various locations in, on, or near the asset; audio information from microphones located at various locations in, on, or near the asset; information sent from a data center about the asset's operating plan, such as route, schedule, and cargo manifest information; information about environmental conditions in the area in which the asset is currently operating or planned to operate, including current and forecast weather conditions; asset control status and operating data generated by systems, such as Positive Train Control (PTC) in a locomotive; and data derived from a combination of any of the above, including, but not limited to, additional data, video and audio analysis, and analytics.

[0162] Figure 1 and 2Field implementations of first and second embodiments of exemplary real-time data acquisition and recording systems (DARS) 100, 200, respectively, in which aspects of the present disclosure may be implemented are shown. The DARS 100, 200 includes a data recorder 154, 254 mounted on a vehicle or mobile asset 148, 248 and communicating with any number of various information sources through any combination of onboard wired and / or wireless data links 170, 270 (e.g., wireless gateways / routers), or communicating with off-board information sources via data links (e.g., wireless data links 146) via a data center 150, 250 of the DARS 100, 200. The data recorder 154, 254 includes an onboard data manager 120, 220, a data encoder 122, 222, a vehicle event detector 156, 256, a queue repository 158, 258, and a wireless gateway / router 172, 272. Additionally, in this implementation, the data recorder 154, 254 may include a crash-hardened memory module 118, 218 and / or an Ethernet switch 162, 262 with or without Power over Ethernet (POE). Exemplary hardened memory modules 118, 218 may be, for example, a crash-resistant event recorder memory module that complies with federal regulations and / or Federal Railroad Administration regulations; a crash-survivable memory unit that complies with federal regulations and / or Federal Aviation Administration regulations; a crash-hardened memory module that complies with any applicable federal regulations; or any other suitable hardened memory device as known in the art. Figure 2 In the second embodiment shown in , the data logger 254 may further include an optional non-crash-hardened removable storage device 219 .

[0163] The wired and / or wireless data links 170, 270 may include any or a combination of discrete signal inputs, standard or proprietary Ethernet, serial connections, and wireless connections. The Ethernet connection may utilize the Ethernet switch 162, 262 of the data logger 154, 254 and may utilize POE. The Ethernet switch 162, 262 may be internal or external and may support POE. Additionally, data from a remote data source (e.g., Figure 1 and 2 Data of the map component 164, 264, route / crew list component 124, 224 and weather component 126, 226 in the implementation scheme can be used from the data center 150, 250 to the onboard data manager 120, 220 and the vehicle event detector 156, 256 via the wireless data link 146, 246 and the wireless gateway / router 172, 272.

[0164] The data recorder 154, 254 collects data or information from a variety of sources (which can vary greatly based on the configuration of the asset) through the onboard data link 170, 270. The data encoder 122, 222 encodes at least a minimum data set, typically defined by a regulatory agency. In this embodiment, the data encoder 122, 222 receives data from a variety of asset 148, 248 sources and data center 150, 250 sources. The information sources may include any number of components in the assets 148, 248, such as any of the following: analog inputs 102, 202, digital inputs 104, 204, I / O modules 106, 206, vehicle controllers 108, 208, engine controllers 110, 210, inertial sensors 112, 212, global positioning systems (GPS) 114, 214, cameras 116, 216, active train control (PTC) / signal data 166, 266, fuel data 168, 268, cellular transmission detectors (not shown), internal drive data, and any additional data signals, and any number of components in the data centers 150, 250, such as any of the route / crew list components 124, 224, weather components 126, 226, map components 164, 264, and any additional data signals. The cameras 116, 216 or image measurement devices and / or video measurement devices include, but are not limited to, 360 degree cameras, fixed cameras, narrow angle cameras, wide angle cameras, 360 degree fisheye view cameras, and / or other cameras inside and outside the asset 148. The data encoder 122, 222 compresses or encodes the data and time synchronizes the data to facilitate efficient real-time transmission and replication to the remote data repository 130, 230. The data encoder 122, 222 transmits the encoded data to the onboard data manager 120, 220, which then saves the encoded data in the collision-hardened memory module 118, 218 and the queue repository 158, 258 for replication to the remote data repository 130, 230 via the remote data manager 132, 232 located in the data center 150, 250. Optionally, the onboard data manager 120, 220 may also save a third copy of the encoded data in Figure 2 The onboard data manager 120, 220 and the remote data manager 132, 232 work together to manage the data replication process. A single remote data manager 132, 232 in a data center 150, 250 can manage the replication of data from multiple assets 148, 248.

[0165] Data from the various input components and data from the in-cab audio / graphical user interface (GUI) 160, 260 is sent to the vehicle event detector 156, 256. The vehicle event detector 156, 256 processes the data to determine whether an event, accident, or other predefined situation has occurred involving the asset 148, 248. When the vehicle event detector 156, 256 detects a signal indicating the occurrence of a predefined event, the vehicle event detector 156, 256 sends the processed data of the occurrence of the predefined event to the onboard data manager 120, 220 along with supporting data surrounding the predefined event. The vehicle event detector 156, 256 detects events based on data from a variety of sources (e.g., analog inputs 102, 202, digital inputs 104, 204, I / O modules 106, 206, vehicle controllers 108, 208, engine controllers 110, 210, inertial sensors 112, 212, GPS 114, 214, cameras 116, 216, route / crew inventory components 124, 224, weather components 126, 226, map components 164, 264, PTC / signal data 166, 266, and fuel data 168, 268, which may vary based on the configuration of the asset). When the vehicle event detector 156, 256 detects an event, the detected asset event information is stored in a queuing repository 158, 258 and may be optionally presented to the crew of the asset 148, 248 via an in-cab audio / graphical user interface (GUI) 160, 260.

[0166] The onboard data manager 120, 220 also sends the data to the queue repository 158. In near real-time mode, the onboard data manager 120, 220 stores the encoded data and any event information received from the data encoder 122, 222 in the collision hardened memory module 118, 218 and in the queue repository 158, 258. Figure 2 In a second embodiment of the invention, the onboard data manager 220 may optionally store the encoded data in the non-crash hardened removable storage device 219. After accumulating five minutes of encoded data in the queue repository 158, 258, the onboard data manager 120, 220 stores the five minutes of encoded data to the remote data repository 130, 230 via the remote data manager 132, 232 in the data center 150, 250 via the wireless data link 146, 246 accessed through the wireless gateway / router 172, 272. In real-time mode, the onboard data manager 120, 220 stores the encoded data and any event information received from the data encoder 122, 222 to the crash hardened memory module 118, 218, and optionally stores the encoded data in the remote data repository 130, 230. Figure 2The data is stored in a non-crash-hardened removable storage device 219 and stored to a remote data repository 130, 230 via a remote data manager 132, 232 in a data center 150, 250 via a wireless data link 146, 246 accessed through a wireless gateway / router 172, 272. The onboard data manager 120, 220 and the remote data manager 132, 232 may communicate via various wireless communication links (e.g., Wi-Fi, cellular, satellite, and private wireless systems) utilizing the wireless gateway / router 172, 272. The wireless data link 146, 246 may be, for example, a wireless local area network (WLAN), a wireless metropolitan area network (WMAN), a wireless wide area network (WWAN), a private wireless system, a cellular telephone network, or any other means of transmitting data from the data logger 154, 254 of the DARS 100, 200 to (in this example) the remote data manager 130, 230 of the DARS 100, 200. When the wireless data connection is unavailable, data is stored in memory and queued in the queue repository 158, 258 until wireless connectivity is restored and the data copy process can continue.

[0167] In parallel with data logging, the data logger 154, 254 continuously and autonomously copies data to the remote data repository 130, 230. The copying process has two modes: real-time mode and near real-time mode. In real-time mode, data is copied to the remote data repository 130, 230 every second. In near real-time mode, data is copied to the remote data repository 130, 230 every five minutes. The rates for near real-time mode and real-time mode are configurable, and the rate for real-time mode can be adjusted to support high-resolution data by copying data to the remote data repository 130, 230 every 0.10 seconds. When the DARS 100, 200 is in near real-time mode, the onboard data manager 120, 220 queues the data in the queue repository 158, 258 before copying the data to the remote data manager 132, 232. The onboard data manager 120, 220 also replicates the vehicle event detector information queued in the queue repository 158, 258 to the remote data manager 132, 232. The near real-time mode is used during normal operation under most conditions in order to improve the efficiency of the data replication process.

[0168] The real-time mode may be initiated based on an event that occurs and is detected by a vehicle event detector 156, 256 onboard the asset 148, 248 or by a request initiated from the data center 150, 250. A typical data center 150, 250 initiated request for real-time mode is initiated when a remotely located user 152, 252 has requested real-time information from a network client 142, 242. A typical reason for the origin of real-time mode onboard an asset 148, 248 is the detection of an event or incident by the vehicle event detector 156, 256, such as an operator initiated emergency stop request, emergency braking activity, rapid acceleration or deceleration on any axis, or loss of input power to the data recorder 154, 254. When transitioning from near real-time mode to real-time mode, all data that has not yet been copied to the remote data repository 130, 230 is copied and stored in the remote data repository 130, 230, and then real-time replication is initiated. The transition between near real-time mode and real-time mode typically occurs in less than five seconds. The data logger 154, 254 reverts to near real-time mode after a predetermined amount of time has passed since the event or incident, a predetermined amount of inactivity, or when the user 152, 252 no longer desires real-time information from the asset 148, 248. The predetermined amount of time required to initiate the transition is configurable and is typically set to ten minutes.

[0169] When the data recorder 154, 254 is in real-time mode, the onboard data manager 120, 220 attempts to continuously empty its queue to the remote data manager 132, 232, thereby storing data to the crash-hardened memory module 118, 218 and optionally to the remote data manager 132, 232. Figure 2 The onboard data manager 120, 220 also sends the detected vehicle information queued in the queue repository 158, 258 to the remote data manager 132, 232.

[0170] Upon receiving data to be copied from the data logger 154, 254 and data from the map component 164, 264, the route / crew list component 124, 224, and the weather component 126, 226, the remote data manager 132, 232 stores the compressed data to the remote data repository 130, 230 in the data center 150, 250 of the DARS 100, 200. The remote data repository 130, 230 may be, for example, a cloud-based data storage device or any other suitable remote data storage device. When the data is received, a process is initiated that causes the data decoder 136, 236 to decode the most recently copied data for / from the remote data repository 130, 230 and send the decoded data to the remote event detector 134, 234. The remote data manager 132, 232 stores the vehicle event information in the remote data repository 130, 230. When the remote event detector 134, 234 receives the decoded data, it processes the decoded data to determine whether an event of interest is found in the decoded data. The decoded information is then used by the remote event detector 134, 234 to detect an event, incident, or other predefined condition occurring in the data with respect to the asset 148, 248. Upon detecting an event of interest from decoded data previously stored in the remote data repository 130, 230, the remote event detector 134, 234 stores the event information and supporting data in the remote data repository 130, 230. When the remote data manager 132, 232 receives the remote event detector 134, 234 information, the remote data manager 132, 232 stores the information in the remote data repository 130, 230.

[0171] A remotely located user 152, 252 can access information related to a particular asset 148, 248 or multiple assets, including vehicle event detector information, using a standard network client 142, 242 (e.g., a web browser) or a virtual reality device (not shown), which in this embodiment can display thumbnail images from a selected camera. The network client 142, 242 communicates the user's 152, 252 request for information to the network server 140, 240 via the network 144, 244 using common network standards, protocols, and technologies. The network 144, 244 can be, for example, the Internet. The network 144, 244 can also be a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network, or any other means of transmitting data from the network server 140, 240 to (in this example) the network client 142, 242. The network server 140, 240 requests the desired data from the data decoder 136, 236. The data decoder 136, 236 obtains the requested data related to a particular asset 148, 248 or multiple assets from the remote data repository 130, 230 upon request to the network server 140, 240. The data decoder 136, 236 decodes the requested data and sends the decoded data to the localizer 138, 238. Localization is the process of converting data into a format desired by an end user, such as converting the data into a user's preferred language and measurement units. The localizer 138, 238 identifies the profile settings set by the user 152, 252 by accessing the network client 142, 242, and uses the profile settings to prepare the information sent to the network client 142, 242 for presentation to the user 152, 252 in the user's preferred language and measurement units, because the original encoded data and detected event information are saved to the remote data repository 130, 230 using Coordinated Universal Time (UTC) and the International System of Units (SI units). The localizer 138, 238 converts the decoded data into a format desired by the user 152, 252, such as a language and measurement unit preferred by the user 152, 252. The localizer 138, 238 sends the localized data in the format preferred by the user 152, 252 to the network server 140, 240 upon request. The network server 140, 240 then sends the localized data of the asset or multiple assets to the network client 142, 242 for viewing and analysis through the viewer, providing playback and real-time display of standard video and 360-degree video. The network client 142, 242 can display and the user 152, 252 can view the data, video and audio of a single asset or the data, video and audio of multiple assets simultaneously.The network client 142, 242 may also provide synchronized playback and real-time display of data and multiple video and audio data from both standard and 360-degree video sources on, in, or near the asset, nearby assets, and / or remotely located sites. The network client 142, 242 may play the video data on a viewer for the user 152, 252, who may interact with the video for viewing and analysis. The user 152, 252 may also download the video data using the network client 142, 242 and may then use a virtual reality device (e.g., Figure 8 A virtual reality device 828) interacts with the video data on the viewer for viewing and analysis.

[0172] The network client 142, 242 is enhanced with a software application that provides playback of the video data and / or 360-degree video in a variety of different modes. The user 152, 252 can select the mode in which the software application presents the video playback, such as, for example, a fisheye view, a rectified panoramic view, a rectified dual panoramic view, and a rectified quad view.

[0173] Users 152, 252 can further use the data sharing system of the present disclosure to share data with remotely located recipient end users with Internet access and modern web browsers in a secure, controlled, tracked and audited manner. Users 152, 252 share the URL of the data, not the file. URL-based data sharing enables users to control, track and audit sensitive data. Users will be able to share data to improve the security of the world's transportation system without worrying about unauthorized data dissemination. Administrators have the authority to increase and / or decrease the local permissions of users 152, 252 and each remotely located recipient end user. The local permissions of users 152, 252 and each remotely located recipient end user determine the permissions that a specific remotely located recipient end user must have to view data on the network client 142, 242. The data sharing system is used by asset 148, 248 owners, operators and investigators to share real-time data on the operating efficiency and safety of vehicles. Data sharing enables rapid evaluation and adjustment of behavior.

[0174] Fig.19 is used to display the Figure 8 web browser 826) or a virtual reality device (e.g. Figure 8 Flowchart of process 990 for sharing data and / or information from assets 148, 248 using virtual reality device 828. Typically, user 152, 252 will use network client 142, 242 to request data center 150, 250 to share asset 148, 248 data 992 ( Fig.19). A typical reason for data sharing is the detection of an incident, such as an operator initiated emergency stop request, emergency braking activity, rapid acceleration or deceleration on any axis, or loss of input power to the DARS 100, 200. No files are downloaded or sent to the remotely located recipient end user. Users 152, 252 will not be able to share more content than their local permissions on the network client 142, 242 allow. The remotely located recipient end user will be able to see the data based on their own local permissions on the network client 142, 242. This sharing activity is recorded in the data center 150, 250 by the network client 142, 242. Administrators can use permission escalation to share data to multiple users 152, 252 who cannot access the data through the network client 142, 242 locally. This permission escalation activity will be recorded in the data center 150, 250 by the network client 142, 242.

[0175] As previously discussed, the user 152, 252 uses the network client 142, 242 to access information, including vehicle event detector 156, 256 information. Using common network standards, protocols and technologies, such as the Internet or private network 144, 244, the network client 142, 242 communicates the information desired by the user 152, 252 with the network server 140, 240. The network server 140, 240 requests the desired data from the data decoder 136, 236. The data is extracted and / or decoded by the data decoder 136, 236, and then the data is localized by the localizer 138, 238, thereby converting the data into the format desired by the user 152, 252, as described above. The network server 140, 240 then sends the localized data to the network client 142, 242 for viewing and analysis 994 ( Fig.19 ).

[0176] The sharer end user 152, 252 may share this information, including the vehicle event detector 156, 256 information and video data, with multiple remotely located recipient end users using the web client 142, 242, regardless of whether the recipient end user has a pre-registered account on the web client 142, 242. The sharer end user 152, 252 may share information and data with multiple remotely located recipient end users, regardless of whether the recipient end user has a pre-registered account on the web client 142, 242. During this process, the web client 142, 242 will generate an email 996 (with a URL pointing to the data in the data center 150, 250) Fig.19). A remotely located recipient end user receives an email with a URL address to access the data. The URL address is not a link to a file. The file is not shared by the recipient end user. The data is not a discrete file, but a series of data pulled from a remote data repository 130 based on a shared web-based viewer link. The URL address sent via email is a link to a web-based viewer that allows the recipient end user to view specific data segments synchronized with still images and videos via the web-based viewer. When the remotely located recipient end user clicks on the URL, they will be able to access the shared information using their own web client 142, 242, and the sharing activity will be recorded in the data center 150, 250 by the web client 142, 242.

[0177] To simplify explanation, process 990 is depicted and described as a series of steps. However, the steps according to the present disclosure may occur in various orders and / or concurrently. In addition, the steps according to the present disclosure may occur together with other steps not presented and described herein. In addition, all steps shown may not be required to implement the method according to the disclosed subject matter.

[0178] The seventh embodiment of the real-time data acquisition and recording system and automated signal compliance monitoring and alarm system described herein provides remotely located users (e.g., asset owners, operators, and investigators) with real-time or near real-time access to a wide range of data related to high-value assets (e.g., event and operational data, video data, and audio data). The automated signal compliance monitoring and alarm system records data related to the asset via a data recorder and streams the data to a remote data repository and remotely located users before, during, and after an incident. The data is streamed to the remote data repository in real-time or near real-time, making the information available at least in the event of an incident or emergency, thereby virtually eliminating the need to locate and download a "black box" in order to investigate an incident involving the asset, and eliminating the need to interact with the data recorder on the asset to request downloads of specific data, locate and transfer files, and use custom applications to view the data. The system of the present disclosure retains typical recording capabilities and adds the ability to stream data to a remote data repository and remote terminal users before, during, and after an incident. In most cases, the information recorded in the data logger is redundant and not necessary because the data has already been collected and stored in a remote data repository.

[0179] The automated signal monitoring and alert system also automatically monitors mobile assets (e.g., locomotives, trains, aircraft, and automobiles) for violations of signal aspects (e.g., stop lights, traffic lights, and / or speed limit signals), or for unsafe operations of mobile assets in an attempt to maintain compliance with signals (e.g., stop lights, traffic lights, and / or speed limit signals), and provides historical and real-time alerts. The automated signal monitoring and alert system combines the use of image analytics, GPS location, braking force, and vehicle speed, and automated electronic notifications to alert personnel onboard and / or offboard the mobile asset in real-time when the mobile asset violates safe operating rules, such as, for example, when the mobile asset passes a stop signal before stopping and receiving authorization (a red light runner), when a mobile asset traveling at a greater speed violates a limit signal indicating a reduced speed limit, and when a mobile asset applies late and / or excessive braking force in an effort to stop before passing a stop / red signal.

[0180] Prior to the automated signal monitoring and alarm system of the present disclosure, operations center personnel relied on the mobile asset crew to report when a safe operating rule was violated. This sometimes resulted in a catastrophic collision between mobile assets, where a subsequent investigation revealed that a safe operating rule violation had occurred. Additionally, excessive braking force may have caused a mechanical failure of a portion of the mobile asset, and in instances where the mobile asset is a locomotive and / or train, excessive braking force may have resulted in a derailment, where a subsequent investigation revealed that a safe operating rule violation was the root cause. The system of the present disclosure enables users to monitor and / or be alerted when a safe operating rule violation has occurred before a mechanical failure, collision, derailment, and / or another incident occurs.

[0181] End users can subscribe to be alerted when safe operating rule violations occur and will receive email, text message, and / or in-browser electronic notifications within minutes of the actual event. End users can utilize historical records to analyze data to identify patterns such as, for example, problem locations, impaired sight lines, equipment failures, and poor crew performance, which can aid in implementing new and safer operating rules or crew education opportunities for continuous improvement. The disclosed system enables end users to utilize continuous electronic monitoring and extensive image analytics to understand any and all times a mobile asset is operating unsafely due to safe operating rule violations and / or non-compliance signals.

[0182] Automated signal monitoring and alert systems are used by vehicle and / or mobile asset owners, operators and investigators to gain real-time visibility and analysis of the operational efficiency and safety of mobile assets. The ability to view operations in real time enables rapid assessment and adjustment of actions. During incidents, real-time information facilitates classification of situations and provides valuable information to first responders. During normal operations, near real-time information can be used to audit crew performance and contribute to network-wide operational safety and awareness.

[0183] The automated signal monitoring and alarm system utilizes outward-facing cameras and / or other cameras, GPS position, speed and acceleration, and vehicle, train and / or mobile asset brake pressure sensor data in a fully integrated, time-synchronized automated system to identify unsafe and potentially catastrophic operating practices to provide real-time feedback to mobile asset crews and managers. The automated signal monitoring and alarm system also provides automated data and video downloads to users using a variety of data sources to allow for a complete understanding of the operating environment at the time of an alarm.

[0184] The data may include, but is not limited to: analog and digital parameters, such as speed, pressure, temperature, current, voltage, and acceleration originating from the asset and / or nearby assets; Boolean data, such as switch position, actuator position, warning light illumination, and actuator commands; Global Positioning System (GPS) data and / or Geographic Information System (GIS) data, such as position, speed, and altitude; internally generated information, such as the regulatory speed limit of the asset given its current location; video and image information from cameras located at various locations in, on, or near the asset; audio information from microphones located at various locations in, on, or near the asset; information sent from a data center about the asset's operating plan, such as route, schedule, and cargo manifest information; information about environmental conditions in the area in which the asset is currently operating or planned to operate, including current and forecast weather conditions; asset control status and operating data generated by systems (such as Positive Train Control (PTC) in a locomotive); and data derived from any combination of the above, including, but not limited to, additional data, video and audio analysis, and analytics.

[0185] Fig.19A field implementation of a seventh embodiment of an exemplary real-time data acquisition and recording system (DARS) 1000 and an automated signal monitoring and alarm system 1080 in which aspects of the present disclosure may be implemented is shown. DARS 1000 is a system that delivers real-time information from a data recording device to a remotely located end user. DARS 1000 includes a data recorder 1054 that is mounted on a vehicle or mobile asset 1048 and communicates with any number of various information sources through any combination of onboard wired and / or wireless data links 1070 (e.g., wireless gateways / routers), or communicates with off-board information sources via a data center 1050 of DARS 1000 via a data link (e.g., wireless data link 1046). Data recorder 1054 includes an onboard data manager 1020, a data encoder 1022, a vehicle event detector 1056, a queue repository 1058, and a wireless gateway / router 1072. Additionally, in this implementation, the data recorder 1054 may include a crash-hardened memory module 1018 and / or an Ethernet switch 1062 with or without Power over Ethernet (POE). Exemplary hardened memory modules 1018 may be, for example, a crash-hardened event recorder memory module that complies with federal regulations and / or Federal Railroad Administration regulations; a crash-survivable memory unit that complies with federal regulations and / or Federal Aviation Administration regulations; a crash-hardened memory module that complies with any applicable federal regulations; or any other suitable hardened memory device as known in the art. In an eighth embodiment, the data recorder may further include an optional non-crash-hardened removable storage device (not shown).

[0186] The wired and / or wireless data link 1070 may include any or a combination of discrete signal inputs, standard or proprietary Ethernet, serial connections, and wireless connections. The Ethernet connection device may utilize the Ethernet switch 1062 of the data logger 1054 and may utilize POE. The Ethernet switch 1062 may be internal or external and may support POE. Additionally, data from a remote data source (e.g., Fig.19 Data from the map component 1064, route / crew list component 1024, and weather component 1026 in the implementation scheme can be used from the data center 1050 to the onboard data manager 1020 and the vehicle event detector 1056 via the wireless data link 1046 and the wireless gateway / router 1072.

[0187] The data recorder 1054 collects data or information from a variety of sources (which may vary greatly based on the configuration of the asset) through an onboard data link 1070. The data encoder 1022 encodes at least a minimum data set, typically defined by a regulatory agency. In this embodiment, the data encoder 1022 receives data from a variety of asset 1048 sources and data center 1050 sources. The information source may include any number of components in the asset 1048, such as any of the following: analog input 1002, digital input 1004, I / O module 1006, vehicle controller 1008, engine controller 1010, inertial sensor 1012, global positioning system (GPS) 1014, camera 1016, active train control (PTC) / signal data 1066, fuel data 1068, cellular transmission detector (not shown), internal drive data, and any additional data signals, and any number of components in the data center 1050, such as any of the route / crew list component 1024, weather component 1026, map component 1064, and any additional data signals. In addition, the asset 1048 information source can be connected to the data recorder 1054 through any combination of wired or wireless data links 1070. The data encoder 1022 compresses or encodes the data and synchronizes the data in time to facilitate efficient real-time transmission and replication to the remote data repository 1030. The data encoder 1022 transmits the encoded data to the onboard data manager 1020, which then saves the encoded data in the anti-collision hardened memory module 1018 and the queue repository 1058 for replication to the remote data repository 1030 via the remote data manager 1032 located in the data center 1050. Optionally, the onboard data manager 1020 can also save a third copy of the encoded data in the non-anti-collision hardened removable storage device of the eighth embodiment. The onboard data manager 1020 and the remote data manager 1032 work together to manage the data replication process. A single remote data manager 1032 in the data center 1050 can manage the replication of data from multiple assets 1048.

[0188] Data from the various input components and data from the in-cab audio / graphical user interface (GUI) 1060 are sent to the vehicle event detector 1056. The vehicle event detector 1056 processes the data to determine whether an event, accident, or other predefined situation involving the asset 1048 has occurred. When the vehicle event detector 1056 detects a signal indicating the occurrence of a predefined event, the vehicle event detector 1056 sends the processed data of the occurrence of the predefined event along with supporting data surrounding the predefined event to the onboard data manager 1020. The vehicle event detector 1056 detects events based on data from a variety of sources (e.g., analog inputs 1002, digital inputs 1004, I / O modules 1006, vehicle controllers 1008, engine controllers 1010, inertial sensors 1012, GPS 1014, cameras 1016, route / crew list components 1024, weather components 1026, map components 1064, PTC / signal data 1066, and fuel data 1068, which may vary based on the configuration of the asset). When the vehicle event detector 1056 detects an event, the detected asset event information is stored in a queuing repository 1058 and optionally presented to the crew of the asset 1048 via an in-cab audio / graphical user interface (GUI) 1060 .

[0189] When the position of the asset 1048 indicates that the signal 1082 has been crossed, excessive braking has occurred and the asset 1048 has stopped within the proximity of the signal 1082, or a speed limit is imposed by the signal aspect, the onboard data manager 1020 will initiate an outward facing camera image analysis to determine the meaning or aspect of the signal 1082, such as Fig. 20 Utilizing prior art image processing techniques, the outward-facing camera lens may be analyzed by a previously trained neural network or artificial intelligence component to interpret the signal aspects and operating rules meaning. In this exemplary embodiment, the analysis and / or processing performed by the neural network or artificial intelligence component is completed in the background. In another embodiment, the analysis and / or processing performed by the neural network or artificial intelligence component is completed on the asset 1048. The output of the signal aspect decoding is combined with other sensor data to determine whether the asset 1048 is in serious violation of the signal indication due to occupying the railway track, which in this exemplary embodiment may result in a train-to-train collision, or operating in an unsafe manner to achieve signal compliance. When an asset 1048 is found to be non-compliant, after associating the railway's business rules with the signal and asset operation, an electronic alert will be stored in the background, and delivered to users who have subscribed to receive such alerts. These alerts can then be mined directly via a database or by using a website graphical user interface or web client 1042 provided to the user.

[0190] Additionally, an audible alarm may be added to the cab of asset 1048 which will alert the crew of an impending signal violation, an impending adverse condition, so that the crew may respond more quickly in the event that the crew is distracted or otherwise unaware of a track obstruction, a stop sign, and / or when asset 1048 is speeding in an area where the signal requires a lower speed limit.

[0191] The automated signal monitoring and alarm system 1080 is also enhanced to automatically perform video analytics to determine the meaning of the signal whenever a monitored asset crosses a signal, automatically perform video analytics to determine the meaning of the signal whenever an asset experiences excessive braking force and stops within a predefined distance, and monitor asset speed to determine whether the asset is moving at a greater speed than is generally authorized by the signal aspect. Image analytics are completed on the asset 1048 to reduce the delay between the actual event and the electronic notification to the user and / or subscriber. The functionality of the automated signal monitoring and alarm system 1080 is enhanced to allow automated inward and outward video downloading at the time of an alarm to enhance the user experience and reduce the work necessary to investigate the incident. The functionality of the automated signal monitoring and alarm system 1080 is also enhanced to provide real-time audible prompts within the non-compliant asset 1048 to alert the crew in the event that the crew is distracted or otherwise fails to follow safe operating practices regarding signal rules and meanings.

[0192] In addition, the automated signal monitoring and alarm system 1080 and / or the video analytics system 910 may receive location information of a signal (e.g., a stop sign, a traffic sign, a speed limit sign, and / or an object signal near the track) from the asset owner, including latitude and longitude coordinates. The video analytics system 910 then determines whether the location information received from the asset owner is correct. If the location information is correct, the video analytics system 910 stores the information and will not recheck the location information again for a predetermined amount of time, such as checking the location information on a monthly basis. If the location information is incorrect, the video analytics system 910 determines the correct location information and reports the correct location information to the asset owner, stores the location information, and will not recheck the location information again for a predetermined amount of time, such as checking the location information on a monthly basis. Storing the location information makes detection of signals (e.g., stop signs, traffic signs, speed limit signs, and / or object signals near the track) easier.

[0193] The onboard data manager 1020 also sends the data to the queue repository 1058. In near real-time mode, the onboard data manager 1020 stores the encoded data and any event information received from the data encoder 1022 in the crash-hardened memory module 1018 and in the queue repository 1058. In the eighth embodiment, the onboard data manager 1020 may also optionally store the encoded data in the non-crash-hardened removable storage device of the eighth embodiment. After five minutes of encoded data have been accumulated in the queue repository 1058, the onboard data manager 1020 stores the five minutes of encoded data to the remote data repository 1030 via the remote data manager 1032 in the data center 1050 via the wireless data link 1046 accessed through the wireless gateway / router 1072. In real-time mode, the onboard data manager 1020 stores the encoded data and any event information received from the data encoder 1022 to the crash-hardened memory module 1018 and optionally to the non-crash-hardened removable storage device of the eighth embodiment, and to the remote data repository 1030 via the remote data manager 1032 in the data center 1050 via the wireless data link 1046 accessed through the wireless gateway / router 1072. The process of copying the data to the remote data repository 1030 requires a wireless data connection between the asset 1048 and the data center 1050. The onboard data manager 1020 and the remote data manager 1032 may communicate via various wireless communication links, such as Wi-Fi, cellular, satellite, and private wireless systems utilizing the wireless gateway / router 1072. The wireless data link 1046 may be, for example, a wireless local area network (WLAN), a wireless metropolitan area network (WMAN), a wireless wide area network (WWAN), a private wireless system, a cellular telephone network, or any other means of transferring data from the data logger 1054 of the DARS 1000 to (in this example) a remote data manager 1030 of the DARS 1000. When a wireless data connection is unavailable, data is stored in memory and queued in a queue repository 1058 until wireless connectivity is restored and the data copy process can continue.

[0194] In parallel with data recording, the data recorder 1054 continuously and autonomously copies data to the remote data repository 1030. The copying process has two modes: real-time mode and near real-time mode. In real-time mode, data is copied to the remote data repository 1030 every second. In near real-time mode, data is copied to the remote data repository 1030 every five minutes. The rates for near real-time mode and real-time mode are configurable, and the rate for real-time mode can be adjusted to support high-resolution data by copying data to the remote data repository 1030 every 0.10 seconds. When the DARS 1000 is in near real-time mode, the onboard data manager 1020 queues the data in the queue repository 1058 before copying the data to the remote data manager 1032. The onboard data manager 1020 also copies the vehicle event detector information queued in the queue repository 1058 to the remote data manager 1032. The near real-time mode is used during normal operation under most conditions in order to improve the efficiency of the data copying process.

[0195] The real-time mode may be initiated based on an event occurring and detected by a vehicle event detector 1056 onboard the asset 1048 or by a request initiated from the data center 1050. A typical data center 1050 initiated request for real-time mode is initiated when a remotely located user 1052 has requested real-time information from a network client 1042. A typical reason for the origin of real-time mode onboard an asset 1048 is the detection of an event or incident by the vehicle event detector 1056, such as an operator initiated emergency stop request, emergency braking activity, rapid acceleration or deceleration on any axis, or loss of input power to the data recorder 1054. When transitioning from near real-time mode to real-time mode, all data that has not yet been copied to the remote data repository 1030 is copied and stored in the remote data repository 1030, and then real-time replication is initiated. The transition between near real-time mode and real-time mode typically occurs in less than five seconds. The data logger 1054 reverts to near real-time mode after a predetermined amount of time has passed since the event or incident, a predetermined amount of inactivity, or when the user 1052 no longer desires real-time information from the asset 1048. The predetermined amount of time required to initiate the transition is configurable and is typically set to ten minutes.

[0196] When the data recorder 1054 is in real-time mode, the onboard data manager 1020 attempts to continuously empty its queue to the remote data manager 1032, storing data to the crash-hardened memory module 1018 and optionally to the non-crash-hardened removable storage device of the eighth embodiment, and simultaneously sending data to the remote data manager 1032. The onboard data manager 1020 also sends the detected vehicle information queued in the queue repository 1058 to the remote data manager 1032.

[0197] Upon receiving the data to be copied from the data logger 1054 and the data from the map component 1064, the route / crew list component 1024, and the weather component 1026, the remote data manager 1032 stores the compressed data in the remote data repository 1030 in the data center 1050 of the DARS 1000. The remote data repository 1030 may be, for example, a cloud-based data storage device or any other suitable remote data storage device. When the data is received, a process is initiated that causes the data decoder 1036 to decode the most recently copied data for / from the remote data repository 1030 and send the decoded data to the remote event detector 1034. The remote data manager 1032 stores the vehicle event information in the remote data repository 1030. When the remote event detector 1034 receives the decoded data, it processes the decoded data to determine whether an event of interest is found in the decoded data. The decoded information is then used by the remote event detector 1034 to detect an event, accident, or other predefined situation occurring in the data regarding the asset 1048. Upon detecting an event of interest from the decoded data, the remote event detector 1034 stores the event information and supporting data in the remote data repository 1030. When the remote data manager 1032 receives the remote event detector 1034 information, the remote data manager 1032 stores the information in the remote data repository 1030.

[0198] A remotely located user 1052 can access information related to a particular asset 1048 or multiple assets, including vehicle event detector information, using a standard network client 1042 (e.g., a web browser) or a virtual reality device (not shown), which in this embodiment can display thumbnail images from a selected camera. The network client 1042 communicates the user's 1052 request for information to the network server 1040 via a network 1044 using common network standards, protocols, and technologies. The network 1044 can be, for example, the Internet. The network 1044 can also be a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network, or any other means that transmits data from the network server 1040 to (in this example) the network client 1042. The network server 1040 requests the desired data from the data decoder 1036. The data decoder 1036 obtains the requested data related to the particular asset 1048 or multiple assets from the remote data repository 1030 upon request to the network server 1040. The data decoder 1036 decodes the requested data and sends the decoded data to the localizer 1038. Localization is the process of converting data into a format desired by the end user, such as converting the data into a language and measurement units preferred by the user. The localizer 1038 identifies the profile settings set by the user 1052 by accessing the network client 1042, and uses the profile settings to prepare the information sent to the network client 1042 for presentation to the user 1052, because the original encoded data and the detected event information are saved to the remote data repository 1030 using Coordinated Universal Time (UTC) and the International System of Units (SI units). The localizer 1038 converts the decoded data into a format desired by the user 1052, such as a language and measurement units preferred by the user 1052. The localizer 1038 sends the localized data in the format preferred by the user 1052 to the network server 1040 upon request. The network server 1040 then sends the localized data for the asset or assets to the network client 1042 for viewing and analysis, providing playback and real-time display of standard video, 360-degree video, and / or other video. The network client 1042 can display and the user 1052 can view the data, video, and audio of a single asset or of multiple assets simultaneously. The network client 1042 can also provide synchronized playback and real-time display of data and multiple video and audio data from image measurement sources, standard video sources, 360-degree video sources, and / or other video sources and / or range measurement sources on, in, or near the asset, nearby assets, and / or remotely located sites.

[0199] Fig.21106 is a flow chart showing a first illustrated embodiment of a process 1100 for determining signal compliance according to an embodiment of the present disclosure. After the DARS 1000 and camera 1016 are installed and connected to various sensors on the asset 1048 (e.g., analog inputs 1002, digital inputs 1004, I / O modules 1006, vehicle controllers 1008, engine controllers 1010, inertial sensors 1012, global positioning systems (GPS) 1014, cameras 1016, active train control (PTC) / signal data 1066, fuel data 1068, cellular transmission detectors (not shown), internal drive data, and any additional data signals) 1102, onboard data from the various sensors and / or event initiated video and / or still images are sent every five minutes to a backend data center 1074 and the camera images are stored onboard the asset 1048, with a capacity of more than 72 hours 1104. The backend data center 1074 services continuous scan data for trigger conditions 1106. If the plot business logic trigger condition is not met, the workflow is canceled and the plot event 1108 is not recorded. If the asset 1048 travels across the track signal 1082 referenced by the latitude and longitude coordinates of all signals stored in the backend data center 1074 1110 and / or the asset 1048 stops within a certain distance in front of the signal 1082 and uses excess braking force to permit stopping before crossing the signal 1082 1112, the backend data center 1074 service scans the data to determine if the train car is in the lead, control, or first position in the train asset 1048 in this illustrated embodiment 1114. The backend data center 1074 uses the first artificial intelligence model to determine if the train car is in the lead, control, or first position in the train asset 1048 1116. If the train car is not in the lead, control, or first position in the train asset 1048, the plot business logic trigger condition is not met, the workflow is canceled and the plot event 1108 is not recorded. If the train car is in the leading, controlling or first position in the train asset 1048, then the backend data center 1074 requests video content 1118 captured for a short period of time from the leading, controlling or first position locomotive before crossing the signal 1082 and / or when the asset 1048 stops. The retrieved video content is passed to and / or stored in the backend data center 1074 and passed on to the second artificial intelligence model, which scans the video content to determine aspects of the signal 1082, such as the combination of colors of each signal light, to determine whether the signal 1082 indicates a stop meaning 1120. The backend data center 1074 determines whether the aspects of the signal 1082 indicate that the asset 1048 must stop and cannot cross the signal 1082 1122. If the aspects of the signal 1082 do not indicate that the asset 1048 must stop and cannot cross the signal 1082, then the plot business logic trigger condition is not satisfied, the workflow is canceled and the plot event 1108 is not recorded.If aspects of signal 1082 do indicate that asset 1048 must stop and cannot cross signal 1082 and a stop signal exists, then the scenario is triggered, stored in the backend data center 1074 database, and an email 1124 is sent to users who have previously selected to be notified when such conditions exist.

[0200] To simplify explanation, process 1100 is depicted and described as a series of steps. However, the steps according to the present disclosure may occur in various orders and / or concurrently. In addition, the steps according to the present disclosure may occur with other steps not presented and described herein. In addition, all steps shown may not be required to implement the method according to the disclosed subject matter.

[0201] The real-time data acquisition and recording system (such as Figure 1 , 2 A connected diagnostic system for use with DARS 100, 200, 1000 (shown in Figures 100, 200, and 1000, respectively) provides real-time access to a wide range of data related to high-value assets (e.g., event and operational data, video data, and audio data), and provides an in-cab experience of a high-value asset to remotely located users, such as asset owners, operators, maintenance technicians, supervisors, control centers, and investigators. The connected diagnostic system minimizes the amount of time and effort required by personnel to perform their daily operations and maintenance of the asset. The connected diagnostic system provides multiple virtual remote instances of an operator display or onboard display and / or a human-machine interface (HMI), which can be accessed on multiple devices anywhere in the world. Various users will benefit from the connected diagnostic system of the present disclosure, such as operators, maintenance technicians, control centers, and supervisors. Operators work on assets and are responsible for operating the assets and ensuring the safety of passengers or cargo carried by the assets. Maintenance technicians are responsible for repairing and maintaining the equipment and machinery of the assets, typically when the assets are stationary in a station or marshalling yard, and maintenance technicians may be dispatched to the main line to eliminate road accidents. Supervisors are responsible for overseeing the operation and maintenance of assets. The control center is the hub for relaying information to the operators, helping them troubleshoot any issues with the asset and working closely with supervisors and maintenance technicians. The control center also provides information about the day-to-day operation of the asset, such as route information, station information, responding to passenger emergencies, etc.

[0202] Prior to the connected diagnostic system of the present disclosure, existing systems such as, for example, a monitoring and diagnostic system (MDS; FIG. 23B ) installed on a mobile asset such as, for example, a train car ( FIG. 23A ) included an onboard processing unit that collected data from the asset, systems and / or devices onboard the asset and relayed the information to an onboard display such as, for example, a train operator display (TOD; 23C ) and / or a human machine interface (HMI). As shown in FIG. 23A , the components of the train car, including the TOD, MDS, heating, ventilation and air conditioning (HVAC), brakes, doors, propulsion, event recorder system (ERS), and passenger information system (PIS) are connected to each other via a vehicle bus shown in blue lines. As shown in FIG. 23D , this information is available to each vehicle and its entire corresponding train set. Currently, using existing systems of the prior art, remote users outside the vehicle (such as a control center, supervisor, maintenance personnel, etc.) are unable to access data and / or information from outside the vehicle in real time. An operator on a vehicle must relay information via a communication channel (e.g., radio or mobile phone) to contact an off-board user, which must be done individually for each vehicle. The information relayed by the operator may be vehicle information or train information. The operator does not have the ability to access and / or relay information for the entire fleet. The disclosed system reduces the need for operators to relay information via communication channels only in special scenarios, thereby allowing a remotely located user to access data and / or information from any asset from anywhere in the world using any remote device (e.g., mobile phone, laptop, tablet, and / or any device with compatible connectivity).

[0203] The connected diagnostic system of the present disclosure provides several benefits over existing monitoring and diagnostic systems of the prior art. Namely, the connected diagnostic system of the present disclosure allows control centers and maintenance crews to rely on remote views of onboard operator displays and / or human machine interfaces (HMIs) and applicant's website information, provides simultaneous access to multiple assets and / or train sets, allows historical data to be stored on the cloud and accessible for years, provides each asset with a portable onboard display and / or HMI, maintains the software on the onboard operator display and / or HMI up to date with the latest technology via over-the-air upgrades, performs remote diagnostics, maintenance manuals and repair logs, provides asset and / or train route and crew information, and allows off-board transmission of emergency situations and alarms.

[0204] The connected diagnostics system also provides remote access to the onboard displays and / or HMI on the asset. By docking the onboard device on the asset and creating a virtual in-cab experience, the connected diagnostics system creates a remote view or remote instance of the onboard displays and / or HMI to the remotely located end user, exactly as it is displayed on the asset. The remotely located end user can view the onboard displays and / or HMI through the remote instance on their remote device and can also navigate to a different screen than the one currently displayed on the onboard display and / or HMI that the operator is viewing. The system of the present disclosure allows viewing of real-time and historical data on-board and off-board assets for an entire fleet, provides the ability to move away from an on-board display and / or HMI to provide portability, thereby providing users with the ability to troubleshoot issues regarding assets; provides simultaneous visual and audible indications of events on-board and off-board assets to remotely located end users; provides natural language processing; provides users with the ability to perform remote self-tests, error confirmation, and other maintenance-related functions, such as providing location-centric error information data that is not accessible to operators; provides the ability to automatically route real-time views of asset-assigned routes, assets, crews, etc. from back-end systems to assets, thereby reducing the need for crews to manually enter asset routes; allows integration from an on-board display and / or HMI on an asset with an off-board maintenance management system, and provides remote updates of technology to any on-board display and / or HMI, any on-board device connected to a connected diagnostic system, and / or any element connected to a connected diagnostic system via continuous over-the-air software upgrades throughout the life cycle of the asset.

[0205] For example, a remotely located user (e.g., an asset owner, operator, supervisor, maintenance crew, control center, and / or investigator) can access a universal web browser to navigate to real-time and / or historical desired data associated with a selected asset to view and analyze the asset's data, information, operational efficiency, and / or safety in real time. The ability to view operations in real time enables rapid assessment and adjustment of behavior. During an incident, for example, real-time information and / or data can facilitate classification of the situation and provide valuable information to first responders. During normal operations, for example, real-time information and / or data can be used to audit crew performance, to assist operators and maintenance crews in troubleshooting, repairing, and / or maintaining assets, and to assist in network-wide operational situational awareness.

[0206] The data may include, but is not limited to: analog and frequency parameters, such as speed, pressure, temperature, current, voltage, and acceleration originating from the asset and / or nearby assets; Boolean data, such as switch position, actuator position, warning light illumination, and actuator commands; Global Positioning System (GPS) data and / or Geographic Information System (GIS) data, such as position, speed, and altitude; internally generated information, such as the regulatory speed limit of the asset given its current location; video and image information from cameras located at various locations in, on, or near the asset; audio information from microphones located at various locations in, on, or near the asset; information sent from a data center about the asset's operating plan, such as route, schedule, and cargo manifest information; information about environmental conditions in the area in which the asset is currently operating or planned to operate, including current and forecast weather conditions; asset control status and operating data generated by systems, such as Positive Train Control (PTC) in a locomotive; and data derived from a combination of any of the above, including, but not limited to, additional data, video and audio analysis, and analytics.

[0207] Fig.24A field implementation of a first embodiment of an exemplary connected diagnostic system 1200 in which aspects of the present disclosure may be implemented is shown. The connected diagnostic system 1200 delivers real-time information from at least one data source onboard a mobile asset (e.g., a data logging device of DARS 100, 200, and 1000) and / or at least one data source remote from the mobile asset (e.g., a remote data repository of DARS 100, 200, and 1000) to a control center 1212 and remotely located end users 1214, and provides asset 1202 data and / or information to all users simultaneously, wherein all users may interact with the data onboard the asset simultaneously. The connected diagnostic system 1200 includes: a remote network platform 1208 that displays a remote instance 1206 of an onboard display and / or HMI 1210 of the mobile asset 1202; an onboard processing unit 1216, such as DARS 100, 200, 1000; an I / O device 1240; and an antenna 1238. The onboard display and / or HMI 1210 may be the onboard display and / or HMI of the mobile asset 1202 itself, or it may be included with the connected diagnostic system 1200 and installed on the mobile asset 1202. The connected diagnostic system 1200 also requires access to a transmission modem. The transmission modem may be the transmission modem of the asset 1202 itself or it may be provided with the connected diagnostic system 1200. Data accessed and / or received remotely in real time via the onboard real-time display 1210 and by a remote device is accessed from at least one data source onboard the mobile asset (e.g., a data logging device of DARS 100, 200, and 1000) and / or at least one data source remote from the mobile asset (e.g., a remote data repository of DARS 100, 200, 1000). In the illustrated embodiment herein, data is collected via the data loggers 154, 254, 1054 of the DARS 100, 200, 1000, respectively, wirelessly transmitted to the cloud via the wireless data links 146, 246, 1046 of the DARS 100, 200, 1000, respectively, the data is stored in the remote data repositories 130, 230, 1030 of the DARS 100, 200, 1000, respectively, and the data is decoded using the data decoders 136, 236, 1036 and localized using the localizers 138, 238, 1038 of the DARS 100, 200, 1000, respectively, before being displayed on the network clients 142, 242, 1042 via the network servers 140, 240, 1040 of the DARS 100, 200, 1000, respectively, as described above.The connected diagnostic system 1200 also displays data using the in-cab audio / GUI 160, 260, 1060 of the DARS 100, 200, 1000, respectively, manages data received from the connected diagnostic system 1200 using the remote data manager 132, 232, 1032 of the DARS 100, 200, 1000, respectively, and tests the data using the onboard processing unit 1216 and / or remote event detector 134, 234, 1034 of the DARS 100, 200, 1000, respectively, to look for predetermined events, scenarios, and / or incidents. Each asset 1202 includes a connected diagnostic system 1200 and a DRAS 100, 200, or 1000. The onboard data from each asset 1202 is sent to a cloud platform 1204 and can be accessed by multiple users on multiple devices anywhere in the world. The connected diagnostic system 1200 of the present disclosure provides portability, can be used by the maintenance crew for asset troubleshooting, and allows multiple instances 1206 of asset 1202 data and / or information to be accessed on remote devices using a remote network platform 1208. Multiple instances 1206 of the same onboard display and / or HMI 1210 of an asset 1202 can be accessed by multiple users, and vice versa. Any remote instance 1206 of the onboard display and / or HMI 1210 can be from any asset 1202 in the fleet and can only be accessed by a user when authenticated and authorized. Optional radio communications can also be used by the onboard crew to contact the control center 1212 and maintenance technicians.

[0208] The various technologies in the connected diagnostic system 1200 provide an onboard processing unit 1216, such as DARS 100, 200, 1000, which serves as a centralized location for data exchange between individual systems on the asset 1202 through multiple communication interfaces and / or channels. The connected diagnostic system 1200 is connected to a remote network platform 1208 and / or an onboard display and / or HMI 1210 and / or multiple onboard displays and / or HMIs 1210, which provide troubleshooting capabilities to personnel by providing recommended corrective actions. The current configuration of the connected diagnostic system 1200 serves as a source of information and / or data that collects, processes, and reports to the remote network platform 1208 and / or onboard display and / or HMI 1210 about the status of the asset 1202 system. Onboard components to which the onboard processing unit 1216 is connected include, but are not limited to: event recorders, such as information sources of DARS 100, 200, 1000 and data recorders 154, 254, 1054, respectively; displays; speedometers; control systems; passenger information and communication systems; vehicle emergency alarms; heating, ventilation and air conditioning (HVAC); brake systems; door systems; auxiliary power systems (APS); and automated passenger counters (APC).

[0209] refer to Fig.25A and25B , the connected diagnostic system 1200 may include onboard displays and / or HMIs 1210 ( Fig.27A and 27B ), which serves as an intelligent dashboard to provide information to a remotely located end user 1214 via an onboard processing unit 1214 that streams data to the remotely located end user when the asset 1202 is stationary or in motion. The remote network platform 1208 and / or onboard display and / or HMI 1210 are connected to the onboard processing unit 1216, which collects and stores information to populate the remote instance 1206 of the onboard display and / or HMI 1210, such as (for example) Fig.31A 1000, for example, to a remotely located end user 1214 and provides access to a remote network platform 1208. The remote network platform 1208 provides remote real-time viewing of the onboard display and / or HMI 1210 of the asset 1202, such as (for example) Fig.27B Onboard Train Operator Display and Fig.27C The onboard display and / or HMI 1210 allows for visibility of events occurring on the asset 1202 (e.g., errors, alarms, warnings, and information messages) and provides recommended troubleshooting steps. The remotely located end user 1214 can also use their remote device to simultaneously view the same data in the remote instance of the onboard display and / or HMI 1210, which allows the remotely located end user 1214 to remotely view any information and / or data onboard the asset 1202. The HMI can be used on the remote network platform 1208 as well as the onboard display 1210. When multiple assets 1202 are connected to each other, Fig.26A and 26B , multiple onboard processing units 1216 communicate with their respective onboard devices 1218 and onboard displays and / or HMIs 1210 to send data to the cloud platform 1204. This data is viewed on a remote network platform 1208, where multiple onboard displays and / or HMIs 1210 from each of the assets 1202 can be accessed simultaneously.

[0210] The onboard processing unit 1216 of the connected diagnostic system 1200 can be connected to various wired and wireless communication channels onboard the asset 1202 and outside the asset 1202 to provide an onboard display and / or HMI 1210 ( Fig.25A and 25B )'s remote view 1226( Fig.27C ) and allows remote visibility of onboard systems to drive troubleshooting and diagnostics. The wired and / or wireless data links of the onboard processing unit 1216 may include any or a combination of discrete signal input and / or output and / or analog signal input and / or output, standard or proprietary Ethernet, serial connections, and wireless connections. Onboard asset communication channel 1228 ( Fig.34 , 35 Examples of onboard asset communication channels 1228 include, but are not limited to, serial communications, controller area network (CAN), local operating network (LON), Ethernet, multifunction vehicle bus (MVB), or any other means for transmitting data from the onboard processing unit 1214 of the connected diagnostic system 1200 to, in this example, an onboard display and / or HMI 1210. Examples of onboard asset communication channels 1228 are described in Fig.28A , 28B and 28C are shown in three illustrative exemplary implementations. Fig.28A A train car 1400 is illustrated, wherein communication channels connecting components of an onboard processing unit 1216 (a train operator display 1210, an Ethernet transmission modem 1402, and input / output (I / O) devices 1240) via an Ethernet network 1232 are shown as yellow lines, communication channels connecting components of the onboard processing unit 1216 (an event recorder system (ERS) 1406, HVAC 1408, doors 1410, propulsion 1412, brakes 1414, and a passenger communication system 1416) via a LON interface 1234 are shown as blue lines, and communication channels connecting components of the onboard processing unit 1216 (ERS 1406) via a serial connection 1236 are shown as green lines. Fig.28B Illustration of a train car 1420, wherein the communication channels of the components (train operator display 1210, Ethernet transmission modem 1422, APC 1424, passenger information system (PIS) 1426, I / O devices 1428, DARS Internet of Things (IoT) gateway 1216, overhead line camera 1432) connected via Ethernet 1232 are shown as yellow lines, and the communication channels of the components (HVAC 1434, APS 1436, doors 1438, traction control unit 1440, brakes 1442 and DARS IoT gateway 1216) connected via LON interface 1234 are shown as blue lines. Fig.28CIllustration of a train car 1450 with communication channels for components connected via Ethernet 1232 (train operator display 1210, Ethernet transmission modem 1452, APC 1454, PIS 1456, I / O devices 1240, and DARS Internet of Things (IoT) gateway 1216) shown as yellow lines and communication channels for components connected via LON interface 1234 (HVAC 1462, APS 1464, doors 1466, traction control unit 1468, brakes 1470, and DARS IoT gateway 1216) shown as blue lines. Offboard asset communication channels 1230 ( Figures 34 to 37 Examples of ) include, but are not limited to, transmitting data from an onboard processing unit 1216 of the connection diagnostic system 1200 to, in this example, a remotely located end user 1214 ( Fig.24 ), wireless local area network (WLAN), wireless metropolitan area network (WMAN), wireless wide area network (WWAN), private wireless system, satellite network, cellular telephone network or any other component. Fig.28A , 28B Examples of the onboard asset communication channel 1230 are shown in the three illustrative exemplary implementations in FIG. 28C. Fig.28A A train car 1400 is shown with the communication channels of components (ie, onboard processing unit 1216, such as a DARS IoT gateway) connected via antenna 1238 shown as orange lines. Fig.28B A train car 1420 is shown with the communication channels of components, namely the onboard processing unit (DARS IoT Gateway) 1216 , connected via antenna 1238 shown as orange lines. Fig.28C A train car 1450 is shown with the communication channels of components connected via antenna 1238, namely the onboard processing unit (DARS IoT Gateway) 1216, shown as orange lines.

[0211] Wireless connectivity enables real-time location information and navigation data, which allows remote tracking of assets 1202 and provides accurate availability and service information to users. Diagnostic logs from onboard processing unit 1216 are streamed to cloud platform 1204 in real time and can be accessed by control center 1212 for reliable serviceability. In the event of an incident, data and / or diagnostic logs are optionally stored in an integrated or standalone crash-hardened memory module 1220 ( Fig.25A and 25B) and / or the crash-hardened memory module 118 of DARS 100, the crash-hardened memory module 218 of DARS 200, optionally stored in the non-crash-hardened storage device 219 of DARS 200 and / or the crash-hardened memory module 1018 of DARS 1000, and still accessible on and / or off the asset 1202. For example, the exemplary hardened memory module 1220 may be: a crash-hardened event recorder memory module that complies with Federal Railroad Administration regulations of federal regulations; a crash-survivable memory unit that complies with federal regulations and / or Federal Aviation Administration regulations; a crash-hardened memory module that complies with any applicable federal regulations; or any other suitable hardened memory device as known in the art.

[0212] The onboard processing unit 1216 collects data from various devices, including but not limited to: an onboard device 1218 and data loggers 154, 254, 1054 of the DARS 100, 200, 1000, respectively; and information sources of the DARS 100, 200, 1000, which may include any number of components in the assets 148, 248, 1048, such as any of the following: analog inputs 102, 202, 1002, digital inputs 104, 204, 1004, I / O modules 106, 206, 1006, vehicle controllers 108, 208, 1008, engine controllers 110, 210, 1010, inertial sensors 112, 212, 1012, global Positioning system (GPS) 114, 214, 1014, camera 116, 216, 1016, active train control (PTC) / signal data 166, 266, 1066, fuel data 168, 268, 1068, cellular transmission detector (not shown), internal drive data and any additional data signals and any number of components in the data center 150, 250, 1050, such as any of the route / crew list components 124, 224, 1024, weather components 126, 226, 1026, map components 164, 264, 1064 and any additional data signals, and making the data immediately available on the cloud platform 1204 for remote monitoring of the asset 1202.

[0213] Prior to the connected diagnostic system 1200 of the present disclosure, existing systems allowed for onboard or off-board viewing of data generated by onboard devices and were typically limited to data from only a single asset or data from a train. Figure 29 provides insight into the workflow of the interaction between an operator, a control center, and a maintenance crew in an existing system. When a vehicle is in operation and there is a problem, the operator typically must contact the control center via radio. The control center then initiates a back-and-forth conversation between the operator and the maintenance crew to troubleshoot the problem or dispatch a crew to further troubleshoot. Dispatching is required in situations where the crew is unable to remotely assist in troubleshooting the vehicle. Timely resolution is required to reduce service disruptions for passengers and maintain customer satisfaction.

[0214] refer to Fig.30 , which provides a view of the workflow of the connected diagnostic system 1200 between operators 1222, control center 1212, and supervisors and / or maintenance crews 1224, which allows for simultaneous remote and onboard viewing of data from onboard displays and / or HMIs 1210 of any asset 1202 in the fleet. Remote access to data can be viewed on multiple devices, including (but not limited to) desktop computers, tablets, mobile phones, and / or other handheld devices. Supervisors, control centers, and other remotely located end users 1214 can remotely view operator displays 1210 ( Fig.28D ), and the connected diagnostic system 1200 provides the remotely located end user 1214 with the ability to navigate to a screen different from the view on the onboard display and / or HMI 1210 onboard the asset 1202, which creates a virtual in-cab experience and allows real-time action to allow remote guidance, troubleshooting, and repair of the asset 1202. The connected diagnostic system 1200 enables rapid problem resolution by reducing the time required to relay information and minimizes service interruptions. In some cases, the control center 1212 is able to dispatch a maintenance crew 1224 before the operator 1222 calls for assistance because the control center 1212 has access to all onboard information. The maintenance crew 1224 also has access to the same information as the operator 1222 and the control center 1212, which helps them quickly identify problems and facilitates faster resolution.

[0215] Prior to the connected diagnostic system 1200 of the present disclosure, existing systems provided access to current data for a single device or on a vehicle in a train. This data was only available on or off the vehicle.

[0216] The connected diagnostic system 1200 is not limited to just a single asset or train. The connected diagnostic system 1200 allows the user to access data for any asset 1202 in the fleet. Using the connected diagnostic system 1200, data is available both on the asset 1202 and off the asset 1202, allowing supervisors and maintenance crews 1224 to compare the performance of an existing asset 1202 with a problem with another set of assets in the fleet to quickly identify and resolve the problem.

[0217] Prior to the connected diagnostic system 1200 of the present disclosure, existing systems only provided access to data from onboard devices on the vehicle via an onboard display and / or HMI. This data was only available on or off the vehicle. Some onboard devices of existing systems also had limited storage capabilities.

[0218] The connected diagnostic system 1200 provides access to current and historical data obtained from the onboard device 1218. Access to historical data is important, for example, by providing supervisors and maintenance personnel with access to historical errors and alarms to determine health trends of the fleet or asset to enable future maintenance decisions. Using the connected diagnostic system 1200, data is available both on the asset 1202 and off the asset 1202. In the illustrated embodiment, the connected diagnostic system 1200 can support up to 100 days of historical data storage, which can be expanded on the asset 1202 as needed. Off the asset 1202, data can be stored for up to 2 years and can also be expanded as needed.

[0219] Prior to the connection diagnostic system 1200 of the present disclosure, existing systems included an onboard display and / or HMI fixed to the vehicle's cab control panel. When there is a problem on the vehicle and the onboard display and / or HMI need to be viewed simultaneously for troubleshooting, this connection mode of the fixed onboard display and / or HMI is not useful to the crew.

[0220] The connected diagnostic system 1200 allows the onboard display and / or HMI 1210 to be detached, which allows the user to move the onboard display and / or HMI 1210 to a desired location throughout the asset 1202 for troubleshooting. The portability of the onboard display and / or HMI 1210 also allows the user to capture images and videos using the onboard display and / or HMI 1210 equipped with a built-in camera (not shown) to troubleshoot the asset 1202. For example, in the existing system, when there is a problem with the door system, the maintenance crew must walk back and forth to solve the problem between the cab and the door. The connected diagnostic system 1200 allows the maintenance crew or technician 1224 to view the data on the onboard display and / or HMI 1210 at the same location where the door is located, for example, while troubleshooting the problem, thereby reducing the time and effort required for troubleshooting. Other examples include, but are not limited to, viewing engine data, light rail vehicle (LRV) leveling to adjust suspension systems for alignment on the platform, pantograph raising, visual inspection of alignment with status on the display, automated checklists on the display - visual inspection and confirmation via screen, and an integrated tablet camera for augmented reality (AR), which allows for an augmented reality troubleshooting experience that can, for example, pinpoint failed components, make recommendations for repaired components, and provide a walkthrough of troubleshooting steps.

[0221] Prior to the connected diagnostic system 1200 of the present disclosure, existing systems provided access to current software version information on or off-vehicle for a single vehicle or fleet view.

[0222] The connected diagnostic system 1200 provides access to current and historical software version information and release dates for all onboard devices 1218. This data is available both on the asset 1202 and off the asset 1202. The connected diagnostic system 1200 is able to maintain a comprehensive record of software updated on all assets 1202, which is helpful during troubleshooting and, in the illustrated exemplary implementation, enables the railroad to comply with operating and maintenance practices, record keeping, configuration management, and change control management.

[0223] Prior to the connected diagnostic system 1200 of the present disclosure, existing systems only provided visual and audible notifications (including natural language processing) on ​​or outside the vehicle of vehicle-related events, including (but not limited to) alarms (such as passenger emergencies), delays, severe weather warnings, errors generated by onboard devices (such as a door opened in transit), information messages, and route information.

[0224] The connected diagnostic system 1200 provides simultaneous visual and audible notifications (including natural language processing) of asset-related events both on the asset 1202 and off the asset 1202. This data is simultaneously available both on the asset 1202 and off the asset 1202, thereby allowing a remotely located user to view a copy of the onboard display and / or HMI 1210, while also allowing the remotely located user to access information that is not currently displayed on the onboard display and / or HMI 1210 but is available to the onboard display and / or HMI 1210. For example, the onboard display and / or HMI 1210 and the remote view 1226 of the onboard display and / or HMI may display the same screen, such as Fig.32A The operation screen 1480 of the train compartment 1400 is shown in FIG. Fig.32B A first example of a fault screen 1482 of a train car 1400 is shown in FIG. Fig.32C A second example of a fault screen 1484 of a train car 1400 is shown in FIG. Fig.32D The maintenance access screen 1486 of the train car 1420 is shown in FIG. Fig.32E The maintenance system status screen 1488 of the train car 1400 is shown in FIG. Fig.32F The maintenance subsystem status screen 1490 of the train car 1400 is shown in FIG. Figure 32G The on-demand real-time signal screen 1492 of the train car 1400 is shown in FIG. Fig.32H The speedometer screen 1494 of the train carriage 1400 is shown in FIG. Fig.32I The setup screen 1496 of the train car 1420 shown in FIG. Fig.32J The settings screen 1496 of the train car 1420 showing the brightness toggle radio button 1498 and Figure 32K and 32L Operation screen 1480 of train car 1500 is shown.

[0225] By providing instant access to both real-time and historical data on and off the asset 1202, users on the asset 1202 and remotely located users can also benefit from real-time alerts, playback serviceability, reporting, and analytics. For example, the onboard display and / or HMI 1210 and the remote view 1226 of the onboard display and / or HMI can display Figure 32M The operation screen 1502 of the train compartment 1400 is shown in FIG. Figure 32N32O , a maintenance screen 1508 of a train car 1420 showing an asset configuration history 1510 including version information, and a setup screen 1514 of a train car 1506 showing a train car 1506 being debugged 1516, shown in FIG. 32P . Insights gained from real-time monitoring are used to prevent failures and reduce unplanned downtime. Due to the availability of real-time diagnostics, instant access to both real-time and historical data enables scheduling of asset 1202 maintenance. In this exemplary embodiment, the connected diagnostic system 1200 allows remote visibility of events for all assets 1202 in the train and the ability to monitor any other asset 1202 in the fleet. Views of error log information, error occurrence and reset times, error severity levels, etc. can be categorized based on date, time, or system. Asset 1202 data is combined with other operational and contextual information (e.g., rail-centric location and weather data) to better understand the conditions that may contribute to errors and problems. Errors or events on asset 1202 may also be confirmed by crew not present on asset 1202 and any remotely located end users 1214 .

[0226] Prior to the connected diagnostic system 1200 of the present disclosure, existing systems allowed self-testing or diagnostics of onboard systems to be initiated only on a single vehicle or on vehicles in a train mode. Some existing systems only support this functionality in a limited capacity.

[0227] The connected diagnostic system 1200 allows self-checks of the onboard device 1218 to be run on the asset 1202 and also to be run remotely for one asset 1202 , selected assets 1202 , and / or for all assets 1202 in the fleet simultaneously, which reduces the effort expended by the maintenance crew 1224 to run self-checks and provide reporting for all assets 1202 .

[0228] Prior to the connected diagnostic system 1200 of the present disclosure, existing systems required the vehicle crew to enter the train's assigned route, which could potentially lead to human error, resulting in potential delays in vehicle service or reporting of incorrect passenger and station information.

[0229] The connected diagnostic system 1200 allows assets 1202 to be automatically assigned to routes, trains, crews, etc. from the backend system, reducing and / or minimizing the need to manually enter train routes into only specialized scenarios in this exemplary embodiment. Off-board personnel (such as supervisors 1224 and control center 1212) can also access real-time information of the train's route information.

[0230] Prior to the connected diagnostic system 1200 of the present disclosure, existing systems maintained repair histories, operation manuals, and maintenance manuals outside of the vehicle. This increased the time required for the maintenance crew to troubleshoot or refer to existing open defects or repair histories on the vehicle.

[0231] The connected diagnostic system 1200 allows the operator 1222 and / or the asset 1202 to connect and integrate with other off-board maintenance management systems from the on-board display and / or HMI 1210 on the asset 1202, for example, to order parts from the on-board display and / or HMI 1210, look up electronic manuals for off-board maintenance, review repair history, view all open issues and / or defects on the asset 1202, etc.

[0232] Prior to the connected diagnostic system 1200 of the present disclosure, onboard displays and / or HMIs and user interfaces of existing systems typically did not continually and / or periodically evolve with the latest technology trends once they were shipped from the factory and therefore were not updated with any new features, such as to include pinch and zoom, touch screens, or modern data graphical representations as they became available.

[0233] The connected diagnostic system 1200 allows the technology of the onboard display and / or HMI 1210, onboard systems and / or onboard subsystems to be updated remotely via over-the-air software updates. The connected diagnostic system 1200 also allows the technology of the onboard display and / or HMI 1210, onboard systems and / or onboard subsystems to be updated in the field by allowing a user (e.g., an operator or maintenance crew) to download the software update and deploy it in the field or by connecting a removable storage device that includes the update. The ability to continuously make these updates throughout the life cycle of the asset 1202 improves maintenance efficiency by providing the ability to utilize modern tools and technologies.

[0234] Prior to the connected diagnostic system 1200 of the present disclosure, onboard data from existing systems was only available on the vehicle or was only available for transfer at less frequent intervals, such as once a day or once a week, or required downloading the onboard data via an onboard device physically connected to the vehicle. Most often, users were unable to capture a fault that occurred on the road for troubleshooting.

[0235] Connected diagnostic system 1200 provides remote access to view operational data of asset 1202, including higher level data such as engine data, front end power (HEP) equipment data, and electrical and power system data, on a second by second or periodic snapshot basis (e.g., every five minutes). Data from asset 1202 can be obtained on demand or based on conditions and / or rules set by the user. Remote data access to settings on demand can also be used to detect when specific asset conditions are met and can be utilized by the user via alerts, reports, etc. This enables the user to understand when data related to specific incidents occurs on the road and remotely dispatch asset 1202 logs to record incidents and / or failures that occur on the road.

[0236] Fig.34 1 is a flow chart of a process 1250 for real-time transmission of data from an onboard system and / or onboard subsystem to an onboard display and / or HMI 1210 and remote access of the onboard display and / or HMI 1210 according to an embodiment of the present disclosure. DARS 100, 200, 1000 are features of the onboard processing unit 1216 software, and the onboard processing unit 1216 is a hardware component connected to the diagnostic system 1200. In this exemplary embodiment, the asset 1202 includes several onboard devices and / or onboard systems, such as doors, brakes, HVAC, propulsion, engine controllers, and fuel systems, which are further subdivided into individual onboard subsystems, such as, for example, door A1, door A2, brake A, brake B, etc. The onboard systems and subsystems are critical to the operation of the asset 1202. The onboard systems and subsystems monitor and sense data and provide information to enable efficient control and operation of the asset 1202. Onboard systems and subsystems 1252 communicate through onboard communication channels 1228 and send data to an onboard processing unit 1216 for further analysis 1254. The onboard processing unit 1216 is responsible for processing data from the various systems connected to it, providing advanced data fusion capabilities, self-monitoring, and integration with third-party devices (such as cameras) to allow accident investigations. Using the data received from the onboard systems, the onboard processing unit 1216 can receive and analyze the status of the asset 1202 and its surrounding infrastructure (such as tracks, speed limits, collisions, and simultaneous real-time monitoring). The data from the onboard systems and subsystems is decoded into individual signals and time-synchronized 1256 to provide meaningful data for user consumption and additional processing by other onboard applications. The decoded data is available on the data bus for simultaneous use by multiple applications that process the data in real-time and in parallel 1258. The individual signals are stored in a compressed data structure so that they can be sent to onboard systems and offboard systems 1260. The data is compressed to maximize the capacity of the data onboard the asset 1202 and enable faster transfer to the offboard system.

[0237] The process 1250 then determines whether the asset 1202 is equipped with an onboard crash-hardened memory module 1262 using the onboard processing unit 1216. If the asset 1202 includes a crash-hardened memory module 1220, the compressed data is stored in the crash-hardened memory module 1220 1264. The crash-hardened memory module 1220 stores the data for a predetermined amount of time, such as at least 48 hours, in case a catastrophic event occurs. The predetermined amount of time is configurable and can be adjusted according to customer needs. The crash-hardened memory module 1220 provides access to the compressed data even when power is cut off, and in the exemplary implementation described herein, maintains the standards set by the Federal Railroad Administration (FRA) and the Institute of Electrical and Electronics Engineers (IEEE). If the asset 1202 does not include a crash-hardened memory module, the limited historical data is stored locally on a flash drive, a solid-state drive, and / or any other local storage device 1266.

[0238] At the same time, the process 1250 uses the onboard processing unit 1216 to determine whether the data is going to an onboard system, an offboard system, or both an onboard system and an offboard system 1268. Data from onboard is sent to a device on the asset 1202 and to an offboard system simultaneously. If the data is going to an onboard system, the uncompressed data is sent via the onboard communication channel 1228 1270 to the onboard display and / HMI 1210 1272, which provides visual and audible information (including natural language processing) to the asset 1202 operator 1222 and maintenance crew 1224, and to the onboard system and / or onboard subsystem 1274. Where the data is used to send a response back to the same onboard system and / or make data (including request data and / or information) from other onboard systems available to the onboard system, available to the onboard systems and subsystems. If the data is going to an off-board system, the data is compressed by the data encoder 122, 222, 1022 of the DARS 100, 200, 1000, respectively, and the compressed data is sent from the on-board system to the off-board system 1278 via the off-board communication channels 1230 1276. The compressed data is also stored in a database 1280 (e.g., the crash-hardened memory module 118, 218, 1018 of the DARS 100, 200, 1000, respectively) and optionally the non-crash-hardened removable storage device 219 of the DARS 200. The compressed data is then sent to a cloud-based real-time server 1284. Once received, the compressed data is then decompressed 1282 using the data decoder 136, 236, 1036 of the DARS 100, 200, 1000 and sent to a cloud-based real-time server 1284 for storage and retransmission for easier accessibility. Data from the cloud-based real-time server can be accessed by both onboard and / or offboard platforms simultaneously. Processed data stored on the cloud-based real-time server can be viewed via a remote web platform 1208 on multiple remote platforms 1286 (e.g., tablets, mobile phones, and laptops) that provide remote access to onboard displays and / or HMI 1210. The remote platform, including any remote device connected to the Internet, provides a web-based viewer that allows users to get data remotely and in real time, which makes data easier to integrate and provides cloud-based access to real-time monitoring capabilities to, for example, view the geographic location of assets and the status of onboard systems and subsystems.

[0239] While the data is stored 1262 and transmitted to the onboard system and / or offboard system 1268, the decoded data is processed to check for anomalies 1288, which includes analysis of anomalies generated directly by the onboard system (e.g., errors generated) or by rules defined to set thresholds for a particular signal and / or set of signals, which is necessary to understand whether the onboard system is performing above normal thresholds and / or exceeding defined thresholds. Anomalies may also include, but are not limited to, temperature, speed, acceleration, brake status, coolant temperature, outside air, inside air, and system health. The user may define rules and thresholds, and may also set the frequency of anomaly alerts. Thresholds may be set internally and the user may define minimum thresholds, maximum thresholds, and threshold ranges. The process 1250 then uses the onboard processing unit 1216 to evaluate the processed signals and determine whether any anomalies are triggered 1290, such as, for example, an anomaly that triggers a speeding event when the speed of the asset 1202 is greater than the previously established speed limit of 45 miles per hour. If no anomaly is triggered, the process 1250 continues with determining whether the data is going to an onboard system, an offboard system, or both an onboard system and an offboard system 1268. If an anomaly is triggered from the processed data, audible and visual alerts (including natural language processing, events, errors, etc.) are generated from the triggered anomaly, and notifications to the onboard and offboard crews are enabled 1292 for items that require further action or investigation, such as, for example, alerting the crew to stay within prescribed speed limits. The process 1250 then continues with determining whether the data is going to an onboard system, an offboard system, or both an onboard system and an offboard system 1268 using the onboard processing unit 1216.

[0240] Fig.351300 is a flow chart of a process 1300 for transmitting historical data from an onboard system to an onboard display and / or HMI 1210 in real time and accessing the onboard display and / or HMI 1210 remotely simultaneously according to an embodiment of the present disclosure. DARS 100, 200, 1000 is a feature of the onboard processing unit 1216 software, and the onboard processing unit 1216 is a hardware component of the connected diagnostic system 1200. The process 1300 is triggered by one of two events: when a user present on the asset 1202 makes a request to access historical data on the asset 1202 using the onboard display and / or HMI 1210; and when a remotely located end user 1214 makes a request to access historical data on a remote platform and / or web client. The process 1300 starts 1302 when a request to view historical data is received from a user onboard the asset 1202 or from a remotely located end user 1214 through the web platform 1208. If the request is received from a remote network platform 1208, which includes access 1304 to a remote instance of an onboard display and / or HMI 1210, then the data is accessed from the cloud storage device 1306 and sent to the remote network platform 1208 for consumption by the user. Data previously accessed from the onboard system and onboard display and / or HMI 1210, including any data and events that occurred from the onboard system and onboard display and / or HMI 1210, is stored in the cloud. Storing data in the cloud allows access to historical data over a longer period of time compared to the onboard processing unit 1216. The onboard processing unit 1216 typically has a limited storage capacity of approximately 100 days of data, which can be expanded as needed. By using a cloud-based storage system, the onboard system can access data from the cloud, including historical data, over a longer period of time, which is typically two years and can also be expanded as needed.

[0241] The request for historical data is then sent 1308 via the offboard asset wireless communication channel 1230. The request is processed by the onboard processing unit 1216 to determine the appropriate onboard system and / or onboard subsystem to receive the information 1310. The request for historical data to be accessed by the onboard display and / or HMI 1210 may include, for example, a historical view 1312 of generated events, errors, etc. The data received from the cloud is then transmitted 1304 via the onboard asset wireless communication channel 1230 1308 to the remote network platform 1208. The remote instance 1206 of the onboard display and / or HMI 1210 displays the historical data requested by the user.

[0242] If the request is received from an onboard platform, the process 1300 determines 1320 using the onboard processing unit 1216 whether the requested data is in local cached data available on the onboard processing unit 1216. If the cached data is not available, the request is processed by the onboard processing unit 1216 to determine the appropriate onboard system and / or onboard subsystem to retrieve the information 1310, and the request to retrieve the data goes offboard. The request for historical data to be accessed by the onboard display and / or HMI 1210 may include, for example, a historical view 1312 of generated events, errors, etc. The data received from the appropriate onboard system and / or subsystem is then communicated to the onboard display and / or HMI 1210 1314 via the onboard asset wired communication channel 1228. The onboard display and / or HMI 1210 displays the data requested by the user 1316, including visual and audible information and natural language processing, and the data may be further used to send responses (which may include the requested data and / or information including errors, self-tests, etc.) to the onboard systems and onboard subsystems 1318. If cached data is available, the cached data including historical data requested by the user is utilized 1322 and sent to the onboard processing unit 1216 1310. The request for historical data to be accessed by the onboard display and / or HMI 1210 may include, for example, a historical view 1312 of generated events, errors, etc. The data received from the local cache is then transmitted to the onboard display and / or HMI 1210 1314 via the onboard asset wired communication channel 1228 and / or the offboard asset wireless communication channel 1230. The onboard display and / or HMI 1210 displays the user requested data 1316, including visual and audible information and natural language processing, and the data may be further used to send responses (which may include requested data and / or information including errors, self-tests, etc.) to onboard systems and onboard subsystems 1318.

[0243] Fig.3613 is a flow chart showing a process 1330 showing the flow of data from a remote platform to an onboard system and onboard display and / or HMI 1210 for performing certain user actions and / or updating software according to an embodiment of the present disclosure. A remote network platform 1208 allows a user to take actions and / or send commands to an onboard system, which allows a remotely located end user 1214 to maintain and monitor an asset 1202. There are established rules and regulations that are set to allow or deny requests sent from a remotely located end user 1214. The process 1330 uses the remote network platform 1208 to monitor the remote network platform 1208 for requests 1332 for actions and / or commands from a remotely located end user 1214, such as, for example, errors, self-tests, and diagnostics. The process 1330 uses the remote network platform 1208 to determine whether a user action is a request 1334 through the remote network platform 1208. If the remote network platform 1208 does not receive a request for an action and / or command from the user 1214, the process 1330 continues with the remote network platform 1208 monitoring the remote network platform 1208 to capture any requests 1336 sent from the remote platform by the remotely located end user 1214. If the remote network platform 1208 does receive a request for an action and / or command from the user 1214, the request is sent 1338 from the remote network platform 1208 to the onboard processing unit 1216 via the offboard asset wireless communication channel 1230. The onboard processing unit 1216 accepts incoming requests 1340 sent by any remote platform. The onboard processing unit 1216 determines the nature of the request 1342 made to the onboard system, such as self-test (e.g., a self-test of a vehicle door may be completed remotely by triggering a command, and once completed, the vehicle door system and / or subsystem may report the status of its self-test), error confirmation, and software update (e.g., updating the software of an onboard system).

[0244] The onboard processing unit 1216 determines whether the request is to remotely update software 1344. If the request is not to remotely update software, the onboard processing unit 1216 determines the nature of the user action request made to the onboard system 1346. The onboard processing unit 1216 then determines whether the request is for the onboard display and / or HMI 1210 or the onboard system and / or onboard subsystem 1348. If the request is for the onboard display and / or HMI 1210, the onboard processing unit 1216 performs the requested user action and the onboard display and / or HMI 1210 displays the user requested action 1350, including visual and audible information and natural language processing. If the request is for an onboard system and / or subsystem, the onboard processing unit 1216 identifies the onboard system and / or onboard subsystem to which the request of the user 1214 is sent 1352.

[0245] If the request is to update software remotely, then the onboard processing unit 1216 determines whether the request is to update the software on the onboard processing unit 1216 1356. If the request is to update the software on the onboard processing unit 1216, then the software on the onboard processing unit 1216 is updated 1358. The onboard processing unit 1216 software is updated periodically to include any new features or vulnerabilities that are fixed as part of the software development and release process. The version information of the specific onboard system and / or subsystem whose software is updated is then made available 1354 to the remote network platform 1208 and onboard display and / or HMI 1210. The software update can also be accomplished directly from the onboard system and / or onboard display and / or HMI 1210 using the wireless communication channel 1230 of the onboard processing unit 1216. The onboard processing unit 1216 then continues to monitor for incoming requests 1340 sent by any remote platform.

[0246] If the request is not to update software on the onboard processing unit 1216, the onboard processing unit 1216 then determines whether the request is to update software on the onboard display and / or HMI 1210 or on the onboard system and / or onboard subsystem 1360. If the request is to update software on the onboard display and / or HMI 1210, the onboard processing unit 1216 determines which specific onboard displays and / or HMIs 1210 need to be updated 1362. The software on the specific onboard display and / or HMI 1210 is updated, and the onboard display and / or HMI 1210 displays version information 1350 of the completed software update, including any visual and audible information and natural language processing. Additionally, in another exemplary implementation, the software update and version information of the software update may be displayed via a pop-up window both on the asset 1202 on the onboard display and / or HMI 1210 and remotely from the asset 1202 on a remote instance of the onboard display and / or HMI on the remote network platform 1208. If the request is to update software on an onboard system and / or subsystem, then the onboard processing unit 1216 determines which specific onboard systems and / or onboard subsystems require updating 1364. The software on the specific onboard systems and / or subsystems is updated, and the version information of the specific onboard systems and / or subsystems for which the software was updated is then made available 1354 on the remote network platform 1208 as well as on the onboard display and / or HMI 1210.

[0247] Fig.371370 is a flow chart showing a process 1370 of data integration between third-party devices for obtaining asset 1202 information and / or data integration with a customer / asset 1202 maintenance management system for providing a real-time view of information within a connected diagnostic system 1200 according to an embodiment of the present disclosure. In the exemplary embodiment described herein, assigning vehicles to trains is a complex task that is typically undertaken by a third-party service or customer. Assignment also involves crew information, route information, train schedules, station information, etc. The assignment data (including (but not limited to) train assignments, route information, crew information, train schedules, and station information) is automatically integrated into the onboard display and / or HMI 1210 1372. The customer and / or asset management system 1374 contains various information 1374, including (but not limited to) vehicle problems, defects, out of service information, electronic manuals, repair history, etc., which is valuable for troubleshooting the vehicle crew. Data from the connected diagnostic system 1200 and / or third-party devices (e.g., train information 1372 and / or customer / asset maintenance management system 1374) is integrated via a wired communication channel, a wireless communication channel, and / or a business-to-business (B2B) communication channel 1376. Data received from the connected diagnostic system 1200 and / or third-party devices (including train information 1372 and / or customer / asset maintenance management system 1374) may be processed and analyzed 1378 before being sent to the cloud. If the data is processed, the data is analyzed, e.g., for anomaly discovery, error checking, checking the validity of the data, and / or further processing by the DARS 100, 200, 1000. The data is sent to the cloud for storage purposes and may be evaluated 1380 by onboard and / or offboard systems as needed. The remote network platform 1208 can access data 1382 on the cloud by remotely accessing the onboard display and / or HMI 1210 using remote devices such as tablets, mobile phones, and laptops, which allows for simultaneous viewing of connected diagnostic system 1200 data and / or third-party data on and off the asset 1202. The data is sent 1384 to the onboard processing unit 1216 via the off-board asset wireless communication channel 1230 for further dissemination 1386. The onboard processing unit 1216 sends 1388 the data to the onboard display and / or HMI 1210 1390 and / or onboard systems and / or subsystems 1392 via the onboard asset wired communication channel 1228. The data from the third-party devices can be used to provide access to data such as, for example, passenger information required for efficient operation and maintenance by the onboard systems and / or subsystems. The onboard display and / or HMI 1210 may provide access to electronic manuals accessed by third party integration between the customer / asset maintenance management system and the onboard processing unit 1216 .The onboard display and / or HMI 1210 may also access additional third party data, such as repair history, problems, and defects, and provide a real-time view of third party data, such as the assigned routes, trains, and crews of the asset 1202 .

[0248] To simplify explanation, processes 1250, 1300, 1330, and 1370 are depicted and described as a series of steps. However, the steps according to the present disclosure may occur in various orders and / or concurrently. In addition, the steps according to the present disclosure may occur with other steps not presented and described herein. In addition, all of the steps shown may not be required to implement the method according to the disclosed subject matter.

[0249] The real-time data acquisition and recording system described herein (such as (for example) respectively in Figure 1 , 2 The operational threat detection system used with the DARS100, 200, 1000 shown in Figures 1 and 20 is a system that provides mobile assets with the ability to obtain and maintain compliance with domestic and / or foreign regulations. The system can work with any brand and model mobile asset, any mobile asset configuration and / or any mode of transportation and / or operator. Although the operational threat detection system and method of the present disclosure have been described in conjunction with the following illustrative exemplary implementation scheme related to locomotives, it is not intended to be limited to the specific form set forth herein, but on the contrary, it is intended to cover such alternatives, modifications and equivalents that can be reasonably included in the spirit and scope of the invention as defined by the appended claims. For example, Canadian Department of Transport Regulation SOR / 2020-178 requires railways to install cameras and microphones in lead locomotives in passenger and freight service on Canadian tracks, with some exceptions. Similarly, proposed Federal Railroad Administration (FRA) regulations FRA-2016-0036 require railroads to install inward-facing and outward-facing cameras on control locomotives in passenger service in the United States, with some exceptions, and 49 C.FR §240 sets forth qualification and certification requirements for locomotive engineers. These regulations also specify various procedures that define the conditions under which video and / or audio from connected cameras and / or microphones may be used while maintaining crew privacy, as well as the method and frequency of verifying digital video recorders on each mobile asset. The operational threat detection system of the present disclosure is supported by applicant's own digital video recorders and OEM digital video recorders, minimizing the amount of time and effort and maximizing the investment value of mobile assets by providing useful video and / or audio data using advanced algorithms and automated system testing in accordance with regulations.

[0250] In the following exemplary embodiments, the operational threat detection system of the present disclosure provides: supporting regulatory compliance by automating the selection of video and / or audio data of interest based on customer configuration and / or regulatory requirements by leveraging second-by-second locomotive, rail vehicle and / or mobile asset data, gyroscope data and integration with railway systems (e.g., train integration), and video data from connected cameras to reduce or eliminate manual work and potential human errors; automating the scanning of video and / or audio data to detect safety-critical events by leveraging artificial intelligence techniques for video and / or audio processing and processes developed using gyroscope data, GPS data, Wi-Power-III data, and / or event recorder data, thereby providing valuable insights into security policy creation and compliance status; automated self-monitoring technology to ensure compliance with regulations and provide video and / or audio data availability for critical incident investigations; an advanced IoT platform to allow for the detection of safety-critical events in addition to application Perform video and / or audio processing on the edge or other Internet browsers for safety-critical event detection in addition to data processing capabilities on the cloud; an extensible framework that allows for the use of additional data sets available on rail vehicles in future new data processing and the use of video and / or audio-based artificial intelligence techniques to detect events critical to rail safety and operations; a consistent approach to data selection and system validation that can be applied across the railroad regardless of personnel changes; a method for automatic and / or on-demand retrieval of tens or hundreds of inward-facing and outward-facing camera video and / or audio recordings of configurable predefined durations; allowing authorized personnel to view and share downloaded video and / or audio files as well as gyroscope data, event recorder data, and location data at any time from any location using a web-based mobile-optimized player; and access to all video and / or audio files and data for mobile assets by authorized users using role- and asset-relationship-based security. These features are implemented using an automated configuration approach to enable seamless operation across any brand, model, and / or asset configuration. In this illustrated exemplary embodiment, the operational threat detection system 1600 is applicable to any model rail vehicle, any rail vehicle configuration, and / or any railroad and / or operator, and works with any brand and model of digital video recorder.

[0251] With the above regulations in effect, a manual process may be used for mobile assets. For example, a first manual method for random downloading and threat detection includes selecting a day of the month using a random download generator to manually download video and / or audio files. Thereafter, on the selected day, in an exemplary implementation described herein, the user creates a list of in-service control locomotives using information available in the dispatch system. Using the random generation process, the user selects between at least one in-service control locomotive and up to 10% of the in-service lead locomotives. The user then must go to the location of each selected locomotive to manually download the video and / or audio files by connecting a laptop or other suitable mobile electronic device to the digital recorder. All video and / or audio files downloaded by the user from the selected locomotive are saved in a central location, such as a back-end or cloud-based storage device, and the video and / or audio files are analyzed for threat detection by viewing the video clips using a specialized player provided by the manufacturer of the digital video recorder. A second manual method for random downloading and threat detection, which can be performed remotely, includes using a remote monitoring service to collect video and / or audio files, event recorder data, and GPS data. The user downloads the video and / or audio files, event recorder data, and GPS data from each selected locomotive to a central location and analyzes the video and / or audio files for threat detection by viewing the video clips using a specialized player provided by the digital video recorder manufacturer.

[0252] Similarly, for example, a first manual method for system verification includes manually downloading video and / or audio files from each selected locomotive during daily inspection to confirm that the device is operational and meets compliance requirements, such as that the field of view of the inward-facing camera covers the operator's facial expressions and console. A second manual method for system verification, which can be performed remotely, includes remotely acquiring video and / or audio files from connected cameras and event recorder data to perform system verification. The user must then review the event recorder data and video and / or audio files using specialized software to confirm system functionality.

[0253] As described above, the operational threat detection system of the present disclosure improves the second manual method for random downloading, threat detection and system verification by automating the process of selecting the leading locomotive in service, selecting remote downloads for the day, threat detection and health monitoring using advanced artificial intelligence techniques, real-time locomotive event recorder data, integrated GPS data, gyroscope data and / or Wi-Power III data. For example, the video random generator and threat detector components of the operational threat detection system implement automated random downloading and threat detection. At least one week before the beginning of the month, designated employees of the railway receive an email message in which a date for the next month is randomly selected for downloading. On the day of the random download, a selection is made between the controlling locomotive in the two services and the leading locomotive in the service with up to 10% of the highest probability of success and useful video and / or audio data. For the selection process, a list of locomotives for leading service is created by using event recorder data, data available from B2B integration with various railway systems and / or video processing of non-inward camera data using artificial intelligence techniques. Thereafter, the list of locomotives for lead service is narrowed down by additional criteria, such as locomotives associated with a train, locomotives in lead service, critical events detected on the day of download, locomotives operating on the selected track, locomotives with digital video recorders in good health, and / or cellular and / or WiFi connections available for remote download.

[0254] Once the lead in-service locomotive is selected, in this exemplary embodiment, the operational threat detection system requests three ten-minute video and / or audio recording segments from three different start times on the selected download day. The system can increase the probability of a successful download by checking additional conditions, such as the locomotive moving at a speed of more than 2 km / h in the last five minutes and / or the locomotive is not in the railroad yard. The system then notifies the designated railroad employee via an automated email message when the video and / or audio files become available in the system's cloud storage. The designated railroad employee can also view the progress of the random selection process when logged into the system's secure portal.

[0255] The operational threat detection system may further automatically scan downloaded video and / or audio files for threat detection based on real-time event recorder data, gyroscope data, Wi-Power-III data, geolocation information, and / or artificial intelligence tools. Some examples of artificial intelligence techniques include (but are not limited to): mobile phone usage detection; inward camera obstruction detection; outward camera obstruction detection; personal entertainment device usage, such as tablets, personal electronic devices, etc.; sleeping posture and / or fatigue detection; near miss detection; and / or intruder detection.

[0256] Similarly, the operational threat detection system may further automatically scan downloaded video and / or audio files for critical event detection. Some examples of critical event detection techniques using other data sets include, but are not limited to, high G events on the y-axis and / or z-axis; high gyroscope rates; emergency braking and / or rapid stopping; high pitch and / or roll of the mobile asset when it stops (usually in advance); loss of power while the mobile asset is moving and / or after the mobile asset begins to move; anomalies in horn and / or ringtone sequences; horn and / or ringtone detection in quiet zones; passenger emergencies; and / or silent alarm events.

[0257] Designated staff can view random downloads of video and / or audio files, progress of threat detection and / or critical event detection by logging into the system’s secure online portal. Access to video and / or audio data on the secure portal is limited to authorized users using enhanced privacy controls built into the operational threat detection system. Designated staff or a few designated staff can also subscribe to automated email notifications. Authorized rail personnel can view video and / or audio files, as well as event recorder data, gyroscope data and / or location data using a web-based, mobile-optimized player at any time and from any location in the world.

[0258] In this exemplary embodiment, the operational threat detection system of the present disclosure also includes a self-monitoring digital video recorder component that continuously monitors the health of its subcomponents and performs checks to verify compliance with regulations. Automated notifications are sent to authorized and subscribed employees in the event of any failure. Examples of semi-automated processes include determining the health of subcomponents required for successful recording, device state change events (such as hard drive removal), and using audio processing to confirm the operation of inward and outward microphones. The semi-automated process then automatically initiates test recordings from all connected cameras during periodic inspections to allow manual confirmation of camera field of view and manual confirmation of audio quality verification.

[0259] Automated time synchronization verification is also performed during the inspection using video and / or audio data from the past 24 hours to confirm GPS, event recorder and / or inward and outward video and / or audio synchronization. For example, the test process may include the following steps: using artificial intelligence technology to determine the time of change of control state (e.g., throttle, horn, bell, reverser, etc.) in the inward video, where the time of change of control state matches the event recorder data to confirm that the inward video and event recorder data are synchronized; using artificial intelligence technology to determine the time of change of control state (e.g., horn, bell, etc.) in the inward and outward microphone audio, where the time of change of control state matches the event recorder data to confirm that the inward and outward microphone audio and event recorder data are synchronized; determining the time when the mobile asset stops in the outward video, where the time of the mobile asset's stop is matched with the GPS speed and wheel speed obtained from the event data recorder to confirm that the GPS data, event recorder data and outward video are synchronized; and automatically time synchronizing the digital video recorder with the integrated GPS when the time deviates from a configurable predefined limit (e.g., + / - 3 seconds). The semi-automatic process then automatically initiates a self-check at a predefined frequency (e.g., Transport Canada - annually, FRA - 92 days). The semi-automatic process generates a time synchronized test report along with a link to the test record. Authorized rail personnel can then view or share the report by simply logging into the secure online portion.

[0260] The operational threat detection system includes an operational threat detector method for automatically detecting critical events that may be precursors to safety-critical accidents. Examples of critical events include (but are not limited to): operating employees using mobile electronic devices while on duty, in addition to company policy regulations; operating employees using personal entertainment devices or personal electronic devices (such as gaming devices, smart watches, Apple watches, etc.) while on duty, in addition to company policy regulations; operating employees reading materials that are not required for the performance of their duties, such as books, novels, newspapers, etc. while on duty; operating employees in a sleeping position while on duty, such as closing their eyes, putting their feet on the control console, tilting their heads to the left or right, nodding, etc.; the presence of unauthorized personnel in the control of rail vehicles (for example, when the rail vehicle is in leading service, there are more than four employees in the cab, etc.), lack of proper safety clothing (for example, lack of the operator's cab is occupied; operating employees are within hearing range of each other but do not clearly and audibly communicate verbally the information they must communicate verbally, such as announcing signals and making calls every five milestones; the engineer's seat is vacant for more than a few minutes while the train is moving; operating employees smoke in the cab; assault detection, such as the use of certain words in speech; detection of the presence of an intruder from video feeds from outward-facing cameras; detection of malfunctions of intersection warning devices, such as the failure of door arms and flashing lights to flash when the locomotive arrives at the intersection; and odor detection, such as VOC detection, fire and / or smoke detection, etc.

[0261] For example, these critical events are detected using second-by-second data collected from different systems using DARS 100, 200, 1000 and available video and / or audio pre-downloaded from rail vehicles using data-based processes and / or artificial intelligence techniques. The subsystems are distributed on the edge or other Internet browsers, and in cloud instances, such as (1) edge systems, such as GPS, Applicant's Wi-Power-III smart power supply, accelerometers, event recorders, cameras and / or microphones; and (2) cloud systems, such as train and weather data collected using B2B integration.

[0262] The operational threat detection system also includes a general framework that allows new detection processes to be added without significant software updates (e.g., using plug-and-play components). Components include new data processes, video and / or audio-based artificial intelligence models, and other data sets available on mobile assets in the future. The framework also includes methods for automatically collecting video and / or audio and mobile asset data (e.g., data obtained from DARS100, 200, 1000), an engine for event detection, a subscription mechanism for intelligent event notifications, and secure viewing and sharing of critical events with authorized users using a web-based mobile-optimized viewer.

[0263] As described above with respect to the manual methods of random downloading and threat detection, the railroad may use a manual process for threat detection in the absence of automation. For example, a first manual method for random video and / or audio downloading includes using a random download generator to select a day of the month to manually download the video and / or audio file. Thereafter, on the selected day, the user creates a list of in-service controlling rail vehicles using information available in the dispatch system. Using the random generation process, the user selects between at least one in-service controlling locomotive or rail vehicle and up to 10% of the in-service leading locomotive or rail vehicle. The user must then go to each selected locomotive location to manually download the video and / or audio file by connecting a laptop or other suitable mobile electronic device to a digital video recorder. A second manual method for random video and / or audio downloading includes using information available in the dispatch system to create a list of crew members and corresponding controlling rail vehicles in service on the selected day within 72 hours after the employee completes his shift, using a random download generator to select the employee for operational testing within a specified time frame. The user must then go to each rail car used by each employee travel and manually download the video and / or audio files by connecting a laptop or other suitable mobile electronic device to the digital video recorder. The user saves all video and / or audio files downloaded from the selected locomotive in a central location, such as a back-office or cloud-based storage device, and analyzes the video and / or audio files for threat detection by viewing the video clips using a specialized player provided by the digital video recorder manufacturer.

[0264] Supervisors may ride with the crew on the locomotive to perform performance analysis, however, the automated operational threat detection system eliminates most, if not all, manual steps. For example, the automated operational threat detection component includes a process for random video and / or audio downloads, which includes initiating a threat detection process upon detection of a trigger, the trigger including, but not limited to, random selection to meet regulatory compliance requirements, such as random selection of vehicles and / or random selection of employees; on-demand crew performance reviews, such as at the beginning or end of a train trip, during severe weather conditions, throughout a train trip, before and after a time frame based on data events and / or triggers, and during a selected time frame; and continuous threat detection of previously downloaded video and / or audio files. Examples of triggers are described in more detail below.

[0265] The process then remotely completes the download of video and / or audio files from all connected cameras via a cellular and / or WiFi connection, and begins scanning the downloaded video and / or audio files of all connected cameras using an AI-based and data-based process for threat and / or critical event detection. The AI ​​technologies used, which may be configured based on the customer's request, may include (but are not limited to): unauthorized mobile phone use; personal entertainment devices and / or personal electronic devices, such as tablets, Apple Watches, smart watches, etc.; inward camera obstruction detection; sleeping posture and / or fatigue detection; operating employees within hearing range of each other but not clearly and audibly communicating verbally the information they must communicate verbally; the presence of unauthorized personnel in the control rail vehicle; the engineer's seat being vacant for more than a few minutes; operating employees reading materials that are not required for the performance of their duties while on duty; intruder detection; signal non-compliance; smoking in the cab; assault detection; and intersection warning device failure.

[0266] The process then correlates one or more critical events within or around (+ / - X minutes, configurable) the video and / or audio file download timeframe. The data-based process for detecting critical events is based on real-time rail vehicle event recorder data, gyroscope parameters, Wi-Power-III data, and / or geo-location information. Example threat and / or critical event detection techniques (which may also be configurable based on customer request) include (but are not limited to) high G event detection on the y-axis and / or z-axis; high gyroscope rate detection; any emergency braking events, such as siren penalties, speeding penalties, engineer initiated emergency, conductor initiated emergency, engineer initiated impact emergency, and conductor initiated impact emergency; emergency stop events; loss of power while the mobile asset is moving and / or after the mobile asset begins to move; horn and / or bell sequence anomalies; horn and / or bell detection in quiet zones; passenger emergencies; silent alarm events; doors opened en route and / or doors stuck open; slack tightening; train entering a station; throttle modulation; running release; stretch braking; power braking; severe cutbacks; hard coupling; passing a station; short shallow air brakes and / or shallow air brakes; inclement weather; high temperature detection on track; and / or possible safety issues.

[0267] Threat detection results include three types of events: (1) alerts after an event is detected; (2) episodes - alerts spanning a specific time frame; and (3) predictive alerts. An example of a predictive alert is when someone wearing body armor enters the cab and / or passenger area, which may be a precursor to an accident. The operational threat detection system can send automated alerts and enable real-time streaming of video and / or audio data at the control center for real-time monitoring and situational awareness. Authorized employees can subscribe to threat detection results through the following methods: subscribing to email message alerts, subscribing to mobile phone SMS alerts, subscribing to email message reports at a predefined configurable frequency (e.g., daily, weekly, etc.), and subscribing through a dashboard in a secure online portal. An authorized employee or several authorized employees can view threat detection events on the applicant's cloud and can use a unified web-based mobile optimized viewer (e.g., the web client 142, 242, 826 or the virtual reality device 828 described above) to analyze video and / or audio files and vehicle data before and after the time frame of the event. Authorized employees can share threat detection results within or outside the railroad using secure web-based sharing, such as the real-time data collection and recorded data sharing system process 990 described above. Authorized employees can also share vehicle data, video files, and / or audio files around the time frame of the threat using a secure web-based viewer (such as the web client 142, 242, 826 or virtual reality device 828 described above). The recipient of the sharing can view threat detection reports and data sets around the time frame in a secure online web portal using multi-factor authentication. The operational threat detection system records data access events to ensure that crew privacy is maintained.

[0268] Some examples of triggers that can be detected by the operational threat detection component include, but are not limited to, randomly selecting to meet regulatory compliance requirements by executing the automated process in the present disclosure, which the railway will use to automate parts of manual and time-consuming processes, detect non-compliance in operations, maintain compliance with regulations, and conduct on-demand fleet performance analysis. As described above, regulations may include foreign and / or domestic regulations, such as Canadian Department of Transport regulations (e.g., Rail Vehicle Voice and Video Recorder Regulations: SOR / 2020-178) and US FRA regulations (e.g., Rail Vehicle Image and Audio Recording Devices for Passenger Trains: File No. FRA-2016-0036). Canadian Department of Transport regulations require railways to install inward-facing cameras and microphones in control vehicles in passenger or freight service on Canadian tracks. Canadian Department of Transport regulations (SOR / 2020-178§30: Threats to the safety of railway operations) define that railways can use downloaded inward-facing video and / or audio files for threat detection (e.g., unauthorized mobile phone use, etc.) to confirm compliance with security measures and build effective security policies. In order to download video and / or audio files from an inward-facing camera, an authorized person must perform the download using a random selection process to select the lead railcar and a day of the month. The proposed FRA regulations would require railroads to install inward-facing and outward-facing cameras on control railcars in passenger service in the United States. The FRA regulations (FRA-2016-0036 §217.9: Operational testing and inspection program; recordkeeping) provide that railroads that utilize inward-facing railcar images and / or cab audio recordings for operational testing and inspection shall ensure that employees are randomly subjected to such testing involving image and / or audio recordings.

[0269] An example of an operational test is to confirm compliance with PED use. PED use is permitted when the following conditions are met: the train is stopped; no one is engaged in repairing, refueling, and / or otherwise preparing the train, locomotive, or railcar for movement; the PED is utilized by only one crew member at a time; if on the ground, the employee is at least 25 feet from the nearest rail; and employees assigned to yard work are not permitted to carry PEDs on their person unless on break or relieved from duty.

[0270] Another example of operational testing is personal protective equipment (PPE) compliance, which requires the wearing of reflective material clothing when required, hard hats or bump caps when required, and safety glasses or protective glasses. The operational threat detection component uses a random download process to minimize the time and effort spent analyzing downloaded video and / or audio files, thereby maximizing the value of the investment in digital video recorders.

[0271] On-demand crew performance analysis is used to confirm compliance with railroad operating rules in various processes, such as efficiency and / or proficiency testing, engineer recertification (49 C.FR §240.127), minor noncompliance detection, and on-demand reviews. The purpose of efficiency / proficiency testing is to manage risk by evaluating and / or verifying compliance with rules and procedures. The assignment of review work is based on ranking operational risks that are mitigated by properly applying rules and thoroughly reviewing operations. Train operating managers perform this test on all locomotive engineers, train masters, and locomotive crews assigned to them through train crew and / or surprise audits. Some of these audits can now be performed using inward-facing camera video and / or audio files available from digital video recorders on each locomotive. Engineer recertification requires railroads to assess the skill performance of train operators (also known as engineers). The purpose of the regulation (49 C.FR §240.127) is to "ensure that only qualified personnel operate locomotives or trains." Engineer recertification may be accomplished using, for example, the applicant's engineer recertification assistant, as described in U.S. Provisional Application No. 63 / 061,548, filed on August 5, 2020, U.S. Non-Provisional Application No. 17 / 394,135, filed on August 4, 2021, and PCT Application No. PCT / US2021 / 44733, filed on August 5, 2021, the entire disclosure of each of which is incorporated herein by reference in its entirety. For minor non-compliance detection, the operational threat detection system includes multiple data-based scans to detect potential non-compliance. Some examples of minor non-compliance detection include abnormal horn and / or bell sequences, horn and / or bell detection in quiet zones, severe cuts and / or passing stations. The automated operational threat detection component executes upon detection of any data-based event configured in the operational threat detection system. A list of configurable alarms and threat detection processes is configurable for a rail asset model. For an on-demand review, a supervisor may, for example, conduct an on-demand performance review of an employee before assigning him or her to ride on a train with executives or on a train full of hazardous materials.

[0272] The operational threat detection system of the present disclosure includes various methods described below for randomly selecting vehicles, randomly selecting employees, and conducting crew performance reviews on demand. One method for randomly selecting vehicles includes a designated employee of the railroad receiving an email message containing the day of the next month for random selection, which day can be configured according to the requirements of the railroad. The random selection of vehicles includes creating a list of rail vehicles for use in leading service by using event recorder data and / or using train data available for B2B integration with various rail systems. The system narrows down the list of candidate rail vehicles by using additional criteria, the list including between at least one locomotive and up to 10% of the controlling locomotives in service, such as (for example), the rail vehicle is associated with the train and moved at a configurable predefined speed (e.g., 2 km / h) within the last configurable predefined time (e.g., 5 minutes), the rail vehicle is operating in a specified location according to regulatory requirements (e.g., the rail vehicle is not in a marshaling yard, the rail vehicle is operating on Canadian tracks, etc.), the digital video recorder is in good health, and / or a cellular or WiFi connection is available for remote download. The operational threat detection system requests video and / or audio files of configurable predefined length (e.g., 30 minutes of video and / or audio in three ten-minute segments) from three configurable predefined different start times (expressed in UTC) on the same selected day of the month. When the randomly downloaded video and / or audio files are available in the cloud, the system notifies the designated railroad employee via an automated email message, and the designated railroad employee can view the progress of the random selection process on a secure online web portal.

[0273] A method for randomly selecting employees includes a designated employee of a railway receiving an email message containing a day for the next month for random selection, which day can be configured according to the railway requirements. The random selection of employees includes creating a list of employees controlling rail vehicles in operation service, randomly selecting employees for performance analysis, and generating a list of controlled locomotives operated by each of the selected employees. Employee and train information is received via B2B integration with various railway systems. A second method of employee identification is based on artificial intelligence technology, such as using facial recognition and employee images added at account creation to identify employees. The system narrows down the list of candidate rail vehicles by using additional criteria, the list includes between at least one locomotive and up to 10% of the controlling locomotives in service, such as the rail vehicle is associated with a train and moves at a configurable predefined speed (e.g., 2 km / h) within the last configurable predefined time (e.g., 5 minutes), the rail vehicle is operated in a specified location according to regulatory requirements (e.g., the rail vehicle is not in a marshaling yard, the rail vehicle is operated on US tracks, etc.), the digital video recorder is in good health, and / or a cellular or WiFi connection is available for remote download. The operational threat detection system requests video and / or audio files of configurable predefined length (e.g., 30 minutes of video and / or audio in three ten-minute segments) from three configurable predefined different start times (expressed in UTC) on the same selected day of the month. When the randomly downloaded video and / or audio files are available in the cloud, the system notifies the designated railroad employee via an automated email message, and the designated railroad employee can view the progress of the random selection process on a secure online web portal.

[0274] A method for on-demand crew performance review includes an authorized employee from a railroad initiating an on-demand performance review and selecting an employee name and ID, and then selecting one or more search criteria for video and / or audio selection, such as the start or end of a train trip, in severe weather conditions, throughout a train trip, around a time frame based on data events and / or triggers; train travel during a selected time frame; and train travel in a selected area or location. An operational threat detection system creates a list of candidate rail vehicles operated by the selected employee based on the selected search criteria. To increase the probability of a successful download, the list of candidate rail vehicles is narrowed by adding additional criteria, such as the rail vehicle is associated with a train and has moved at a speed exceeding a configurable predefined speed (e.g., 2 km / h) within the last configurable predefined time (e.g., 5 minutes), the digital recorder is in good health, and / or a cellular or WiFi connection is available for remote download. The operational threat detection system requests video and / or audio files of configurable predefined length (e.g., 30 minutes of video and / or audio in three ten-minute segments) from three configurable predefined different start times (expressed in UTC) on the same selected day of the month. When the randomly downloaded video and / or audio files are available in the cloud, the system notifies the designated railroad employee via an automated email message, and the designated railroad employee can view the progress of the video and / or audio file transfer on a secure online web portal.

[0275] Fig.38 1700 is a flow chart showing a process 1700 for remotely and randomly downloading video files, audio files, second-by-second data from a DARS 100, 200, 1000, and / or downloading data from a third-party API associated with a selected mobile asset (e.g., a train, locomotive, and / or railcar) according to an illustrative exemplary embodiment of the present disclosure. The operational threat detection system 1600 determines whether the audit type is a random locomotive selection, a random employee selection, or an on-demand crew performance analysis 1702. If the audit type is a random locomotive selection or a random employee selection, the process 1700 determines whether the audit date was randomly selected in the previous month or preconfigured for the railroad 1704, and then sends an email notification indicating the selected random day in the next month to an authorized employee 1706. At the start of the selected audit date, the process 1700 determines whether the audit type is a random locomotive selection or a random employee selection 1708.

[0276] If the audit type is random locomotive selection, then the process 1700 determines a configurable predefined number of locomotives to be selected 1710, which must include between at least one in-service controlling / lead locomotive up to a configurable X% (e.g., 10%) of in-service controlling / lead locomotives with the highest probability of successful random download among all in-service controlling / lead locomotives. Once the in-service controlling locomotives are selected, the process 1700 immediately begins randomly downloading video and / or audio files from all connected cameras from each selected in-service controlling locomotive 1712. The downloaded video and / or audio files are then scanned for threat and / or critical event detection using an artificial intelligence model or process to detect critical events, the artificial intelligence process including (but not limited to) possible mobile phone use detection process, sleep detection process, personal entertainment device and / or personal electronic device (e.g., Apple Watch), use detection process, intersection failure, ... n new processes 1714. The process 1700 then correlates 1716 one or more threats and / or critical events within or around the time frame (+ / - a configurable predefined duration (e.g., 5 minutes)) of the downloaded video and / or audio files. The audit report is then displayed 1718 in the compliance asset dashboard of the operational threat detection system 1600. Once the report is uploaded to the cloud, the process 1700 enables sharing 1720 of the audit results.

[0277] If the audit type is random employee selection, the process 1700 creates a list of employees, such as conductors and engineers operating in the lead locomotive, and then uses a random generator to narrow the list of employees to a configurable X% (e.g., 10%) 1722. The list of in-service lead locomotives is also narrowed to include the lead locomotive with the highest probability of successful random download 1724. Once the in-service controlling locomotive is selected, the process 1700 immediately begins randomly downloading video and / or audio files from all connected cameras from each selected in-service controlling locomotive 1712. The downloaded video and / or audio files are then scanned for threat and / or critical event detection using an artificial intelligence model or process to detect critical events, including (but not limited to) possible mobile phone use detection process, sleep detection process, personal entertainment device and / or personal electronic device (e.g., Apple Watch), use detection process, intersection failure, ... N new processes 1714. The process 1700 then correlates 1716 one or more threats and / or critical events within or around the time frame (+ / - a configurable predefined duration (e.g., 5 minutes)) of the downloaded video and / or audio files. The audit report is then displayed 1718 in the compliance asset dashboard of the operational threat detection system 1600. Once the report is uploaded to the cloud, the process 1700 enables sharing 1720 of the audit results.

[0278] If the audit type is an on-demand crew performance analysis, the process 1700 selects at least one employee ID and / or name for performance evaluation 1726, and then selects performance review criteria, such as the start or end of a train trip in a specific time frame, a failure at an intersection under severe weather conditions in a specific time frame, a complete train trip in a specific time frame, and a time frame around the data-based event 1728. The output of the search criteria will determine the time frame of the audit. After performing an on-demand crew performance review 1730 while driving through a failed intersection or driving in severe weather, the process 1700 downloads the video 1732 in severe weather and around the intersection. Once downloaded, the process 1700 performs threat and / or critical event detection using artificial intelligence models to detect possible mobile phone use, sleep detection, personal entertainment device use (such as Apple Watch), intersection failure, ... N new algorithms to detect various threats and / or events 1714 and associate one or more threats and / or critical events within and / or around (+ / -X minutes) the video and / or audio download time frame 1716, where X is configurable. The audit report is then displayed 1718 in the compliance asset dashboard of the operational threat detection system 1600. Once the report is uploaded to the cloud, the process 1700 enables sharing 1720 of the audit results.

[0279] After performing an on-demand crew performance review 1730 while driving through a faulty intersection or driving in inclement weather, the process 1700 also creates a list of candidate rail vehicles operated by the selected employee or a few selected employees 1734, and then the list of in-service leading locomotives is narrowed down to include the leading locomotive with the highest probability of successful random download 1724. Once the in-service controlling locomotive is selected, the process 1700 immediately begins downloading video and / or audio files from all connected cameras from each selected in-service controlling locomotive 1712. Once downloaded, the process 1700 performs threat and / or critical event detection using artificial intelligence models to detect possible mobile phone use, sleep detection, personal entertainment device use (such as Apple Watch), intersection failure, ... N new algorithms to detect various threats and / or events 1714 and associate one or more threats and / or critical events within and / or around (+ / -X minutes) the video and / or audio download time frame 1716, where X is configurable. The audit report is then displayed 1718 in the compliance asset dashboard of the operational threat detection system 1600. Once the report is uploaded to the cloud, process 1700 enables sharing 1720 of the audit results.

[0280] Fig.391800 is a flow chart showing a process 1800 for remotely narrowing down a list of leading locomotives according to an illustrative exemplary embodiment of the present disclosure. The operational threat detection system 1600 determines whether event recorder data is available within a selected service time frame (e.g., 06:00 AM - 11:00 AM) 1802. If event recorder data is not available within the selected service time, the process 1800 determines whether train data is available for the selected locomotive 1804. If train data is not available, the process 1800 extracts a thumbnail for the selected service time 1806. The outward thumbnail is analyzed, and the process 1800 determines whether the outward camera is obstructed 1808. If the outward camera is obstructed, the process 1800 terminates. If the outward camera is not obstructed, the outward thumbnail is analyzed again, and the process 1800 determines whether the selected locomotive is in leading service for a configurable predefined duration (e.g., greater than X minutes, such as greater than 30 minutes) 1810. If the selected locomotive is not in the leading service for the configurable predefined duration, the process 1800 terminates. If the selected locomotive is in the leading service for the configurable predefined duration, the selected locomotive is added to the locomotive list in leading service 1812 and the process 1800 screens the locomotive list in leading service for the locomotive with the highest random download probability 1814. The screening process 1900 is described in detail below. If train data is available for the selected locomotive, the process 1800 determines whether the selected locomotive is in the leading service for a configurable predefined duration (e.g., greater than X minutes, such as greater than 30 minutes) 1816. If the selected locomotive is in the leading service for the configurable predefined duration, the selected locomotive is added to the locomotive list in leading service 1812 and the process 1800 screens the locomotive list in leading service for the locomotive with the highest random download probability 1814. The screening process 1900 is described in detail below.

[0281] If event recorder data is available within the selected service time, the process 1800 determines whether the vehicle type is a vehicle with an engine or a vehicle without an engine 1818. If the vehicle is a vehicle without an engine, the process 1800 determines whether the local reverser is forward or reverse 1820. If the local reverser is forward or reverse, the process 1800 determines whether the selected locomotive has been in leading service for a configurable predefined duration (e.g., at least X minutes, such as at least 30 minutes) 1822. If the selected locomotive has been in leading service for a configurable predefined duration, the selected locomotive is added to a list of locomotives in leading service 1812 and the process 1800 screens the list of locomotives in leading service to find the locomotive with the highest probability of random download 1814. The screening process 1900 is described in detail below.

[0282] If the vehicle is a vehicle with an engine, then the process 1800 determines whether the reverser is forward or reverse 1824. If the reverser is not forward or reverse, then the process 1800 determines whether the air brake is in a leading passenger cut-in state or a freight passenger cut-in state 1826. If the air brake is not in a leading passenger cut-in state or a freight passenger cut-in state, then the process 1800 determines whether the selected locomotive is in a distributed power mode in leading service 1828. If the selected locomotive is not in a distributed power mode in leading service, then the process 1800 terminates. If the reverser is forward or reverse, then the process 1800 determines whether the air brake is in a leading passenger cut-in state or a freight passenger cut-in state 1826. If the air brake is not in a leading passenger cut-in state or a freight passenger cut-in state, then the process 1800 determines whether the selected locomotive is in a distributed power mode in leading service 1828. If the selected locomotive is not in a distributed power mode in leading service, then the process 1800 terminates. If the air brake is in the leading passenger cut-in state or the freight passenger cut-in state, the process 1800 determines whether the selected locomotive is in the distributed power mode in the leading service 1828. If the selected locomotive is not in the distributed power mode in the leading service, the process 1800 terminates. If the selected locomotive is in the distributed power mode in the leading service, the process 1800 determines whether the selected locomotive has been in the leading service for a configurable predefined duration (e.g., at least X minutes, such as at least 30 minutes) 1822. If the selected locomotive is in the leading service for the configurable predefined duration, the selected locomotive is added to the list of locomotives in the leading service 1812 and the process 1800 screens the list of locomotives in the leading service for the locomotive with the highest random download probability 1814. The screening process 1900 is described in detail below.

[0283] Fig.401 is a flow chart showing a process 1900 for remotely screening a list of locomotives in lead service according to an illustrative exemplary embodiment of the present disclosure. The operational threat detection system 1600 determines whether the DARS 100, 200, 1000 or other data acquisition and recording system of the selected locomotive is in contact with the applicant's back office 1902. If the DARS 100, 200, 1000 of the selected locomotive is not in contact with the applicant's back office, the selected locomotive is removed from the list of locomotives in lead service for randomization 1904, and the process 1900 terminates. If the DARS 100, 200, 100 of the selected locomotive is in contact with the applicant's back office, the process 1900 determines whether the selected locomotive has a working locomotive voice and video recorder system (LVVR system) 1906. If the selected locomotive does not have a working LVVR system, the selected locomotive is removed from the list of locomotives in lead service for randomization 1904, and the process 1900 terminates. If the selected locomotive does have a working LVVR system, the process 1900 determines whether the selected locomotive is at a maintenance center 1908. If the selected locomotive is at a maintenance center, the selected locomotive is removed from the list of locomotives in leading service for randomization 1904, and the process 1900 terminates. If the selected locomotive is not at a maintenance center, the process 1900 determines whether the selected locomotive is at a station 1910. If the selected locomotive is at a station, the selected locomotive is removed from the list of locomotives in leading service for randomization 1904, and the process 1900 terminates. If the selected locomotive is not at a station, the DARS 100, 200, 1000 of the selected locomotive is added to the most likely vehicle list for the random generator 1912. The most likely vehicle list for the random generator is screened to include X% (e.g., 10%) of the locomotives in the list 1914. The process 1900 then requests that the random video and / or audio files be transferred to the cloud 1916. The process 2000 of transferring random video and / or audio files to the cloud is described in detail below.

[0284] Fig.411 is a flow chart showing a process 2000 for transferring random video and / or audio files of a selected locomotive to the cloud according to an illustrative exemplary implementation of the present disclosure. The operational threat detection system 1600 randomly selects three different service time windows to randomly download video and / or audio files of a selected locomotive 2002. In this exemplary implementation, the process 2000 sends a request to the DARS 100, 200, 1000 for three ten-minute video files to be transferred for each selected time 2004. The DARS 100, 200, 1000 then creates the video and / or audio files and transfers the three files to the backend 2006. The video and / or audio files of the selected locomotive are then processed for threat detection 2008. The process 2100 for detecting threats and / or critical events in the downloaded video and / or audio files of the selected locomotive is described in detail below.

[0285] Fig.42 21 is a flow chart showing a process 2100 for detecting threats and / or critical events in downloaded video and / or audio files of a selected locomotive according to an illustrative exemplary implementation of the present disclosure. The threat detection system 1600 is operated to create a list of threats and / or events applicable to the selected mobile asset 2102, which may include, for example, mobile phone detection, sleep detection, camera obstruction, intersection failure, etc. A random video and / or audio file is selected from the mobile asset for applicable threat detection and / or event detection processing 2104. The selected random video and / or audio file from the selected camera is sent for threat detection processing 2106. The process 2100 then determines whether the type of threat and / or event to be detected is mobile phone detection, camera obstruction, intersection failure, and / or sleep detection 2108. If the type of threat to be detected is mobile phone detection, then the selected random video and / or audio file is processed for reference below Fig.43 The detailed process 2200 performs mobile phone usage detection. If the threat type to be detected is camera obstacle detection, then a random video and / or audio file is selected and processed for reference below. Fig.44 The detailed process 2300 performs camera obstacle detection. If the threat type to be detected is an intersection fault, then a random video and / or audio file is selected and processed for reference below. Fig.45 The process 2400 described in detail performs sleep detection. Once a random video and / or audio file is selected to be processed for threat detection, the process 2100 determines whether any threats are detected 2110. If no threats are detected, the process 2100 terminates. If any threats are detected, the process 2100 displays the threat detection results (including predictions and bounding boxes) 2112 on a secure online web portal and then begins processing the next random video and / or audio file from the same vehicle or the next vehicle.

[0286] Fig.43 2200 is a flow chart showing a process 2200 for detecting mobile phone usage in downloaded video and / or audio files of a selected locomotive according to an illustrative exemplary implementation of the present disclosure. Operational threat detection system 1600 converts each 10 seconds of the downloaded video and / or audio file into 10 frames and loops through each frame 2202. An object detection machine learning model (such as the object detection machine learning model described above with respect to DARS 1000) is applied to each frame 2204. Process 2200 determines whether a prediction is made that indicates that a mobile phone is present in the frame for more than two frames 2206. If the prediction indicates that a mobile phone is present in more than two frames, then process 2200 returns a determination 2208 of mobile phone usage detection to process 2100. If the prediction indicates that a mobile phone is not present in more than two frames, then process 2200 determines whether there are more frames left for process 2210. If there are not any more frames left for the process, then process 2200 returns a determination 2212 that mobile phone usage was not detected to process 2100. If there are more frames left to process, then process 2200 loops back to step 2204 and applies the object detection machine learning model to the next frame.

[0287] Fig.442300 is a flow chart showing a process 2300 for detecting camera obstructions in a downloaded video and / or audio file of a selected locomotive according to an illustrative exemplary implementation of the present disclosure. The operational threat detection system 1600 receives thumbnails and / or images from the downloaded video and / or audio files to determine camera obstruction detection on the inward camera 2302. The process 2300 classifies the thumbnails and / or images into three different classes, such as cab vacant or occupied, camera view obstructed, and / or camera hardware problem 2304. If the thumbnail and / or image is classified as cab vacant or occupied, the process 2300 uses a cab vacant / occupied classification model on the received thumbnail and / or image to predict whether the cab is vacant or the cab is occupied 2306. The process 2300 then generates a prediction output 2308 by referencing the predictions from each class and then returns the prediction output to the process 2200, the prediction output including a cab vacant or occupied prediction percentage, a camera view obstructed prediction percentage, and / or a camera hardware problem prediction percentage 2310. If the thumbnail and / or image is classified as a camera hardware problem, the process 2300 uses the camera hardware problem classification model on the received thumbnail and / or image to predict whether the camera has water 2312. The process 2300 then generates a prediction output 2308 by referencing the predictions from each class and then returns the prediction output to the process 2200, the prediction output including the cab vacant or occupied prediction percentage, the camera view obstruction prediction percentage, and / or the camera hardware problem prediction percentage 2310. If the thumbnail and / or image is classified as a camera view obstruction, the process 2300 uses the camera view obstruction classification model on the received thumbnail and / or image to predict whether the camera view is obstructed 2314. The process 2300 then generates a prediction output 2308 by referencing the predictions from each class and then returns the prediction output to the process 2200, the prediction output including the cab vacant or occupied prediction percentage, the camera view obstruction prediction percentage, and / or the camera hardware problem prediction percentage 2310.

[0288] Fig.45 2400 is a flow chart showing a process 2400 for detecting intersection faults in downloaded video and / or audio files of a selected locomotive according to an illustrative exemplary implementation of the present disclosure. The operational threat detection system 1600 receives video and / or audio files from a front-facing camera to perform an intersection status check, which in this illustrative exemplary implementation includes checking 25 seconds before the mobile asset arrives at the intersection and checking 5 seconds after the mobile asset arrives at the intersection, and extracts images from the video file for artificial intelligence processing 2402. The process 2400 then detects intersection objects in each image, such as individual gate arms, mast light flashers, boom flashers, masts, lights on gate arms, etc. 2404.

[0289] If the intersection object in the image is an individual gate arm, the process 2400 detects the status 2406 of the individual gate arm (e.g., shut down, damaged, stuck at 45 degrees, unknown, and / or other), detects the position 2412 of the individual gate arm (e.g., two four-way intersections - left near, right near, left far, and right far, etc.), detects the status 2420 of the individual gate arm lights (e.g., flashing, steady, dark, and / or unknown, etc.), calculates the overall individual gate arm status 2424 (e.g., the gate arm status that fails when any of the gate arms is damaged or stuck, etc.), and provides protection outputs 2428 for different intersection objects (e.g., gate arm - operable, mast flasher - operable, boom flasher - inoperable, and other objects - inoperable, etc.).

[0290] If the intersection object in the image is a masthead flasher, the process 2400 detects individual masthead flashers and calculates the masthead flasher count on the intersection 2410, calculates the status of the flasher (e.g., on, dark, flashing, unknown, etc.) and the individual flasher frequency 2418, confirms the overall flasher alignment (e.g., flasher broken, incorrect horizontal alignment of the flasher, etc.) 2422, calculates the overall flasher status (e.g., flasher damaged, dark or not flashing alternately, etc.) 2426 and provides protection outputs for different intersection objects 2428 (e.g., gate arm - operable, mast flasher - operable, boom flasher - inoperable and other objects - inoperable, etc.).

[0291] Process 2400 also checks for other types of intersection fault events 2414 (e.g., complete loss of power, etc.), checks boom flasher status 2416, and then provides protection outputs 2428 for different intersection objects (e.g., gate arm - operable, mast flasher - operable, boom flasher - inoperable, and other objects - inoperable, etc.).

[0292] Fig.4625 is a flow chart showing a process 2500 for associating critical events in a downloaded video and / or audio file for a selected locomotive according to an illustrative exemplary implementation of the present disclosure. The operational threat detection system 1600 creates a list of critical events applicable to the selected locomotive 2502, which may include, for example, emergency stops, engineer-initiated emergency situations, power braking, loss of power while moving, etc. The process 2500 then processes the downloaded video and / or audio file to detect specific critical events around the time of the random download (+ / - a configurable predefined time length (e.g., + / - 10 minutes) 2504. If a specific critical event is not detected, the process 2500 loops back to step 2504 and processes the next frame of the downloaded video and / or audio file. Once the frames of the downloaded video and / or audio file have been processed to detect all critical events, the detected critical event is associated 2506 with the specific random video and / or audio file download. The detected critical event is displayed on the randomizer page 2508 and the process 2500 proceeds to the next random video and / or audio file, or if there are no more random video and / or audio files for the previous selected locomotive then proceeds to the next selected locomotive and loops back to step 2504. The process 2500 continues until all random video and / or audio files downloaded from each selected locomotive are critical event associated processes.

[0293] To simplify explanation, processes 1700 to 2500 are depicted and described as a series of steps. However, the steps according to the present disclosure may occur in various orders and / or concurrently. In addition, the steps according to the present disclos...

Claims

1. A method for identifying at least one of a configurable predefined threat, a configurable predefined event, and non-compliance with applicable mobile asset regulations involving at least one mobile asset, comprising: receiving data using a data acquisition and recording system onboard the mobile asset based on at least one data signal from at least one data source onboard the mobile asset and at least one data source remote from the mobile asset, the data being collected during a configurable predetermined time frame; identifying at least one of the threat, the critical event, and noncompliance with applicable mobile asset regulations using an artificial intelligence model; associating at least one of the threat, the critical event, and noncompliance with applicable mobile asset regulations within and around at least one of a configurable predetermined amount of time before the time frame, a configurable predetermined amount of time after the time frame, and a configurable predetermined amount of time during the time frame; and Audit results based on the data are displayed on a dashboard of the mobile assets using a mobile device optimized and driven by conversational artificial intelligence.

2. The method according to claim 1, further comprising: The audit results are shared on the condition that authorized employees request to share the audit results.

3. The method according to claim 1, further comprising: identifying, based on at least one search criterion, an audit type comprising at least one of random mobile asset selection, random employee selection, and on-demand crew performance analysis; Under the condition that the audit type is random mobile asset selection, identifying a set of configurable predetermined mobile assets; identifying a list of employees associated with the mobile asset, provided that the audit type is random employee selection; Conditional upon the audit type being random employee selection, revising the employee list using a random generator to include a configurable set of predetermined employees associated with the mobile asset; and Employees selected for performance evaluation, provided that the audit type is an on-demand crew performance analysis.

4. The method according to claim 3, further comprising: Select at least one performance review criterion; identifying a set of candidate mobile assets including at least one mobile asset operated by the employee; and The set of candidate mobile assets is revised to include a subset of candidate mobile assets that includes a highest probability of successful random download.

5. The method according to claim 3, further comprising: Select at least one performance review criterion; reviewing on-demand crew performance when said mobile asset is traveling through at least one faulted intersection during inclement weather; and At least one multimedia file is downloaded around the at least one faulty intersection and under inclement weather.

6. The method according to claim 1, further comprising: Identify at least one audit date; and The at least one audit date is sent to at least one authorized employee.

7. The method according to claim 4, further comprising: determining whether an event data recorder associated with the mobile asset is available within the time frame; Under the condition that the event data recorder is available, identifying an asset type associated with the mobile asset, the asset type comprising one of an engine-on and an engine-off; determining whether the mobile asset is in lead service for at least a configurable predetermined amount of time, conditional upon the asset type being no engine and conditional upon the local reverser associated with the mobile asset being one of forward and reverse; and The mobile asset is added to a set of leading serving mobile assets.

8. The method according to claim 4, further comprising: determining whether an event data recorder associated with the mobile asset is available within the time frame; determining, conditional upon a local reverser associated with the mobile asset being one of forward and reverse, that a state of an air brake is associated with the mobile asset, the state comprising one of a passenger cut-in and a freight passenger cut-in; determining whether the mobile asset is in a distributed power mode in a lead service; Under the condition that the asset type is engineless and under the condition that the mobile asset is in a distributed power mode in a lead service, determining whether the mobile asset is in a lead service for at least a configurable predetermined amount of time; and The mobile asset is added to a set of leading serving mobile assets.

9. The method according to claim 4, further comprising: determining whether an event data recorder associated with the mobile asset is available within the time frame; determining whether asset data is available for the mobile asset under the condition that the event data recorder associated with the mobile asset is unavailable within the time frame; determining, subject to the availability of the asset data, whether the mobile asset has been in a lead service for at least a configurable predetermined amount of time; and The mobile asset is added to a set of predecessor serving mobile assets upon condition that the mobile asset is in predecessor service for the configurable predetermined amount of time.

10. The method according to claim 4, further comprising: determining whether an event data recorder associated with the mobile asset is available within the time frame; determining whether asset data is available for the mobile asset under the condition that the event data recorder associated with the mobile asset is unavailable within the time frame; obtaining, under the condition that asset data is unavailable, at least one thumbnail image associated with the mobile asset for the time frame; analyzing the at least one thumbnail image to determine whether an outward-facing camera associated with the mobile asset is obstructed; analyzing the at least one thumbnail image to determine whether the mobile asset was in leading service for at least a configurable predetermined amount of time, provided that the outward-facing camera is not obstructed; and The mobile asset is added to a set of predecessor serving mobile assets upon condition that the mobile asset is in predecessor service for the configurable predetermined amount of time.

11. The method according to claim 4, further comprising: Determine whether the data collection and recording system communicates with the background; Under the condition that the data collection and recording system does not communicate with the background, removing the mobile asset from the candidate mobile asset subset; Under the condition that the data acquisition and recording system communicates with the background, determining whether the mobile asset includes a working locomotive voice and video recorder (LVVR) system; removing the mobile asset from the subset of candidate mobile assets on the condition that the mobile asset does not include the working LVVR system; Under the condition that the mobile asset includes the working LVVR system, determining whether the mobile asset is at a maintenance center; Under the condition that the mobile asset is at the maintenance center, removing the mobile asset from the candidate mobile asset subset; Under the condition that the mobile asset is not in the maintenance center, determining whether the mobile asset is in the station; removing the mobile asset from the subset of candidate mobile assets under the condition that the mobile asset is at the station; adding the mobile asset to the subset of candidate mobile assets including the highest probability of successful random downloading, under the condition that the mobile asset is not at the station; Under the condition that the mobile asset is not at the station, screening the subset of candidate mobile assets including the highest probability of successful random download to include a configurable predetermined number of candidate mobile assets; and At least one multimedia file associated with the mobile asset is requested.

12. The method according to claim 1, further comprising: creating a list of at least one of the at least one configurable predefined threat, the at least one configurable predefined event, and the at least one non-compliance with applicable mobile asset regulations; selecting a random multimedia file associated with the mobile asset; determining whether at least one of the at least one configurable predefined threat, the at least one configurable predefined event, and the at least one non-compliance with applicable mobile asset regulations is detected in the random multimedia file; and At least one of the at least one configurable predefined threat, the at least one configurable predefined event, and the at least one noncompliance with applicable mobile asset regulations detected in the random multimedia file is displayed.

13. The method according to claim 12, further comprising: identifying an event type including at least one of a personal electronic device detection, a camera obstruction, and an intersection fault; Converting the random multimedia file into a set of frames; and Conditional upon the personal electronic device being detected in at least two frames in the set of frames, a prediction of detection of the personal electronic device is identified using the object detection machine learning model.

14. The method according to claim 12, further comprising: identifying at least one class including at least one of a cab vacant, a cab occupied, a camera view obstruction, and a camera hardware problem associated with the random multimedia file; and Produces predicted percentages for each class.

15. The method according to claim 12, further comprising: determining, based on the random multimedia file, from at least one front-facing camera associated with the mobile asset, a state of the intersection within a configurable first predetermined time before the mobile asset arrives at the intersection and within a configurable second predetermined time after the mobile asset arrives at the intersection; Extracting images from the random multimedia files for processing; and At least one intersection object in the random multimedia file is detected.

16. The method according to claim 15, further comprising: detecting individual gate arm states based on the at least one intersection object; detecting the position of said individual door arms; Detect the status of individual door arm lights; Determine overall door arm status; and A prediction output for the individual door arm is provided.

17. The method according to claim 15, further comprising: determining an intersection fault event based on the at least one intersection object; and A prediction output of a failure event at the intersection is provided.

18. The method of claim 15, further comprising: determining a boom flasher status based on the at least one intersection object; and A predicted output of the cantilever flasher is provided.

19. The method of claim 15, further comprising: detecting individual masthead light flashers based on the at least one intersection object; determining a count of individual masthead light flashers on the at least one intersection object; determining the individual masthead light flasher status and frequency of said individual masthead light flasher; Confirm overall flasher alignment; Determine overall flasher status; and A predicted output of the individual masthead light flashers is provided.

20. A system for identifying at least one of a configurable predefined threat, a configurable predefined event, and non-compliance with applicable mobile asset regulations involving at least one mobile asset, comprising: a data acquisition and recording system onboard the at least one mobile asset, the data acquisition and recording system comprising an event detector, a gyroscope, a global positioning system, and a digital video recording adapted to receive data obtained from at least one of onboard the at least one mobile asset and offboard the at least one mobile asset; and A processing component adapted to obtain at least one multimedia file from the at least one mobile asset and process the at least one multimedia file to determine at least one of threat detection, critical event detection, and non-compliance with the applicable mobile asset regulations.

21. The system of claim 20, further adapted to: receiving data using the data acquisition and recording system onboard the mobile asset based on at least one data signal from at least one of at least one data source onboard the mobile asset and at least one data source remote from the mobile asset, the data being collected during a configurable predetermined time frame; identifying at least one of said threat detection, said critical event detection, and said non-compliance with applicable mobile asset regulations using an artificial intelligence model; correlating at least one of the threat detection, the critical event detection, and noncompliance with applicable mobile asset regulations within and around at least one of a configurable predetermined amount of time before the time frame, a configurable predetermined amount of time after the time frame, and a configurable predetermined amount of time during the time frame; and Audit results based on the data are displayed on a dashboard of the mobile asset.

22. The system of claim 20, further adapted to: The audit results are shared on the condition that authorized employees request to share the audit results.

23. The system of claim 20, further adapted to: identifying, based on at least one search criterion, an audit type comprising at least one of random mobile asset selection, random employee selection, and on-demand crew performance analysis; Under the condition that the audit type is random mobile asset selection, identifying a set of configurable predetermined mobile assets; identifying a list of employees associated with the mobile asset, provided that the audit type is random employee selection; Conditional upon the audit type being random employee selection, revising the employee list using a random generator to include a configurable set of predetermined employees associated with the mobile asset; and Employees selected for performance evaluation, provided that the audit type is an on-demand crew performance analysis.

24. The system of claim 23, further adapted to: Select at least one performance review criterion; identifying a set of candidate mobile assets including at least one mobile asset operated by the employee; and The set of candidate mobile assets is revised to include a subset of candidate mobile assets that includes a highest probability of successful random download.

25. The system of claim 23, further adapted to: Select at least one performance review criterion; reviewing on-demand crew performance when the mobile asset is traveling through at least one faulted intersection during inclement weather; and At least one multimedia file is downloaded around the at least one faulty intersection and under inclement weather.

26. The system of claim 20, further adapted to: Identify at least one audit date; and The at least one audit date is sent to at least one authorized employee.

27. The system of claim 24, further adapted to: determining whether an event data recorder associated with the mobile asset is available within the time frame; Under the condition that the event data recorder is available, identifying an asset type associated with the mobile asset, the asset type comprising one of an engine-on and an engine-off; determining whether the mobile asset is in lead service for at least a configurable predetermined amount of time, conditional upon the asset type being no engine and conditional upon the local reverser associated with the mobile asset being one of forward and reverse; and The mobile asset is added to a set of leading serving mobile assets.

28. The system of claim 24, further adapted to: determining whether an event data recorder associated with the mobile asset is available within the time frame; determining, conditional upon a local reverser associated with the mobile asset being one of forward and reverse, that a state of an air brake is associated with the mobile asset, the state comprising one of a passenger cut-in and a freight passenger cut-in; determining whether the mobile asset is in a distributed power mode in a lead service; Under the condition that the asset type is engineless and under the condition that the mobile asset is in a distributed power mode in a lead service, determining whether the mobile asset is in a lead service for at least a configurable predetermined amount of time; and The mobile asset is added to a set of leading serving mobile assets.

29. The system of claim 24, further adapted to: determining whether an event data recorder associated with the mobile asset is available within the time frame; determining whether asset data is available for the mobile asset under the condition that the event data recorder associated with the mobile asset is unavailable within the time frame; determining, subject to the availability of the asset data, whether the mobile asset has been in a lead service for at least a configurable predetermined amount of time; and The mobile asset is added to a set of predecessor serving mobile assets upon condition that the mobile asset is in predecessor service for the configurable predetermined amount of time.

30. The system of claim 24, further adapted to: determining whether an event data recorder associated with the mobile asset is available within the time frame; determining whether asset data is available for the mobile asset under the condition that the event data recorder associated with the mobile asset is unavailable within the time frame; obtaining, under the condition that asset data is unavailable, at least one thumbnail image associated with the mobile asset for the time frame; analyzing the at least one thumbnail image to determine whether an outward-facing camera associated with the mobile asset is obstructed; analyzing the at least one thumbnail image to determine whether the mobile asset was in leading service for at least a configurable predetermined amount of time, provided that the outward-facing camera is not obstructed; and The mobile asset is added to a set of predecessor serving mobile assets upon condition that the mobile asset is in predecessor service for the configurable predetermined amount of time.

31. The system of claim 24, further adapted to: Determine whether the data collection and recording system communicates with the background; Under the condition that the data collection and recording system does not communicate with the background, removing the mobile asset from the candidate mobile asset subset; Under the condition that the data acquisition and recording system communicates with the background, determining whether the mobile asset includes a working locomotive voice and video recorder (LVVR) system; removing the mobile asset from the subset of candidate mobile assets on the condition that the mobile asset does not include the working LVVR system; Under the condition that the mobile asset includes the working LVVR system, determining whether the mobile asset is at a maintenance center; Under the condition that the mobile asset is at the maintenance center, removing the mobile asset from the candidate mobile asset subset; Under the condition that the mobile asset is not in the maintenance center, determining whether the mobile asset is in the station; removing the mobile asset from the subset of candidate mobile assets under the condition that the mobile asset is at the station; adding the mobile asset to the subset of candidate mobile assets including the highest probability of successful random downloading, under the condition that the mobile asset is not at the station; Under the condition that the mobile asset is not at the station, screening the subset of candidate mobile assets including the highest probability of successful random download to include a configurable predetermined number of candidate mobile assets; and At least one multimedia file associated with the mobile asset is requested.

32. The system of claim 20, further adapted to: creating a list of at least one of the at least one configurable predefined threat, the at least one configurable predefined event, and the at least one non-compliance with applicable mobile asset regulations; selecting a random multimedia file associated with the mobile asset; determining whether at least one of the at least one configurable predefined threat, the at least one configurable predefined event, and the at least one non-compliance with applicable mobile asset regulations is detected in the random multimedia file; and At least one of the at least one configurable predefined threat, the at least one configurable predefined event, and the at least one noncompliance with applicable mobile asset regulations detected in the random multimedia file is displayed.

33. The system of claim 32, further adapted to: identifying an event type including at least one of a personal electronic device detection, a camera obstruction, and an intersection fault; Converting the random multimedia file into a set of frames; and Conditional upon the personal electronic device being detected in at least two frames in the set of frames, a prediction of detection of the personal electronic device is identified using the object detection machine learning model.

34. The system of claim 32, further adapted to: identifying at least one class including at least one of cab vacant, cab occupied, camera view obstruction, and camera hardware problem associated with the random multimedia file; and Produces predicted percentages for each class.

35. The system of claim 32, further adapted to: determining, based on the random multimedia file, from at least one front-facing camera associated with the mobile asset, a state of the intersection within a configurable first predetermined time before the mobile asset arrives at the intersection and within a configurable second predetermined time after the mobile asset arrives at the intersection; Extracting images from the random multimedia files for processing; and At least one intersection object in the random multimedia file is detected.

36. The system of claim 35, further adapted to: detecting individual gate arm states based on the at least one intersection object; detecting the position of said individual door arms; Detect the status of individual door arm lights; Determine overall door arm status; and A prediction output for the individual door arm is provided.

37. The system of claim 35, further adapted to: determining an intersection fault event based on the at least one intersection object; and A prediction output of a failure event at the intersection is provided.

38. The system of claim 35, further adapted to: determining a boom flasher status based on the at least one intersection object; and A predicted output of the cantilever flasher is provided.

39. The system of claim 35, further adapted to: detecting individual masthead light flashers based on the at least one intersection object; determining a count of individual masthead light flashers on the at least one intersection object; determining the individual masthead light flasher status and frequency of said individual masthead light flasher; Confirm overall flasher alignment; Determine overall flasher status; and A predicted output of the individual masthead light flashers is provided.

Images