Methods and systems

Abstract

We provide an engineer recertification assistant utilizing a real-time data acquisition and recording system (DARS), a DARS viewer, and a video analytics system for mobile assets. [Solution] DARS includes a data recorder and an onboard data manager. The video analytics system processes video data from at least one camera and motion data from the data recorder for critical events and regulatory requirements based on the mobile asset operator's operational performance, and displays the processed video and motion data, along with episodes, exceptions, and user comments, on a display device via a web portal. The engineer recertification assistant determines automated score-based recommendations for certifying or decertifying the mobile asset operator, or directly certifies or decertifies the mobile asset operator for overall non-compliance.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Provisional Application No. 63 / 061,548, filed on August 5, 2020, and to U.S. Non - Provisional Application No. 17 / 394,135, filed on August 4, 2021, to the extent permitted by law, the contents of which are hereby incorporated by reference in their entirety.

[0002] The disclosures of this application are shown and described below, including the applicant's U.S. Provisional Application No. 61 / 624,142 filed on April 13, 2012, the applicant's U.S. Non-Provisional Application No. 13 / 861,826 filed on April 12, 2013, currently issued as U.S. Patent No. 9,285,294 on March 15, 2016, the applicant's U.S. Non-Provisional Application No. 14 / 608,423 filed on January 29, 2015, currently issued as U.S. Patent No. 9,285,295 on March 15, 2016, and filed on January 15, 2016. The applicant's U.S. Non-Provisional Application No. 14 / 996,925, now U.S. Patent No. 9,915,535, issued on March 13, 2018; the applicant's U.S. Provisional Application No. 62 / 337,227, filed on May 16, 2016; the applicant's U.S. Non-Provisional Application No. 15 / 595,650, filed on May 15, 2017, now U.S. Patent No. 9,934,623, issued on April 3, 2018; the applicant's U.S. Non-Provisional Application No. 15 / 907,486, filed on February 28, 2018, now U.S. Patent No. 15 / 907,486, filed on October 15, 2019. U.S. Patent No. 10,445,951 issued, U.S. Provisional Application No. 62 / 337,225 filed by the applicant on May 16, 2016, U.S. Non-Provisional Application No. 15 / 595,689 filed by the applicant on May 15, 2017, U.S. Patent No. 10,410,441 issued on September 10, 2019, concurrently pending U.S. Non-Provisional Application No. 16 / 385,745 filed by the applicant on April 16, 2019, U.S. Provisional Application No. 62 / 337,228 filed by the applicant on May 16, 2016, May 1, 2017 This information may be used in connection with the present applicant's U.S. Nonprovisional Application No. 15 / 595,712, filed on 5 August 27, 2019, U.S. Patent No. 10,392,038, filed on 27 August 2019, the present applicant's U.S. Provisional Application No. 62 / 825,943, filed on 29 March 2019, the present applicant's U.S. Provisional Application No. 62 / 829,730, filed on 5 April 2019, and the present applicant's concurrently pending U.S. Nonprovisional Application No. 16 / 833,590, filed on 28 March 2020, the contents of which are incorporated herein by reference in their entirety. Each of the above disclosures in their entirety is incorporated herein by reference.All patent applications, patents, and publications referenced herein are incorporated herein by reference in their entirety, except for any definitions, subject matter disclaimers, or abandonments, and unless the incorporated material conflicts with the express disclosure herein (in which case the language of this disclosure shall prevail).

[0003] (Field of invention) This disclosure relates to the automation of a process for assessing the skill performance of railway train operators or engineers responsible for the safe movement of high-value mobile rail assets. [Background technology]

[0004] High-value mobile assets such as locomotives, aircraft, mass transit systems, mining equipment, portable medical equipment, cargo, ships, and military vessels typically utilize onboard data acquisition and recording "black box" systems and / or "event recorder" systems. These data acquisition and recording systems, such as event data recorders or flight data recorders, record various system parameters used for incident investigation, crew performance evaluation, fuel efficiency analysis, maintenance planning, and predictive diagnostics. Typical data acquisition and recording systems include digital and analog inputs, pressure switches, and pressure transducers, and they record data from various onboard sensor devices. Recorded data may include parameters such as speed, distance traveled, location, fuel level, engine revolutions per minute (RPM), fluid level, operator control, pressure, current and predicted weather conditions, and ambient conditions. In addition to basic event and operational data, video and audio event / data recording capabilities are also deployed in many of these same mobile assets. Typically, data is extracted from data recorders after an asset-involved incident occurs, an investigation is required, and the data recorders are recovered. Certain situations may arise where data recorders cannot be recovered or the data is otherwise unavailable. In these situations, data such as event and behavioral data, video data, and audio data captured by data acquisition and recording systems are urgently needed, regardless of whether the data acquisition and recording systems or physical access to the data is available. [Overview of the Initiative]

[0005] This disclosure relates, in general terms, to an engineer re-authentication assistant used for authenticating or de-authenticating engineers or operators in high-value mobile assets. The teachings herein can provide real-time or near real-time access to data such as event and motion data, video data, and audio data recorded by a real-time data acquisition and recording system on a high-value mobile asset. One implementation of a method for automating the assessment of the performance skills of a designated mobile asset operator, comprising: receiving a request from a user using a web portal, including a designated mobile asset operator and a specified time range; receiving data related to the mobile asset operator and the specified time range using a data acquisition and recording system, wherein the data is based on at least one signal from at least one of at least one data sources remote from the mobile asset, which is at least one data source mounted on the mobile asset comprising at least one camera and at least one data recorder of the data acquisition and recording system; processing the data into processed data using an artificial intelligence component of a video analysis system; and displaying the processed data, including at least one video, on a display device using a web portal.

[0006] One implementation of a system for automating the assessment of the performance skills of a designated mobile asset operator includes a web portal adapted to receive requests from a user, including a designated mobile asset operator of a mobile asset and a designated time range; a data acquisition and recording system installed on the mobile asset and adapted to receive data related to the designated mobile asset operator and the designated time range, wherein the data is based on signals from at least one of at least one data sources installed on the mobile asset, which includes at least one camera and at least one data recorder of the data acquisition and recording system, and at least one data source remote from the mobile asset; and an artificial intelligence component of a video analysis system adapted to process the data into processed data, wherein the web portal is adapted to display the processed data, including at least one video, on a display device.

[0007] Modifications of these aspects and other aspects of this disclosure are described in further detail below. [Brief explanation of the drawing]

[0008] The description in this specification refers to the attached drawings, and through several of the drawings, similar reference numbers refer to the same parts. [Figure 1] A field implementation of the first embodiment of the exemplary real-time data acquisition and recording system according to the implementation of this disclosure is illustrated. [Figure 2] A field implementation of a second embodiment of the exemplary real-time data acquisition and recording system according to the implementation of this disclosure is illustrated. [Figure 3] This is a flowchart of the process for recording data and / or information from mobile assets, based on the implementation of this disclosure. [Figure 4] This is a flowchart of the process for adding data and / or information from mobile assets after a power outage, based on the implementation of this disclosure. [Figure 5] This figure illustrates exemplary intermediate and full record blocks stored in a crash-hardened memory module according to the implementation of this disclosure. [Figure 6] This figure illustrates exemplary intermediate recording blocks within a crash-hardened memory module before and after a power outage, according to the implementation of the present disclosure. [Figure 7] This figure illustrates an exemplary recording segment within a crash-hardened memory module after power has been restored, according to the implementation of the present disclosure. [Figure 8] This diagram illustrates the field implementation of the first embodiment of the real-time data acquisition and recording system viewer according to the implementation of this disclosure. [Figure 9] This is a flowchart of the process for recording video data, audio data, and / or information from mobile assets, according to the implementation of this disclosure. [Figure 10] This is a flowchart of the process for recording video data, audio data, and / or information from mobile assets, according to the implementation of this disclosure. [Figure 11] This flowchart illustrates an exemplary fisheye view of a 360-degree camera for a real-time data acquisition and recording system viewer, based on the implementation of the disclosure. [Figure 12] This figure illustrates an exemplary panoramic view of a 360-degree camera of a real-time data acquisition and recording system viewer, according to the implementation of the present disclosure. [Figure 13] This figure illustrates an exemplary quad view of a 360-degree camera for a real-time data acquisition and recording system viewer, according to the implementation of the present disclosure. [Figure 14] This flowchart illustrates an exemplary dewarp view of a 360-degree camera in a real-time data acquisition and recording system viewer, based on the implementation of the disclosure. [Figure 15] This diagram illustrates the field implementation of the first embodiment of the data acquisition and recording system video content analysis system according to the implementation form of this disclosure. [Figure 16A] A diagram illustrating exemplary track detection according to an implementation of the present disclosure. [Figure 16B] A diagram illustrating exemplary track detection and switch detection according to an implementation of the present disclosure. [Figure 16C] A diagram illustrating exemplary track detection according to an implementation of the present disclosure and counting the number of tracks and signal detections. [Figure 16D] A diagram illustrating exemplary level crossing and track detection according to an implementation of the present disclosure. [Figure 16E] A diagram illustrating exemplary dual overhead signal detection according to an implementation of the present disclosure. [Figure 16F] A diagram illustrating exemplary multi-track detection according to an implementation of the present disclosure. [Figure 16G] A diagram illustrating exemplary switch and track detection according to an implementation of the present disclosure. [Figure 16H] A diagram illustrating exemplary switch detection according to an implementation of the present disclosure. [Figure 17] A flowchart of a process for determining the internal state of a mobile asset according to an implementation of the present disclosure. [Figure 18] A flowchart of a process for determining object detection and obstacle detection occurring outside a mobile asset according to an implementation of the present disclosure. [Figure 19] Illustrates a field implementation form of the seventh embodiment of an exemplary real-time data acquisition and recording system according to an implementation of the present disclosure. [Figure 20] A diagram illustrating exemplary signal detection of an automatic signal compliance monitoring and alert system according to an implementation of the present disclosure. [Figure 21] A flowchart of the first embodiment of a process for determining signal compliance according to an implementation of the present disclosure. [Figure 22]FIG. 0 is a diagram of a first embodiment of an engineer re-certification assistant showing a digital video recorder (DVR) video clip screenshot according to an implementation of the present disclosure. [Figure 23] FIG. 3 is a diagram of a first embodiment of an engineer re-certification assistant showing an existing web page enhanced with a predefined event of engineer re-certification such as a signal crossing according to an implementation of the present disclosure. [Figure 24] FIG. 6 is a diagram of a first embodiment of an engineer re-certification assistant showing a screenshot and efficiency according to an implementation of the present disclosure. [Figure 25] FIG. 9 is a flowchart of a first embodiment of a process for assessing skill performance showing a target process according to an implementation of the present disclosure. [Figure 26] FIG. 12 is a screenshot of a first embodiment of an engineer re-certification assistant showing a user selecting an engineer monitoring ride according to an implementation of the present disclosure. [Figure 27] FIG. 15 is a screenshot of a first embodiment of an engineer re-certification assistant showing automatic download of a video for an event of interest according to an implementation of the present disclosure. [Figure 28] FIG. 18 is a diagram of a first embodiment of an engineer re-certification assistant showing a plurality of screenshots depicting sections of a user's engineer evaluation report rules according to an implementation of the present disclosure. [Figure 29] FIG. 21 is a screenshot of a first embodiment of an engineer re-certification assistant showing report generation according to an implementation of the present disclosure. [Figure 30] FIG. 24 is a diagram of a first embodiment of an engineer re-certification assistant showing an engineer evaluation report and an operator score card according to an implementation of the present disclosure. [Figure 31] FIG. 27 is a screenshot of a first embodiment of an engineer re-certification assistant showing a live demonstration of a downloaded video, thumbnail, and icon of a train passing a signal along the line according to an implementation of the present disclosure. [Figure 32] This is a screenshot of the first embodiment of the Engineer Re-authentication Assistant, showing an Engineer Road Foreman (RFE) user who determines the assets and time range for which the user wishes to evaluate engineers on a DVR video download webpage, as implemented in this disclosure. [Figure 33] This is a flowchart of a first embodiment of a process for assessing skill performance according to the implementation of the present disclosure. [Figure 34] This is a flowchart illustrating the operation of an emergency brake with a collision detection system according to an embodiment of the present disclosure. [Figure 35] This flowchart illustrates the operation of fuel compensation using the accelerometer-based pitch and roll of the present invention. [Figure 36] This flowchart illustrates the operation of detecting potentially rough operating conditions using the accelerometer of the present invention. [Figure 37] This flowchart illustrates the operation of the engine operation detection system using the accelerometer of the present invention. [Figure 38] This is a flowchart illustrating the operation of the inertial navigation and dead reckoning navigation system of the present invention. [Figure 39] This figure shows a first embodiment of a mobile asset data recorder and transmitter system, illustrating the components according to the implementation of the present disclosure. [Modes for carrying out the invention]

[0009] A first embodiment of the real-time data acquisition and recording system described herein provides remote users, such as asset owners, operators, and investigators, with real-time or near real-time access to a wide range of data, including event and behavioral data, video data, and audio data, related to high-value assets. The data acquisition and recording system records asset-related data via a data recorder and streams the data to a remote data repository and remote users before, during, and after an incident occurs. The data is streamed to the remote data repository in real-time or near real-time, making the information available at least until the time of the incident or emergency, thereby substantially eliminating the need to find and download a “black box” to investigate an incident involving an asset, and eliminating the need to request the download of specific data, find and transfer files, and interact with the data recorder on the asset to use custom applications to view the data. The system of this disclosure retains typical recording capabilities and adds the ability to stream data to a remote data repository and remote end users before, during, and after an incident. In most situations, the information recorded in the data recorder is redundant and unnecessary because the data has already been acquired and stored in the remote data repository.

[0010] Prior to the system described in this disclosure, data was extracted from a “black box” or “event recorder” after an incident occurred and an investigation was required. Data files containing time segments recorded by the “black box” had to be downloaded and read from the “black box” and then viewed by a user using proprietary software. The user had to obtain physical or remote access to the asset, select the desired data to be downloaded from the “black box,” download the file containing the desired information to a computing device, and use a custom application running on the computing device to find the appropriate file containing the desired data. The system described in this disclosure eliminates the need for the user to perform these steps and requires the user to use only a common web browser to navigate to the desired data. Remotely located users may access a common web browser to navigate to the desired data related to a selected asset and view and analyze the operational efficiency and safety of the asset in real time or near real time.

[0011] Remote users, such as asset owners, operators, and / or investigators, may access a common web browser to navigate to desired live and / or historical data related to selected assets, allowing them to visualize and analyze the operational efficiency and safety of assets in real time or near real time. The ability to visualize operations in real time or near real time enables rapid assessment and adjustment of behavior. During an incident, for example, real-time information and / or data can facilitate situation triage and provide valuable information to first responders. During normal operation, for example, near real-time information and / or data can be used to audit crew performance and support situational awareness across the network.

[0012] The data may include, but is not limited to, data derived from any combination of the above, including, analog and frequency parameters such as speed, pressure, temperature, current, voltage, and acceleration arising from the asset and / or nearby assets; Boolean data such as switch position, actuator position, warning light illumination, and actuator commands; global positioning system (GPS) data and / or geographic information system (GIS) data such as position, speed, and altitude; internally generated information such as restricted speed limits for the asset considering its current location; audio information from microphones located at various locations within, on, or near the asset; information about the asset's operational plan transmitted to the asset from the data center, such as route, schedule, and cargo manifest information; information about environmental conditions, including current and predicted weather conditions in the area where the asset is currently operating or planned to operate; asset control status and operational data generated by systems such as positive train control (PTC) for locomotives; and additional data, video, and audio analysis and analysis.

[0013] Figures 1 and 2 illustrate field implementations of the first and second embodiments of exemplary real-time data acquisition and recording systems (DARS) 100, 200, in which aspects of the present disclosure may be implemented. DARS 100, 200 are systems that deliver real-time information from data recording devices to remotely located end users. DARS 100, 200 include data recorders 154, 254 that are installed on vehicles or mobile assets 148, 248 and communicate with any number of diverse information sources through any combination of onboard wired and / or wireless data links 170, 270 such as wireless gateways / routers, or communicate with non-onboard information sources via data links such as wireless data link 146 through DARS 100, 200 data centers 150, 250. The data recorders 154, 254 include onboard data managers 120, 220, data encoders 122, 222, vehicle event detectors 156, 256, queuing repositories 158, 258, and wireless gateways / routers 172, 272. Furthermore, in this implementation, the data recorders 154, 254 may include crash-hardened memory modules 118, 218 and / or Ethernet switches 162, 262 with or without power over Ethernet (PoE). Exemplary hardened memory modules 118, 218 may be, for example, crash-suitable event recorder memory modules compliant with the Code of Federal Regulations and / or the Federal Railroad Administration Regulations, crash-survivable memory units compliant with the Code of Federal Regulations and / or the Federal Aviation Administration Regulations, crash-hardened memory modules compliant with any applicable Code of Federal Regulations, or any other suitable hardened memory device known in the art. In the second embodiment shown in Figure 2, the data recorder 254 may further include an optional non-crash-hardened removable storage device 219.

[0014] The wired and / or wireless data links 170, 270 may include one or a combination of discrete signal input, standard or proprietary Ethernet, serial connection, and wireless connection. Ethernet-connected devices may utilize the Ethernet switches 162, 262 of the data recorders 154, 254 and may utilize PoE. The Ethernet switches 162, 262 may be internal or external and may support PoE. Furthermore, data from remote data sources such as the map components 164, 264, route / crew manifest components 124, 224, and weather components 126, 226 in the implementation configurations of Figures 1 and 2 is available from the data centers 150, 250 to the onboard data managers 120, 220 and vehicle event detectors 156, 256 via wireless data links 146, 246 and wireless gateways / routers 172, 272.

[0015] The data recorders 154 and 254 collect data or information from a wide variety of sources, which can vary widely based on the asset configuration, via the onboard data links 170 and 270. The data encoders 122 and 222 encode at least a minimum set of data as typically defined by regulatory bodies. In this implementation, the data encoders 122 and 222 receive data from a wide variety of asset sources 148 and 248 and data center sources 150 and 250. The information sources may include any number of components within assets 148, 248, such as analog inputs 102, 202, digital inputs 104, 204, I / O modules 106, 206, vehicle controllers 108, 208, engine controllers 110, 210, inertial sensors 112, 212, Global Positioning System (GPS) 114, 214, cameras 116, 216, positive train control (PTC) / signal data 166, 266, fuel data 168, 268, cellular transmit detector (not shown), internal drive data and any additional data signals, as well as any number of components within data centers 150, 250, such as route / crew manifest components 124, 224, weather components 126, 226, map components 164, 264 and any additional data signals. The data encoders 122 and 222 compress or encode the data and synchronize the data in time to facilitate efficient real-time transmission and replication to the remote data repositories 130 and 230. The data encoders 122 and 222 transmit the encoded data to the onboard data managers 120 and 220, which then store the encoded data in crash-hardened memory modules 118 and 218 and queuing repositories 158 and 258 for replication to the remote data repositories 130 and 230 via the remote data managers 132 and 232 located within the data centers 150 and 250. Optionally, the onboard data managers 120 and 220 can store a tertiary copy of the encoded data in a non-crash-hardened removable storage device 219, as shown in the second embodiment in Figure 2.The onboard data managers 120, 220 and the remote data managers 132, 232 operate in harmony to manage the data replication process. A single remote data manager 132, 232 within data centers 150, 250 can manage data replication from multiple assets 148, 248.

[0016] Data from various input components and from the in-cabin audio / graphical user interface (GUI) 160, 260 is transmitted to the vehicle event detectors 156, 256. The vehicle event detectors 156, 256 process the data to determine whether an event, incident, or other predefined situation has occurred that involves assets 148, 248. When the vehicle event detectors 156, 256 detect a signal indicating that a predefined event has occurred, they transmit the processed data indicating the occurrence of the predefined event, along with supporting data surrounding the event, to the onboard data managers 120, 220. The vehicle event detectors 156, 256 detect events based on data from a wide variety of sources, including analog inputs 102, 202, digital inputs 104, 204, I / O modules 106, 206, vehicle controllers 108, 208, engine controllers 110, 210, inertial sensors 112, 212, GPS 114, 214, cameras 116, 216, route / crew manifest components 124, 224, weather components 126, 226, map components 164, 264, PTC / signal data 166, 266, and fuel data 168, 268, which may vary depending on the asset configuration. When the vehicle event detectors 156 and 256 detect an event, the detected asset event information is stored in the queuing repositories 158 and 258 and can optionally be presented to the crew of assets 148 and 248 via the in-cab audio / graphical user interfaces (GUIs) 160 and 260.

[0017] The onboard data managers 120 and 220 also transmit data to the queuing repository 158. In near real-time mode, the onboard data managers 120 and 220 store encoded data and any event information received from the data encoders 122 and 222 in the crash-hardened memory modules 118 and 218 and the queuing repositories 158 and 258. In the second embodiment of Figure 2, the onboard data manager 220 can optionally store encoded data in a non-crash-hardened removable storage device 219. After five minutes of encoded data have accumulated in the queuing repositories 158 and 258, the onboard data managers 120 and 220 store five minutes of encoded data in the remote data repositories 130 and 230 via remote data managers 132 and 232 in the data centers 150 and 250, through wireless data links 146 and 246 accessed through wireless gateways / routers 172 and 272. In real-time mode, the onboard data managers 120 and 220 store encoded data and arbitrary event information received from the data encoders 122 and 222 in crash-hardened memory modules 118 and 218, optionally in the non-crash-hardened removable storage device 219 shown in Figure 2, and in remote data repositories 130 and 230 via remote data managers 132 and 232 in data centers 150 and 250, accessed through wireless data links 146 and 246 accessed via wireless gateways / routers 172 and 272. The onboard data managers 120 and 220 and the remote data managers 132 and 232 can communicate via various wireless communication links such as Wi-Fi, cellular, satellite, and private wireless systems utilizing the wireless gateways / routers 172 and 272.The wireless data links 146, 246 could be, for example, a wireless local area network (WLAN), a wireless metropolitan area network (WMAN), a wireless wide area network (WWAN), a private wireless system, a cellular network, or any other means of transferring data from the data recorders 154, 254 of DARS 100, 200 to the remote data managers 130, 230 of DARS 100, 200 in this example. If wireless data connectivity is unavailable, the data is stored in memory and queued in the queuing repositories 158, 258 until wireless connectivity is restored and the data replication process can resume.

[0018] In parallel with data recording, data recorders 154 and 254 continuously and autonomously replicate data to remote data repositories 130 and 230. The replication process has two modes: real-time mode and near real-time mode. In real-time mode, data is replicated to remote data repositories 130 and 230 every second. In near real-time mode, data is replicated to remote data repositories 130 and 230 every 5 minutes. The rates used for near real-time mode and real-time mode are configurable, and the rate used for real-time mode can be adjusted to support high-resolution data by replicating data to remote data repositories 130 and 230 every 0.10 seconds. When DARS 100 and 200 are in near real-time mode, the onboard data managers 120 and 220 queue data to queuing repositories 158 and 258 before replicating the data to remote data managers 132 and 232. The onboard data managers 120 and 220 also replicate vehicle event detector information queued in queuing repositories 158 and 258 to remote data managers 132 and 232. Under most conditions, near real-time mode is used during normal operation to improve the efficiency of the data replication process.

[0019] Real-time mode can be initiated based on events that have occurred and been detected by vehicle event detectors 156, 256 installed on assets 148, 248, or by requests initiated from data centers 150, 250. A typical real-time mode request initiated by data centers 150, 250 is initiated when remotely located users 152, 252 request real-time information from web clients 142, 242. Typical reasons for real-time mode occurring on assets 148, 248 are the detection of events or incidents by vehicle event detectors 156, 256, such as an operator initiating an emergency stop request, emergency braking activity, rapid acceleration or deceleration on any axle, or loss of input power to data recorders 154, 254. When transitioning from near real-time mode to real-time mode, all data that has not yet been replicated to the remote data repositories 130, 230 is replicated and stored in the remote data repositories 130, 230, and then live replication is initiated. The transition between near real-time mode and real-time mode typically occurs in less than 5 seconds. After a predetermined time has elapsed since an event or incident, after a predetermined period of inactivity, or when users 152, 252 no longer desire real-time information from assets 148, 248, the data recorders 154, 254 return to near real-time mode. A predetermined time required to initiate the transition is configurable and is typically set to 10 minutes.

[0020] When data recorders 154 and 254 are in real-time mode, the onboard data managers 120 and 220 attempt to continuously empty their queues to remote data managers 132 and 232, storing data in crash-hardened memory modules 118 and 218, optionally in the non-crash-hardened removable storage device 219 (Figure 2), and simultaneously transmitting data to the remote data managers 132 and 232. The onboard data managers 120 and 220 also transmit detected vehicle information queued in queuing repositories 158 and 258 to the remote data managers 132 and 232.

[0021] Upon receiving data replicated from data recorders 154, 254 along with data from map components 164, 264, route / crew manifest components 124, 224, and weather components 126, 226, the remote data managers 132, 232 store the compressed data in remote data repositories 130, 230 within the DARS 100, 200 data centers 150, 250. The remote data repositories 130, 230 may be, for example, cloud-based data storage or any other suitable remote data storage. Once the data is received, a process is initiated in which data decoders 136, 236 decode the recently replicated data to and from the remote data repositories 130, 230 and transmit the decoded data to remote event detectors 134, 234. The remote data managers 132, 232 store vehicle event information in the remote data repositories 130, 230. When the remote event detectors 134 and 234 receive the decoded data, they process the data to determine whether an event of interest has been found within it. The decoded information is then used by the remote event detectors 134 and 234 to detect events, incidents, or other predetermined conditions in the data occurring in assets 148 and 248. If an event of interest is detected from the decoded data, the remote event detectors 134 and 234 store the event information and supporting data in the remote data repositories 130 and 230. When the remote data managers 132 and 232 receive the information from the remote event detectors 134 and 234, they store that information in the remote data repositories 130 and 230.

[0022] Remotely located users 152, 252 can access information, including vehicle event detector information, related to a specific asset 148, 248, or multiple assets, using a standard web client 142, 242 such as a web browser, or, in this implementation, a virtual reality device (not shown) capable of displaying thumbnail images from a selected camera. The web clients 142, 242 communicate the user 152, 252's information requests to the web servers 140, 240 via the network 144, 244 using common web standards, protocols, and techniques. The network 144, 244 could be, for example, the Internet. The network 144, 244 could also be a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a virtual private network (VPN), a mobile phone network, or any other means of transferring data from the web servers 140, 240 to the web clients 142, 242 in this example. Web servers 140 and 240 request the desired data from data decoders 136 and 236. In response to the request from web servers 140 and 240, data decoders 136 and 236 obtain the requested data from remote data repositories 130 and 230, relating to a specific asset 148, 248 or a combination of assets. Data decoders 136 and 236 decode the requested data and send the decoded data to localizers 138 and 238. Localization is the process of converting data into a format desired by the end user, for example, converting the data to the user's preferred language and units of measurement.Localizers 138 and 238, by accessing web clients 142 and 242, identify the profile settings configured by users 152 and 252, and use the profile settings to prepare information to be sent to web clients 142 and 242 for presentation to users 152 and 252, since the raw encoded data and detected event information are stored in remote data repositories 130 and 230 using coordinated universal time (UTC) and the International System of Units (SI units). Localizers 138 and 238 convert the decoded data into a format desired by users 152 and 252, such as their preferred language and units of measurement. Localizers 138 and 238 send the localized data to web servers 140 and 240 in the user's preferred format, as requested. Subsequently, web servers 140, 240 transmit localized data of an asset or multiple assets to web clients 142, 242 for viewing and analysis, and provide playback and real-time display of standard video and 360-degree video. Web clients 142, 242 can display, and users 152, 252 can view data, video, and audio related to a single asset, or view data, video, and audio related to multiple assets simultaneously. Web clients 142, 242 can also provide synchronized playback and real-time display of data along with multiple video and audio data from both standard and 360-degree video sources located on, within, or near the asset, nearby assets, and / or remotely located sites.

[0023] Figure 3 is a flowchart illustrating a process 300 for recording data and / or information from assets 148, 248, according to one implementation of the present disclosure. Data recorders 154, 254 receive data signals from various input components, including physical or computed data elements from assets 148, 248 and data centers 150, 250, such as speed, latitude coordinates, longitude coordinates, horn detection, throttle position, weather data, map data, and / or route and / or crew data 302. Data encoders 122, 222 create a record containing a structured set of bits used to construct and record data signal information 304. The encoded record is then transmitted to onboard data managers 120, 220, which combine the set of records in chronological order into record blocks containing up to 5 minutes of data 306. Intermediate record blocks contain less than 5 minutes of data, and full record blocks contain the full 5 minutes of data. Each record block contains all the data necessary to fully decode the contained signals, including data integrity checks. At a minimum, each recording block must begin with a start record and end with an end record.

[0024] To ensure that all encoded signal data is saved in the crash-hardened memory module 118 and optionally in the non-crash-hardened removable storage device 219 of Figure 2 if the data recorders 154 and 254 lose power or are exposed to extreme temperature or mechanical stress due to a collision or other catastrophic event, the onboard data managers 120 and 220 store intermediate record blocks in the crash-hardened memory module 118 and optionally in the non-crash-hardened removable storage device 219 of Figure 2 at a predetermined rate 308, as illustrated in an exemplary representation in Figure 5, where the predetermined rate is configurable and / or variable. Intermediate record blocks are saved at least once per second, but may be saved at a frequency of once every tenth of a second. The rate at which intermediate record blocks are saved depends on the sampling rate of each signal. All intermediate record blocks contain the entire set of records since the last full record block. To prevent data corruption or loss exceeding one second when the data recorders 154 and 254 lose power while storing data in crash-hardened memory modules 118 and 218 or the optional non-crash-hardened removable storage device 219 of data recorder 254 in Figure 2, the data recorders 154 and 254 can alternately swap between two temporary storage locations in the crash-hardened memory modules 118 and 218 and the optional non-crash-hardened removable storage device 219 of Figure 2 when recording each intermediate record block. Each time a new intermediate record block is saved to a temporary crash-hardened memory location, it overwrites any existing previously stored intermediate record blocks in that location.

[0025] In this embodiment, every 5 minutes, when the data recorders 154, 254 are in near real-time mode, the onboard data managers 120, 220 store the entire recording block, including the encoded signal data from the last 5 minutes, in the recording segment within the crash-hardened memory modules 118, 218 shown in Figure 7, and send a copy of the entire recording block to the remote data managers 132, 232 for storage in the remote data repositories 130, 230 for a predetermined retention period, such as 2 years. The crash-hardened memory modules 118, 218 and / or the optional non-crash-hardened removable storage device 219 of the data recorder 254 in Figure 2 store the recording segment of the most recent recording block for a specified storage duration, which in this implementation is the duration specified by the federal government for which the data recorders 154, 254 must store operational and / or video data in the crash-hardened memory modules 118, 218 using an additional 24-hour buffer, and then overwrite.

[0026] Figure 4 is a flowchart illustrating a process 400 for appending data and / or information from assets 148, 248 after a power outage, according to one implementation of the present disclosure. When power is restored, data recorders 154, 254 identify the last intermediate record block stored in one of two temporary crash-hardened memory locations 402 and verify the last intermediate record block using a 32-bit cyclic redundancy check included in the last record of all record blocks 404. The verified intermediate record block is then appended to a crash-hardened memory record segment, and that record segment, which may contain data up to 5 minutes prior to the power loss, is transmitted to remote data managers 132, 232 and stored for a retention period 406. The encoded signal data is stored in a cyclic buffer for a specified storage duration in the crash-hardened memory modules 118, 218 and / or the optional non-crash-hardened removable storage device 219 of the data recorder 254 in Figure 2. Since the crash-hardened memory recording segment is divided into multiple recording blocks, data recorders 154, 254 delete old recording blocks as needed to free up memory space whenever all recording blocks are saved to the crash-hardened memory modules 118, 218 and / or the optional non-crash-hardened removable storage device 219 of data recorder 254 in Figure 2.

[0027] Figure 6 illustrates exemplary intermediate recording blocks to data recorders 154 and 254 before and after power loss and power recovery. When intermediate recording block 602, stored in temporary storage location 2 (2016 / 2 / 1 10:10:08 AM), is valid, it is appended to recording segment 702 (Figure 7) in the crash-hardened memory modules 118 and 218 of data recorder 254 and / or the optional non-crash-hardened removable storage device 219 in Figure 2, as shown in Figure 7. If the intermediate record block stored in temporary storage location 2 at (2016 / 2 / 1 10:10:08 AM) is invalid, the intermediate record block in temporary storage location 1 at (2016 / 2 / 1 10:10:07 AM) is verified, and if it is valid, it is appended to the recording segment in the optional non-crash-hardened removable storage device 219 of the crash-hardened memory modules 118, 218, and / or the data recorder 254 in Figure 2.

[0028] Whenever any recording block needs to be stored in the crash-hardened memory modules 118, 218 and / or the optional non-crash-hardened removable storage device 219 of the data recorder 254 in Figure 2, the recording segment is immediately flushed to disk. The data recorders 154, 254 alternate between two different temporary storage locations when saving intermediate recording blocks, so that there is always one temporary storage location that has not been changed or flushed to the crash-hardened memory or non-crash-hardened removable storage device, thereby ensuring that at least one of the two intermediate recording blocks stored in the temporary storage location is valid, and that whenever the data recorders 154, 254 lose power, the data recorders 154, 254 do not lose data for more than one second at most. Similarly, when data recorders 154 and 254 are writing data to crash-hardened memory modules 118 and 218 and / or the optional non-crash-hardened removable storage device 219 of data recorder 254 in Figure 2 every tenth of a second, data recorders 154 and 254 will not lose data for more than a tenth of a second whenever data recorders 154 and 254 lose power.

[0029] For the sake of simplicity, processes 300 and 400 are presented and described as a series of steps. However, the steps according to this disclosure can be performed in various orders and / or simultaneously. In addition, the steps according to this disclosure may be performed in conjunction with other steps not presented and described herein. Furthermore, not all illustrated steps are required to carry out the methods according to the disclosed subject matter.

[0030] A third embodiment of the real-time data acquisition and recording system and viewer described herein provides remote users, such as asset owners, operators, and investigators, with real-time or near real-time access to a wide range of data, including event and behavioral data, video data, and audio data, of high-value assets. The data acquisition and recording system records asset-related data via a data recorder and streams the data to a remote data repository and remote users before, during, and after an incident occurs. The data is streamed to the remote data repository in real-time or near real-time, making the information available at least until the time of the incident or emergency, thereby substantially eliminating the need to find and download a “black box” to investigate an incident involving an asset, and eliminating the need to request the download of specific data, find and transfer files, and interact with the data recorder on the asset to use custom applications to view the data. The system of this disclosure retains typical recording capabilities and adds the ability to stream data to a remote data repository and remote end users before, during, and after an incident. In most situations, the information recorded in the data recorder is redundant and unnecessary because the data has already been acquired and stored in the remote data repository.

[0031] Prior to the system described in this disclosure, data was extracted from a “black box” or “event recorder” after an incident occurred and an investigation was required. Data files containing time segments recorded by the “black box” had to be downloaded and read from the “black box” and then viewed by a user using proprietary software. The user had to obtain physical or remote access to the asset, select the desired data to be downloaded from the “black box,” download the file containing the desired information to a computing device, and use a custom application running on the computing device to find the appropriate file containing the desired data. The system described in this disclosure eliminates the need for the user to perform these steps and requires the user to use only a common web browser to navigate to the desired data. Remotely located users may access a common web browser to navigate to the desired data related to a selected asset and view and analyze the operational efficiency and safety of the asset in real time or near real time.

[0032] Remote users, such as asset owners, operators, and / or investigators, may access a common web browser to navigate to desired live and / or historical data related to selected assets, allowing them to visualize and analyze the operational efficiency and safety of assets in real time or near real time. The ability to visualize operations in real time or near real time enables rapid assessment and adjustment of behavior. During an incident, for example, real-time information and / or data can facilitate situation triage and provide valuable information to first responders. During normal operation, for example, near real-time information and / or data can be used to audit crew performance and support situational awareness across the network.

[0033] The third embodiment of the real-time data acquisition and recording system uses at least one, or any combination thereof, of image measurement devices, video measurement devices, and range measurement devices located within, on, or near the mobile asset, as part of the data acquisition and recording system. Image measurement devices and / or video measurement devices include, but are not limited to, 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye view cameras, and / or other cameras. Range measurement devices include, but are not limited to, radar and light detection and ranging ("LIDAR"). LIDAR is a surveying method that measures the distance to a target by irradiating the target with pulsed laser light and measuring the reflected pulses with a sensor. Prior to the systems of this disclosure, the "black box" and / or "event recorder" did not include a 360-degree camera or other camera located within, on, or near the mobile asset. The system disclosed herein adds the ability to use and record video using 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye view cameras, radar, LiDAR, and / or other cameras as part of a data acquisition and recording system, providing remote data repositories and remote users and investigators with 360-degree, narrow-angle, wide-angle, fisheye, and / or other views of the mobile asset, either within the mobile asset, on the mobile asset, or of the mobile asset, before, during, and after an incident involving the mobile asset occurs. The ability to view 360-degree video and / or other video in real time or near real time enables rapid assessment and adjustment of crew behavior. Owners, operators, and investigators can visualize and analyze the operational efficiency and safety of people, vehicles, and infrastructure, and investigate or inspect incidents. The ability to view 360-degree video and / or other video from the mobile asset enables rapid assessment and adjustment of crew behavior. During an incident, for example, 360-degree video and / or other video can facilitate situation triage and provide valuable information to the first responder and investigator.During normal operation, for example, 360-degree video and / or other video can be used to audit crew performance and assist in situational awareness across the entire network. 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye view cameras, radar, LiDAR and / or other cameras provide a complete image of the situation to provide surveillance video for law enforcement and / or railway police, inspection of critical infrastructure, monitoring of level crossings, view track work progress, crew audits both inside the driver's cab and on the premises, and real-time remote monitoring.

[0034] Conventional systems required users to download video files containing time segments in order to view the video files using proprietary software applications or other external video playback applications. The data acquisition and recording system of this disclosure provides 360-degree video, other videos, image information and audio information, and distance measurement information that can be displayed to remote users through the use of virtual reality devices and / or through standard web clients, thereby eliminating the need to download and use external applications to view the video. In addition, users located remotely can view 360-degree video and / or other videos in various modes through the use of virtual reality devices or through standard web clients such as web browsers, thereby eliminating the need to download and use external applications to view the video. Conventional video systems required users to download video files containing time segments of data that could only be viewed using proprietary application software or other external video playback applications that the user had to purchase separately.

[0035] The data may include, but is not limited to, video and image information from cameras located in, on, or near the asset, as well as audio information from microphones located in, on, or near the asset. A 360-degree camera is a camera that provides a 360-degree spherical field of view, a 360-degree hemispherical field of view, and / or a 360-degree fisheye field of view. The use of 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye view cameras, and / or other cameras in, on, or near the asset provides the ability to use and record video using 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye view cameras, and / or other cameras as part of DARS, thereby making 360-degree views and / or other views of the asset in, on, or near the asset available to remote data repositories, remotely located users, and investigators before, during, and after an incident.

[0036] Figure 8 illustrates a field implementation of a third embodiment of an exemplary real-time data acquisition and recording system (DARS) 800 in which aspects of the present disclosure may be implemented. The DARS 800 is a system that delivers real-time information, video information, and audio information from a data recorder 808 on a mobile asset 830 to a remotely located end user via a data center 832. The data recorder 808 is installed on a vehicle or mobile asset 830 and communicates with any number of different information sources through any combination of wired and / or wireless data links, such as a wireless gateway / router (not shown). The data recorder 808 comprises a crash-hardened memory module 810, an onboard data manager 812, and a data encoder 814. In a fourth embodiment, the data recorder 808 may also include a non-crash-hardened removable storage device (not shown). An exemplary enhanced memory module 810 could be, for example, a crash-resistant event recorder memory module compliant with the Federal Code of Regulations and / or the Federal Railroad Administration Code of Regulations, a crash-survivable memory unit compliant with the Federal Code of Regulations and / or the Federal Aviation Administration Code of Regulations, a crash-enhanced memory module compliant with any applicable Federal Code of Regulations, or any other suitable enhanced memory device known in the Art. The wired and / or wireless data link may include one or a combination of discrete signal input, standard or proprietary Ethernet, serial connection, and wireless connection.

[0037] The data recorder 808 collects video data, audio data, and other data and / or information from a wide variety of sources that may vary depending on the asset configuration, via an onboard data link. In this implementation, the data recorder 808 receives data from a video management system 804 that continuously records video and audio data from 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye view cameras, radar, LiDAR, and / or other cameras 802 and fixed camera 806 located within, on, or near asset 830. The video management system 804 stores the video and audio data in a crash-hardened memory module 810, and the video and audio data may also be stored in a non-crash-hardened removable storage device according to a fourth embodiment. Different versions of the video data are created using different bitrates or spatial resolutions, and these versions are separated into variable-length segments, such as thumbnails, 5-minute low-resolution segments, and 5-minute high-resolution segments.

[0038] The data encoder 814 encodes at least a minimum set of data as typically defined by regulatory bodies. The data encoder 814 receives video and audio data from the video management system 804, compresses or encodes the data, and synchronizes the data in time to facilitate efficient real-time transmission and replication to the remote data repository 820. The data encoder 814 transmits the encoded data to the onboard data manager 812, which then transmits the encoded video and audio data to the remote data repository 820 via the remote data manager 818 located in the data center 830, in response to on-demand requests from a remotely located user 834 or in response to specific operating conditions observed on asset 830. The onboard data manager 812 and the remote data manager 818 operate in harmony to manage the data replication process. The remote data manager 818 in the data center 832 can manage data replication from multiple assets. The video and audio data stored in the remote data repository 820 is available to the web server 822 for access by the remotely located user 834.

[0039] The onboard data manager 812 also transmits data to a queuing repository (not shown). The onboard data manager 812 monitors the video and audio data stored in the crash-hardened memory module 810 and / or an optional non-crash-hardened removable storage device of the fourth embodiment via the video management system 804 and determines whether it is in near real-time mode or real-time mode. In near real-time mode, the onboard data manager 812 stores encoded data, including video data, audio data, and any other data or information received from the data encoder 814, as well as any event information, in the crash-hardened memory module 810 and / or an optional non-crash-hardened removable storage device of the fourth embodiment and the queuing repository. After five minutes of encoded data have accumulated in the queuing repository, the onboard data manager 812 stores the five minutes of encoded data in the remote data repository 820 via a wireless data link 816 through a remote data manager 818 in the data center 832. In real-time mode, the onboard data manager 812 stores encoded data, including video data, audio data, and any other data or information, as well as any event information, received from the data encoder 814, in the remote data repository 820 via the wireless data link 816 through the remote data manager 818 in the data center 832 at configurable intervals such as every second or every 0.10 seconds. The onboard data manager 812 and the remote data manager 818 can communicate via various wireless communication links. The wireless data link 816 could be, for example, a wireless local area network (WLAN), a wireless metropolitan area network (WMAN), a wireless wide area network (WWAN), a private wireless system, a mobile phone network, or any other means of transferring data from the data recorder 808 to the remote data manager 818 in this example.The process of remotely transmitting and retrieving video and audio data from asset 830 requires a wireless data connection between asset 830 and data center 832. When the wireless data connection is unavailable, the data is stored and queued in crash-hardened memory module 810 and / or an optional non-crash-hardened removable storage device of the fourth embodiment until wireless connectivity is restored. The video, audio, and any other additional data retrieval processes resume as soon as wireless connectivity is restored.

[0040] In parallel with data recording, the data recorder 808 continuously and autonomously replicates the data to the remote data repository 820. The replication process has two modes: real-time mode and near real-time mode. In real-time mode, data is replicated to the remote data repository 820 every second. In near real-time mode, data is replicated to the remote data repository 820 every 5 minutes. The rates used for near real-time mode and real-time mode are configurable, and the rate used for real-time mode can be adjusted to support high-resolution data by replicating data to the remote data repository 820 every 0.10 seconds. Under most conditions, near real-time mode is used during normal operation to improve the efficiency of the data replication process.

[0041] Real-time mode can be initiated based on events occurring on asset 830 or by requests initiated from data center 832. A typical real-time mode request initiated by data center 832 is initiated when a remotely located user 834 requests real-time information from web client 826. Typical reasons for real-time mode occurring on asset 830 include the detection of an event or incident, such as an operator initiating an emergency stop request, emergency braking activity, rapid acceleration or deceleration on any axis, or loss of input power to data recorder 808. When transitioning from near real-time mode to real-time mode, all data that has not yet been replicated to the remote data repository 820 is replicated and stored in the remote data repository 820, and then live replication is initiated. The transition between near real-time mode and real-time mode typically occurs in less than 5 seconds. After a predetermined time has elapsed since the event or incident, after a predetermined period of inactivity, or when user 834 no longer desires real-time information from asset 830, data recorder 808 returns to near real-time mode. A predetermined time required to initiate the migration is configurable and is typically set to 10 minutes.

[0042] When the data recorder 808 is in real-time mode, the onboard data manager 812 attempts to continuously empty its queue to the remote data manager 818, store the data in the crash-hardened memory module 810, the optional non-crash-hardened removable storage device of the fourth embodiment, and simultaneously transmit the data to the remote data manager 818.

[0043] When the remote data manager 818 receives video data, audio data, and any other data or information to be replicated from the data recorder 808, it stores the data in the remote data repository 820 within the data center 830. The remote data repository 820 may be, for example, a cloud-based data storage or any other suitable remote data storage. When data is received, a process is initiated in which a data decoder (not shown) decodes the recently replicated data from the remote data repository 820 and sends the decoded data to a remote event detector (not shown). The remote data manager 818 stores vehicle event information in the remote data repository 820. When the remote event detector receives encoded data, it processes the decoded data to determine whether an event of interest has been found in the decoded data. The encoded information is then used by the remote event detector to detect events, incidents, or other predetermined conditions in the data occurring in asset 830. When the remote event detector detects an event of interest from the decoded data previously stored in the remote data repository 820, it stores the event information and supporting data in the remote data repository 820.

[0044] Video data, audio data, and any other data or information are transmitted to the remote data repository 820 by the onboard data manager 812 in response to on-demand requests from user 834, and in response to specific operating conditions observed on asset 830. The video data, audio data, and any other data or information stored in the remote data repository 820 are available on the web server 822 for user 834 to access. A remotely located user 834 can access video data, audio data, and any other data or information related to a specific asset 830 or multiple assets stored in the remote data repository 820 using a standard web client 826 such as a web browser, or in this implementation, a virtual reality device 828 that can display thumbnail images of selected cameras. The web client 826 communicates user 834's requests for video, audio, and / or other information to the web server 822 via the network 824 using common web standard protocols and techniques. The network 824 may be, for example, the internet. Network 824 may also be a local area network (LAN), metropolitan area network (MAN), wide area network (WAN), virtual private network (VPN), mobile phone network, or any other means of transferring data from web server 822 to web client 826 in this example. Web server 822 requests the desired data from remote data repository 820. Web server 822 then sends the requested data to web client 826, which provides playback and real-time display of standard video, 360-degree video, and / or other video. Web client 826 plays back the video data, audio data, and any other data or information for viewing and analysis for user 834, who can interact with the 360-degree video data and / or other video data and / or still image data.User 834 can also use the web client 826 to download video data, audio data, and any other data or information, and then use the virtual reality device 828 to interact with the 360-degree video data for viewing and analysis.

[0045] The web client 826 can be enhanced with a software application that provides playback of 360-degree video and / or other video in various different modes. The user 834 can select a mode in which the software application presents video playback, such as a fisheye view as shown in Figure 11, a panoramic view as shown in Figure 12, a double panoramic view (not shown), a quad view as shown in Figure 13, and a dewarp view as shown in Figure 14.

[0046] Figure 9 is a flowchart illustrating a process 840 for recording video data, audio data, and / or information from asset 830 according to one implementation of the present disclosure. The video management system 804 receives data signals from various input components 842, such as a 360-degree camera, a fixed camera, a narrow-angle camera, a wide-angle camera, a 360-degree fisheye view camera, radar, LiDAR, and / or other cameras 802, as well as a fixed camera 806 on, near, or on asset 830. The video management system 804 then stores the video data, audio data, and / or information in a crash-hardened memory module 810 and / or an optional non-crash-hardened removable storage device 844 in a fourth embodiment, using any combination of industry standard formats, such as still images, thumbnails, still image sequences, or compressed video formats. The data encoder 814 creates a recording containing a structured set of bits used to construct and record data signal information 846. In near real-time mode, the video management system 804 stores video data in the crash-hardened memory module 810 and / or an optional non-crash-hardened removable storage device of the fourth embodiment, while sending only limited video data, such as thumbnails or very short low-resolution video segments, to the remote data repository 820 848 without being mounted.

[0047] In another embodiment, the encoded records are then transmitted to an onboard data manager 812, which combines the series of records in chronological order to form record blocks containing up to 5 minutes of data. Intermediate record blocks contain less than 5 minutes of data, and full record blocks contain the full 5 minutes of data. Each record block contains all the data necessary to fully decode the contained signals, including data integrity checks. At a minimum, a record block must begin with a start record and end with an end record.

[0048] To ensure that all encoded signal data is stored in the crash-hardened memory module 810 and / or an optional non-crash-hardened removable storage device of the fourth embodiment when the data recorder 808 loses power, the onboard data manager 812 stores intermediate record blocks in the crash-hardened memory module 810 and / or an optional non-crash-hardened removable storage device of the fourth embodiment at a predetermined rate, the predetermined rate being configurable and / or variable. Intermediate record blocks are stored at least once per second, but may also be stored at a frequency of once every tenth of a second. The rate at which intermediate record blocks are stored depends on the sampling rate of each signal. All intermediate record blocks contain the entire set of records since the last full record block. To prevent data corruption or loss exceeding one second when the data recorder 808 loses power while storing data in the crash-hardened memory module 810, the data recorder 808 may alternate between two temporary storage locations within the crash-hardened memory module 810 when recording each intermediate record block. Whenever a new intermediate record block is saved to a temporary crash-hardened memory location, any existing intermediate record blocks previously stored in that location are overwritten.

[0049] In this embodiment, every 5 minutes, when the data recorder 808 is in near real-time mode, the onboard data manager 812 stores the entire recording block, including the last 5 minutes of encoded signal data, in a recording segment in the crash-hardened memory module 810 and / or an optional non-crash-hardened removable storage device of the fourth embodiment, and sends a copy of the entire recording block, including the 5 minutes of video data, audio data, and / or information, to the remote data manager 818 for storage in the remote data repository 820 for a predetermined retention period, such as 2 years. The crash-hardened memory module 810 and / or the optional non-crash-hardened removable storage device of the fourth embodiment store the recording segment of the most recent recording block for a specified storage duration, which in this implementation is the duration specified by the federal government for which the data recorder 808 must store operational and / or video data in the crash-hardened memory module 810 using an additional 24-hour buffer, and is then overwritten.

[0050] Figure 10 is a flowchart illustrating the process 850 for viewing data and / or information from asset 830 through a web browser 826 or virtual reality device 828. When an event occurs, or when a remotely located authorized user 834 requests a segment of video data stored in crash-hardened memory module 810 via web client 826, the onboard data manager 812, in response to the event, begins transmitting video data to the onboard device in real time at the best available resolution, taking into account the bandwidth of the wireless data link 816. The remotely located user 834 initiates a request for specific video and / or audio data in a specific view mode 852 via web client 826, which communicates the request to web server 822 via network 824. The web server 822 requests the specific video and / or audio data from remote data repository 820 and transmits the requested video and / or audio data to web client 826 854 via network 824. The web client 826 displays the video and / or audio data in the view mode specified by user 834 856. The user 834 can then download specific video and / or audio data and view it on the virtual reality device 828. In another implementation, in real-time mode, thumbnails are first transmitted at one-second intervals, followed by short segments of low-resolution video, and then short segments of high-resolution video.

[0051] For the sake of simplicity, processes 840 and 850 are presented and described as a series of steps. However, the steps according to this disclosure can be performed in various orders and / or simultaneously. In addition, the steps according to this disclosure may be performed in conjunction with other steps not presented and described herein. Furthermore, not all illustrated steps are required to carry out the method according to the disclosed subject matter.

[0052] A fifth embodiment of the real-time data acquisition and recording system and video analysis system described herein provides a remotely located user with real-time or near-real-time access to a wide range of data, including event and behavioral data, video data, and audio data of high-value assets. The data acquisition and recording system records data related to the asset and streams the data to a remote data repository and the remotely located user before, during, and after an incident occurs. The data is streamed to the remote data repository in real-time or near-real-time, making the information available at least until the time of the incident or emergency, thereby virtually eliminating the need to find and download a “black box” to investigate an incident involving the asset. DARS performs video analysis of recorded video data of mobile assets to determine, for example, cab occupancy, truck detection, and detection of objects near trucks. Remote users can use a common web browser to navigate to and view desired data related to selected assets, interact with the data acquisition and recording system on the assets, request downloads of specific data, locate or transfer files, and view the data without needing to use custom applications.

[0053] DARS provides remote users with access to video data and video analysis performed by the video analysis system by streaming data to a remote data repository and remote users before, during, and after an incident, thereby eliminating the need for users to manually download, extract, and play videos to review the video data and determine whether crew members or unauthorized personnel were present at the time of the incident, track detection, object detection near the track, investigation, or any other point of interest, or to determine cab occupancy. In addition, the video analysis system provides cab occupancy determination, track detection, object detection near the track, and lead and trailing unit determination by processing image and video data in real time, thereby ensuring that correct data is always available to the user. For example, real-time image processing ensures that a locomotive designated as the trailing locomotive is not in lead service, in order to enhance railway safety. Conventional systems provided locomotive positions within trains by using the train formation functionality in the dispatch system. Sometimes, the information is not updated in real time, and crew members may change locomotives when deemed necessary, so the dispatch system information can become outdated.

[0054] Prior to the system of this disclosure, inspection crews and / or asset crews had to manually inspect the condition of the truck, manually check whether the vehicle was in the leading or trailing position, manually survey the location of individual objects of interest, manually create a database of the geographical locations of all objects of interest, periodically perform manual field surveys of each object of interest to verify their locations, identify changes in geographical location that differ from the original survey, manually update the database when objects of interest have changed location due to repairs or additional infrastructure development since the original database was created, select and download desired data from digital video recorders and / or data recorders, inspect the downloaded data and / or video offline, check the truck for obstacles, and vehicle operators had to physically check for obstacles and / or switch changes. The system of this disclosure eliminates the need for users to perform these steps and requires users to use a common web browser to navigate to the desired data. Asset owners and operators can automate and improve the efficiency and safety of mobile assets in real time, proactively monitor track status, and obtain alert information in real time. The system of this disclosure eliminates the need for asset owners and operators to download data from data recorders to monitor track status and investigate incidents. As an active safety system, DARS can help operators check for any obstacles, send alerts in real time, and / or store information offline, and send alert information for remote monitoring and storage. Current and historical track detection information and / or information on the detection of objects near the track can be stored in a remote data repository in real time to help users view the information when needed. Remotely located users may access a common web browser and navigate to desired data related to selected assets to view and analyze the operational efficiency and safety of assets in real time or near real time.

[0055] The real-time data acquisition and recording system of the fifth embodiment can be used to continuously monitor an object of interest and to identify in real time when the object of interest moves or is damaged, is obscured by leaves, and / or is under repair and requires maintenance. DARS has the ability to use video, image, and / or audio information to detect and identify various infrastructure objects, such as rail tracks, in video, to follow the track as the mobile asset moves, and to create, audit, and periodically update a database of objects of interest with geographic locations. The real-time data acquisition and recording system of the fifth embodiment uses at least one or any combination of image measurement devices, video measurement devices, and range measurement devices within, on, or near the mobile asset as part of the data acquisition and recording system. Image measurement devices and / or video measurement devices include, but are not limited to, 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye view cameras, and / or other cameras. Range measurement devices include, but are not limited to, radar and light detection and ranging ("LIDAR"). LiDAR is a surveying method that measures the distance to a target by irradiating the target with pulsed laser light and measuring the reflected pulses with a sensor.

[0056] DARS can automatically inspect track conditions, such as counting the number of tracks present, identifying the current track on which moving assets are moving, and detecting any obstacles or defects present, including ballast spills, track damage, off-gauge tracks, misaligned switches, switch overturns, track flooding, and snow accumulation, allowing for the planning of any preventative maintenance to avoid any catastrophic events. DARS can also detect rail track switches and track changes. DARS can further detect changes in data location, including objects being lost, obstructed, and / or not present in their expected location. Track detection, infrastructure diagnostic information, and / or infrastructure monitoring information can be displayed to the user through the use of any standard web client, such as a web browser, thereby eliminating the need to download files from the data recorder and use proprietary application software or other external applications to view the information, as required by conventional systems. This process can be extended to automatically create, audit, and / or update databases with geographical locations of interest to ensure compliance with the Code of Federal Regulations. The system described in this disclosure utilizes cameras previously installed to comply with the Code of Federal Regulations to perform a variety of tasks that previously required human interaction, dedicated vehicles, and / or alternative equipment. DARS enables these tasks to be performed automatically as mobile assets travel throughout the region as part of normal revenue service and daily operations. DARS can be used to save countless manpower and time by utilizing the normal operation of vehicles and previously installed cameras to accomplish tasks that previously required manual work. DARS can also perform tasks previously performed using dedicated vehicles, preventing the closure of truck segments to inspect and locate trucks and objects of interest, which often resulted in loss of revenue service and expensive equipment to purchase and maintain.DARS further reduces the amount of time humans need to be near rail tracks, thereby reducing overall accidents and potential loss of life.

[0057] The data includes measured analog and frequency parameters such as speed, pressure, temperature, current, voltage, and acceleration from mobile assets and / or nearby mobile assets; measured Boolean data such as switch position, actuator position, warning light illumination, and actuator commands; location, speed, and altitude information from the Global Positioning System (GPS), as well as additional data from Geographic Information Systems (GIS) such as latitude and longitude of various objects of interest; internally generated information such as speed limits for mobile assets considering their current location; train control status and operation data generated by systems such as Positive Train Control (PTC); vehicle and inertial parameters such as speed, acceleration, and location received from GPS; and GIS data such as latitude and longitude of various objects of interest. The data may include, but is not limited to, information about the operation plan of the mobile asset transmitted from the data center to the mobile asset, such as video and image information from at least one camera located in, on, or near the mobile asset, audio information from at least one microphone located in, on, or near the mobile asset, route, schedule, and cargo manifest information, information about environmental conditions such as current and predicted weather in the area where the mobile asset is currently operating or is planned to operate, and data derived from any combination of the above sources, including additional data, video, and audio analysis and analysis.

[0058] "Track" may include, but is not limited to, the rails and sleepers of a railway used for locomotives and / or train transport. "Object of Interest" may include, but is not limited to, various objects of infrastructure installed and maintained within the vicinity of a railway track, which may be identified using artificial intelligence such as supervised learning or reinforcement learning of asset camera images and videos. Supervised learning and / or reinforcement learning utilizes a previously labeled dataset defined as "training" data to enable remote and autonomous identification of objects in, on, or in the view of a camera near a mobile asset. Supervised learning and / or reinforcement learning trains a neural network model to identify patterns occurring in visual images obtained from the camera. These patterns, such as people, barriers, cars, trees, signals, and switches, may be found in a single image. Sequential frames in a video can also be analyzed for patterns such as flashing signals, moving cars, or people falling asleep. DARS may or may not require human interaction at any stage of the implementation, including, but not limited to, labeling the training dataset required for supervised learning and / or reinforcement learning. Objects of interest include, but are not limited to, tracks, track centerline points, milepost markers, signals, barriers, switches, level crossings, and text-based signs. "Video analysis" refers to any tangible information collected by analyzing video and / or images recorded from image measuring devices, video measuring devices, and / or range measuring devices, such as at least one camera, including but not limited to, a 360-degree camera, a fixed camera, a narrow-angle camera, a wide-angle camera, a 360-degree fisheye view camera, radar, LiDAR, and / or other cameras within, on, or near the mobile asset. Video analysis systems can also be used in any mobile asset, living area, space, or room, including surveillance cameras, to enhance video surveillance.In mobile assets, video analytics systems provide remote users with an economical and efficient way to detect autonomous cabin occupancy events.

[0059] Figure 15 illustrates a field implementation of a fifth embodiment of an exemplary real-time data acquisition and recording system (DARS) 900 in which aspects of the present disclosure may be implemented. The DARS 900 is a system that delivers real-time information, video information, and audio information from a data recorder 902 on a mobile asset 964 to a remotely located end user 968 via a data center 966. The data recorder 902 is installed on a vehicle or mobile asset 964 and communicates with any number of different information sources through any combination of wired and / or wireless data links 942, such as a wireless gateway / router (not shown). The data recorder 902 collects video data, audio data, and other data or information from a wide variety of sources, which may vary depending on the asset configuration, via the onboard data link 942. The data recorder 902 includes, within the asset 964, local memory components such as a crash-hardened memory module 904, an onboard data manager 906, and a data encoder 908. In a sixth embodiment, the data recorder 902 may also include a non-crash-hardened removable storage device (not shown). An exemplary hardened memory module 904 could be, for example, a crash-suitable event recorder memory module compliant with the Code of Federal Regulations and / or the Federal Railroad Administration Regulations, a crash-survivable memory unit compliant with the Code of Federal Regulations and / or the Federal Aviation Administration Regulations, a crash-hardened memory module compliant with any applicable Code of Federal Regulations, or any other suitable hardened memory device known in the Art. The wired and / or wireless data link may include one or a combination of discrete signal input, standard or proprietary Ethernet, serial connection, and wireless connection.

[0060] DARS900 further comprises a video analysis system 910 including a track and / or object detection and infrastructure monitoring component 914. The track detection and infrastructure monitoring component 914 comprises an artificial intelligence component 924 such as a supervised learning and / or reinforcement learning component, or other neural network or artificial intelligence component, an object detection and location component 926, and an obstacle detection component 928 that detects camera obstacles such as obstacles and / or personnel that obstruct the camera's view, which are present on or near the track. In this implementation, live video data is captured by at least one camera 940 mounted in the driver's cab of asset 964, on asset 964, or near asset 964. The camera 940 is positioned at an appropriate height and angle to capture video data within and around asset 964 and to obtain a sufficient amount of view for further processing. Live video data and image data are captured by the camera 940 in front of and / or around asset 964 and fed to the track and / or object detection and infrastructure monitoring component 914 for analysis. The track detection and infrastructure monitoring component 914 of the video analysis system 910 processes live video and image data frame by frame to detect the presence of rail tracks and any objects of interest. Camera position parameters such as height, angle, shift, focal length, and field of view may be supplied to the track and / or object detection and infrastructure monitoring component 914, or the camera 940 may be configured to allow the video analysis system 910 to detect and determine the camera position and parameters.

[0061] To perform state determinations such as cab occupancy detection, the video analysis system 910 uses supervised learning and / or reinforcement learning components 924, and / or other artificial intelligence and learning algorithms to evaluate asset data 934, such as video data from camera 940, speed, GPS data, and inertial sensor data, weather component 936 data, and route / crew manifests, as well as GIS component data 938. Cab occupancy detection is inherently susceptible to environmental noise sources such as light reflected from clouds and sunlight passing through buildings and trees while the asset is in motion. To handle environmental noise, the supervised learning and / or reinforcement learning components 924, object detection and location components 926, obstacle detection components, asset component 934 data, which may include speed, GPS data, and inertial sensor data, weather component 936 data, and other learning algorithms are configured together to form internal and / or external state determinations involving the mobile asset 964. The track and / or object detection and infrastructure monitoring component 914 may also include a facial recognition system adapted to enable authorization of access to the locomotive as part of a locomotive security system, a fatigue detection component adapted to monitor crew attention, and an activity detection component for detecting unauthorized activities such as smoking.

[0062] In addition, the video analysis system 910 may receive location information from the asset owner, including the latitude and longitude coordinates of signals such as stop signals, traffic signals, speed limit signals, and / or object signals near trucks. The video analysis system 910 then determines whether the location information received from the asset owner is correct. If the location information is correct, the video analysis system 910 stores the information and does not recheck the location information again for a predetermined period of time, such as checking the location information on a monthly basis. If the location information is incorrect, the video analysis system 910 determines the correct location information, reports the correct location information to the asset owner, stores the location information, and does not recheck the location information again for a predetermined period of time, such as checking the location information on a monthly basis. Storing location information provides easier detection of signals such as stop signals, traffic signals, speed limit signals, and / or object signals near trucks.

[0063] Artificial intelligence, such as supervised learning and / or reinforcement learning of the track using the artificial intelligence component 924, is performed by utilizing various information obtained from a sequence of video and / or image frames, and by using additional information received from the data center 966 and the vehicle data component 934, including inertial sensor data and GPS data, to determine the training data. The object detection and location component 926 determines object detection data by distinguishing rail tracks, signs, signals, etc. from other objects, using the training data received from the supervised learning and / or reinforcement learning component 924, as well as specific information about the mobile assets 964 and the railway, such as track width and curvature, sleeper positioning, and vehicle speed. The obstacle detection component 928 improves accuracy and determines obstacle detection data by utilizing object detection data received from the object detection and location component 926, such as information on camera obstacles such as obstacles present on or near the track and / or personnel obstructing the camera's view, as well as additional information from the weather component 936, route / crew manifest data and GIS data component 938, and vehicle data component 934, which includes inertial sensor data and GPS data. Mobile asset data from vehicle data component 934 includes, but is not limited to, speed, location, acceleration, yaw / pitch rate, and level crossings. Any additional information received from and utilized from the data center 966 includes, but is not limited to, day / night details and geographical location of the mobile asset 964.

[0064] Information processed by the infrastructure object, track and / or object detection and infrastructure monitoring component 914, as well as diagnostic and monitoring information, is transmitted via the onboard data link 942 to the data encoder 908 of the data recorder 902 for encoding. The data recorder 902 stores the encoded data in the crash-hardened memory module 904, optionally in the optional non-crash-hardened removable storage device of the sixth embodiment, and transmits the encoded information via the wireless data link 944 to the remote data manager 946 in the data center 966. The remote data manager 946 stores the encoded data in the remote data repository 948 in the data center 966.

[0065] To determine obstacle detection 928 or object detection 926 such as trucks in front of an asset, objects on and / or near the truck, obstacles on or near the truck, and / or obstacles obstructing the camera view (964), the vehicle analysis system 910 processes and evaluates camera images and video data from camera 940 in real time using supervised learning and / or reinforcement learning components 924, or other artificial intelligence, object detection and location components 926, and obstacle detection components 928, as well as other image processing algorithms. The truck and / or object detection and infrastructure monitoring component 914 uses the processed video data together with asset component 934 data, which may include speed, GPS data, and inertial sensor data, weather component 936 data, and route / crew, manifest, and GIS component 938 data to determine external conditions such as leading and mating mobile assets in real time. When processing image and video data for track and / or object detection, for example, the video analysis system 910 automatically configures the camera 940 parameters required for track detection, detects travel through switches, counts the number of tracks, detects any additional tracks along the side of asset 964, determines which track asset 964 is currently traveling on, detects track geometry defects, detects track runoff scenarios such as detecting water near the track within the defined limits of the track, and detects lost gradient or track scenarios. Object detection accuracy depends on the existing lighting conditions within and around asset 964. DARS 900 handles different lighting conditions with the help of additional data collected from the installed asset 964 and data center 966.The DARS900 can be enhanced to operate in a variety of lighting conditions, in a variety of weather conditions, to detect more objects of interest, to integrate with existing database systems to automatically create, audit, and update data, to detect multiple tracks, to operate consistently with curved tracks, to detect any obstacles, to detect any track defects that could cause safety issues, and to operate in low-cost embedded systems.

[0066] Internal and / or external state determinations from the video analysis system 910, including object detection and location such as cab occupancy, truck detection and detection of objects near the truck, and obstacle detection such as obstacles on or near the truck and obstacles obstructing the camera, are provided to the data recorder 902 via the onboard data link 942, along with any data from the vehicle management system (VMS) or digital video recorder component 932. The data recorder 902 stores the internal and / or external state determination, object detection and location component 926 data, and obstacle detection component 928 data in the crash-hardened memory module 904, optionally in a non-crash-hardened removable storage device of the sixth embodiment, and in the remote data repository 948 via the remote data manager 946 located in the data center 966. The web server 958 provides the internal and / or external state determination, object detection and location component 926 information, and obstacle detection component 928 information to a remotely located user 968 via the web client 962 upon request.

[0067] The data encoder 908 encodes at least a minimum set of data as typically defined by regulatory bodies. The data encoder 908 receives video, image, and audio data from one of the cameras 940, the video analysis system 910, and the video management system 932, and compresses or encodes and time-synchronizes the data to facilitate efficient real-time transmission and replication to the remote data repository 948. The data encoder 908 transmits the encoded data to the onboard data manager 906, which then transmits the encoded video, image, and audio data to the remote data repository 948 via the remote data manager 946 located within the data center 966, in response to on-demand requests from the user 968 or to specific operating conditions observed on the asset 964. The onboard data manager 906 and the remote data manager 946 operate in harmony to manage the data replication process. The remote data manager 946 within the data center 966 can manage the replication of data from multiple assets 964.

[0068] The onboard data manager 908 determines, based on the prioritization of detected events, whether detected events, internal and / or external state determinations, object detection and location, and / or obstacle detection should be queued or transmitted immediately. For example, under normal operating conditions, detecting an obstacle on the track is far more urgent than detecting whether someone is in the driver's cab of asset 964. The onboard data manager 908 also transmits data to a queuing repository (not shown). In near real-time mode, the onboard data manager 988 stores the encoded data and any event information received from the data encoder 908 in the crash-hardened memory module 904 and the queuing repository. After five minutes of encoded data have accumulated in the queuing repository, the onboard data manager 906 stores the five minutes of encoded data in the remote data repository 948 via the wireless data link 944 through the remote data manager 946 in the data center 966. In real-time mode, the onboard data manager 908 stores the encoded data and any event information received from the data encoder 908 in the crash-hardened memory module 904 and the remote data repository 948 via the wireless data link 944 and the remote data manager 946 in the data center 966 at configurable predetermined intervals, such as every second or every 0.10 seconds.

[0069] In this implementation, the onboard data manager 906 transmits video data, audio data, internal and / or external state determination, object detection and location information, obstacle detection information, and any other data or event information to the remote data repository 948 via a wireless data link 944 through a remote data manager 946 in the data center 966. The wireless data link 944 could be, for example, a wireless local area network (WLAN), a wireless metropolitan area network (WMAN), a wireless wide area network (WWAN), a wireless virtual private network (WVPN), a mobile phone network, or any other means of transferring data from the data recorder 902 to the remote data manager 946 in this example. The process of remotely reading data from asset 964 requires a wireless connection between asset 964 and the data center 966. When the wireless data connection is unavailable, the data is stored and queued until wireless connectivity is restored.

[0070] In parallel with data recording, the data recorder 902 continuously and autonomously replicates the data to the remote data repository 948. The replication process has two modes: real-time mode and near real-time mode. In real-time mode, data is replicated to the remote data repository 10 every second. In near real-time mode, data is replicated to the remote data repository 15 every 5 minutes. The rates used for near real-time mode and real-time mode are configurable, and the rate used for real-time mode can be adjusted to support high-resolution data by replicating data to the remote data repository 15 every 0.10 seconds. Under most conditions, near real-time mode is used during normal operation to improve the efficiency of the data replication process.

[0071] Real-time mode can be initiated based on events occurring on asset 964 or by requests initiated from data center 966. A typical real-time mode request initiated by data center 966 is initiated when a remotely located user 968 requests real-time information from web client 962. Typical reasons for real-time mode occurring on asset 964 include the detection of an event or incident involving asset 964, such as an operator initiating an emergency stop request, emergency braking activity, rapid acceleration or deceleration on any axis, or loss of input power to data recorder 902. When transitioning from near real-time mode to real-time mode, all data that has not yet been replicated to the remote data repository 948 is replicated and stored in the remote data repository 948, and then live replication is initiated. The transition between near real-time mode and real-time mode typically occurs in less than 5 seconds. After a predetermined time has elapsed since the event or incident, after a predetermined period of inactivity, or when user 968 no longer desires real-time information from asset 964, data recorder 902 returns to near real-time mode. A predetermined time required to initiate the migration is configurable and is typically set to 10 minutes.

[0072] When the data recorder 902 is in real-time mode, the onboard data manager 906 attempts to continuously empty its queue to the remote data manager 946, optionally store the data in the crash-hardened memory module 940, optionally in the optional non-crash-hardened removable storage device of the sixth embodiment, and simultaneously transmit the data to the remote data manager 946.

[0073] Upon receiving video data, audio data, internal and / or external state determination, object detection and location information, fault detection information, and any other data or information replicated from the data recorder 902, the remote data manager 946 stores the data received from the onboard data manager 906, including encoded data and detected event data, in a remote data repository 948 within the data center 966. The remote data repository 948 may be, for example, a cloud-based data storage or any other suitable remote data storage. When data is received, a process is initiated in which the data decoder 954 decodes the recently replicated data from the remote data repository 948 and sends the decoded data to the track / object detection / location information component 950, which looks up the stored data for additional "post-processed" events. In this implementation, the track / object detection / location information component 950 includes an object / obstacle detection component for determining internal and / or external state determination, object detection and location information, and obstacle detection information. When the track / object detection / location information component 950 detects internal and / or external information, object detection and location information, and / or obstacle detection information, it stores the information in the remote data repository 948.

[0074] A remotely located user 968 can access video data, audio data, internal and / or external status determination, object detection and location information, fault detection information, and any other information stored in the remote data repository 948, including track information, asset information, and cab occupancy information related to a specific asset 964 or multiple assets, using a standard web client 962 such as a web browser, or, in this implementation, a virtual reality device (not shown) such as the virtual reality device 828 in Figure 8, which can display thumbnail images of selected cameras. The web client 962 communicates the user 968's information request to the web server 958 via the network 960 using common web standards, protocols, and techniques. The network 960 could be, for example, the Internet. The network 960 could also be a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a virtual private network (VPN), a mobile phone network, or any other means of transferring data from the web server 958 to the web client 962 in this example. The web server 958 requests the desired data from the remote data repository 948, and the data decoder 954 retrieves the requested data related to a specific asset 964 from the remote data repository 948 in response to the request from the web server 958. The data decoder 954 decodes the requested data and sends the decoded data to the localizer 956. The localizer 956, knowing that the raw encoded data and detected event information are stored in the remote data repository 948 using coordinated Coordinated Universal Time (UTC) and the International System of Units (SI units), accesses the web client 962 and, by using the profile settings, identifies the profile settings set by user 968 and prepares the information being sent to the web client 962 for presentation to user 968. The localizer 956 converts the decoded data into a format desired by user 968, including user 968's preferred units of measurement and language.The localizer 956, upon request, sends the localized data to the web server 958 in the user 968's preferred format. The web server 958 then sends the localized data to the web client 962 for viewing and analysis, providing playback and real-time display of standard video and 360-degree video, along with internal and / or external condition determination, object detection and location information, and obstacle detection information, such as track and / or object detection (Figure 16A), track and switch detection (Figure 16B), track and / or object detection, track counting and signal detection (Figure 16C), level crossing and track and / or object detection (Figure 16D), dual overhead signal detection (Figure 16E), multi-track and / or multi-object detection (Figure 16F), switch and track and / or object detection (Figure 16G), and switch detection (Figure 16H).

[0075] The web client 962 is enhanced with a software application that provides playback of 360-degree video and / or other video in various different modes. The user 968 can select a mode in which the software application presents video playback, such as fisheye view, dewarp view, panoramic view, double panoramic view, and quad view.

[0076] Figure 17 is a flowchart showing a process 970 for determining the internal state of asset 964 in one implementation of the present disclosure. The video analysis system 910 receives data signals from various input components 972, including but not limited to cameras 940, 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye view cameras, radar, LiDAR, and / or other cameras, on, within, or near asset 964, a vehicle data component 934, a weather component 936, and a route / manifest / GIS component 938. The video analysis system 910 processes the data signals using a supervised learning and / or reinforcement learning component 974 to determine the internal state 976, such as cab occupancy.

[0077] Figure 18 is a flowchart showing a process 980 for determining object detection / location and obstacle detection occurring outside and inside asset 964, according to one implementation of the present disclosure. The video analysis system 910 receives data signals from various input components 982, including but not limited to cameras 940, 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye view cameras, radar, LiDAR, and / or other cameras, vehicle data component 934, weather component 936, and route / manifest / GIS component 938, on, inside, or near asset 964. The video analysis system 910 processes the data signals using supervised learning and / or reinforcement learning component 924, object detection / location component 926, and obstacle detection components 928, 984 to determine obstacle detection 986, and object detection and location, such as the presence of a truck (988).

[0078] For the sake of simplicity, processes 970 and 980 are shown and described as a series of steps. However, the steps according to this disclosure can be carried out in various orders and / or simultaneously. In addition, the steps according to this disclosure may be carried out in conjunction with other steps not shown and described herein. Furthermore, not all illustrated steps are required to carry out the method according to the disclosed subject matter.

[0079] A seventh embodiment of the real-time data acquisition and recording system and automated signal compliance monitoring and alerting system described herein provides remote users, such as asset owners, operators, and investigators, with real-time or near real-time access to a wide range of data, including event and behavioral data, video data, and audio data, related to high-value assets. The automated signal compliance monitoring and alerting system records asset-related data via a data recorder and streams the data to a remote data repository and remote users before, during, and after an incident occurs. The data is streamed to the remote data repository in real-time or near real-time, making the information available at least until the time of the incident or emergency, thereby substantially eliminating the need to find and download a “black box” to investigate an incident involving an asset, and eliminating the need to request the download of specific data, find and transfer files, and interact with the data recorder on the asset to use a custom application to view the data. The system of this disclosure retains typical recording capabilities and adds the ability to stream data to a remote data repository and remote end users before, during, and after an incident. In most situations, the information recorded in the data recorder is redundant and unnecessary because the data has already been retrieved and stored in a remote data repository.

[0080] The automated signal monitoring and alerting system also automatically monitors mobile assets such as locomotives, trains, airplanes, and automobiles for operating unsafely in violation of signal types such as stop signals, traffic signals, and / or speed limit signals, or in attempting to maintain compliance with such signals, and provides historical and real-time alerts regarding them. The automated signal monitoring and alerting system combines the use of image analysis, GPS location, braking force, and vehicle speed, as well as automated electronic notifications, to alert personnel on board and / or not on board mobile assets in real time when a mobile asset violates safe operating rules, such as when a stop signal is passed by a mobile asset before it has a right to stop and receive (running a red light), when a speed limit signal indicating a deceleration limit is violated by a mobile asset moving at a higher speed, and when a mobile asset applies slow and / or excessive braking force to stop before it has passed a stop / red light.

[0081] Prior to the automated signal monitoring and alerting system of this disclosure, operations center personnel relied on mobile asset crews to report when safety operating rules were violated. Occasionally, catastrophic mobile asset collisions occurred, and subsequent investigations revealed that safety operating rule violations had occurred. In addition, excessive braking force may have caused mechanical failure in parts of the mobile asset, and in situations where the mobile asset was a locomotive and / or train, excessive braking force may have resulted in derailment, and subsequent investigations found safety operating rule violations to be the root cause. The system of this disclosure enables users to monitor and / or receive alerts when safety operating rule violations occur, before mechanical failures, collisions, derailments, and / or other accidents occur.

[0082] End users can subscribe to be alerted when a safety operation rule violation occurs and receive email, text message, and / or in-browser electronic notifications within minutes of the actual event occurring. End users can use the historical records to analyze the data and identify patterns such as the location of the problem, impaired line of sight, faulty equipment, and underperforming crew members, which may be useful when implementing new and safer operation rules or crew training opportunities for continuous improvement. The system of this disclosure enables end users to leverage continuous electronic monitoring and extensive image analysis to understand any and all times when mobile assets are operating unsafely due to safety operation rule violations and / or non-compliance with signals.

[0083] Automated signal monitoring and alerting systems are used by vehicle and / or mobile asset owners, operators, and investigators to visualize and analyze the operational efficiency and safety of mobile assets in real time. The ability to view operations in real time enables rapid assessment and adjustment of behavior. During an incident, real-time information can facilitate situation triage and provide valuable information to the first responder. During normal operation, near real-time information can be used to audit crew performance and support operational safety and awareness across the entire network.

[0084] The automated signal monitoring and alerting system is a fully integrated, time-synchronized automated system that utilizes outward-facing cameras and / or other cameras, GPS location, speed, and acceleration data, as well as vehicle, train, and / or mobile asset brake pressure sensor data to identify unsafe and potentially catastrophic operational practices and provide real-time feedback to mobile asset crews and management. The automated signal monitoring and alerting system also provides automated data and video downloads to users with various data sources to enable complete knowledge of the operating environment when an alert is triggered.

[0085] The data may include, but is not limited to, data derived from any combination of the above, including, analog and digital parameters such as speed, pressure, temperature, current, voltage, and acceleration arising from the asset and / or nearby assets; Boolean data such as switch position, actuator position, warning light illumination, and actuator commands; Global Positioning System (GPS) data and / or Geographic Information System (GIS) data such as position, speed, and altitude; internally generated information such as restricted speed limits for the asset considering its current location; video and image information from cameras located at various locations within, on, or near the asset; audio information from microphones located at various locations within, on, or near the asset; information regarding the asset's operational plan transmitted from the data center to the asset, such as route, schedule, and cargo manifest information; information regarding environmental conditions, including current and predicted weather conditions, for the area in which the asset is currently operating or is planned to operate; asset control status and operational data generated by systems such as positive train control (PTC) in locomotives; and additional data, video, and audio analysis and analysis.

[0086] Figure 19 illustrates a field implementation of a seventh embodiment of an exemplary real-time data acquisition and recording system (DARS) 1000 and an automatic signal monitoring and alerting system 1080 in which aspects of the present disclosure may be implemented. The DARS 1000 is a system that delivers real-time information from a data recording device to a remotely located end user. The DARS 1000 includes a data recorder 1054 that is installed on a vehicle or mobile asset 1048 and communicates with any number of different information sources through any combination of onboard wired and / or wireless data links 1070 such as a wireless gateway / router, or communicates with non-onboard information sources via a data link such as a wireless data link 1046 through the DARS 1000 data center 1050. The data recorder 1054 comprises an onboard data manager 1020, a data encoder 1022, a vehicle event detector 1056, a queuing repository 1058, and a wireless gateway / router 1072. Furthermore, in this implementation, the data recorder 1054 may include a crash-hardened memory module 1018 and / or an Ethernet switch 1062 with or without Power over Ethernet (PoE). An exemplary hardened memory module 1018 may be, for example, a crash-suitable event recorder memory module compliant with the Code of Federal Regulations and / or the Federal Railroad Administration Regulations, a crash-survivable memory unit compliant with the Code of Federal Regulations and / or the Federal Aviation Administration Regulations, a crash-hardened memory module compliant with any applicable Code of Federal Regulations, or any other suitable hardened memory device known in the Art. In the eighth embodiment, the data recorder may further include an optional non-crash-hardened removable storage device (not shown).

[0087] The wired and / or wireless data link 1070 may include one or a combination of discrete signal input, standard or proprietary Ethernet, serial connection, and wireless connection. Ethernet-connected devices may utilize the Ethernet switch 1062 of the data recorder 1054 and may utilize PoE. The Ethernet switch 1062 may be internal or external and may support PoE. Furthermore, data from remote data sources such as the map component 1064, route / crew manifest component 1024, and weather component 1026 in the implementation configuration of Figure 19 is available from the data center 1050 to the onboard data manager 1020 and vehicle event detector 1056 via the wireless data link 1046 and wireless gateway / router 1072.

[0088] The data recorder 1054 collects data or information from a wide variety of sources, which can vary widely based on the asset configuration, via the onboard data link 1070. The data encoder 1022 encodes at least a minimum set of data as typically defined by regulatory bodies. In this implementation, the data encoder 1022 receives data from a wide variety of asset sources 1048 and data center sources 1050. The information source may include any number of components within asset 1048, such as analog input 1002, digital input 1004, I / O module 1006, vehicle controller 1008, engine controller 1010, inertial sensor 1012, Global Positioning System (GPS) 1014, camera 1016, positive train control (PTC) / signal data 1066, fuel data 1068, cellular transmit detector (not shown), internal drive data, and any additional data signals, as well as any number of components within data center 1050, such as route / crew manifest component 1024, weather component 1026, map component 1064, and any additional data signals. Furthermore, the asset 1048 information source can be connected to data recorder 1054 via any combination of wired or wireless data links 1070. The data encoder 1022 compresses or encodes and time-synchronizes the data to facilitate efficient real-time transmission and replication to the remote data repository 1030. The data encoder 1022 transmits the encoded data to the onboard data manager 1020, which then stores the encoded data in the crash-hardened memory module 1018 and the queuing repository 1058 for replication to the remote data repository 1030 via the remote data manager 1032 located in the data center 1050. Optionally, the onboard data manager 1020 may store a tertiary copy of the encoded data in a non-crash-hardened removable storage device according to the eighth embodiment. The onboard data manager 1020 and the remote data manager 1032 operate in harmony to manage the data replication process.A single remote data manager 1032 within data center 1050 can manage the replication of data from multiple assets 1048.

[0089] Data from various input components and data from the in-cab audio / graphical user interface (GUI) 1060 are transmitted to the vehicle event detector 1056. The vehicle event detector 1056 processes the data to determine whether an event, incident, or other predefined situation involving asset 1048 has occurred. When the vehicle event detector 1056 detects a signal indicating that a predefined event has occurred, it transmits the processed data indicating the occurrence of the predefined event, along with supporting data surrounding the predefined event, to the onboard data manager 1020. The vehicle event detector 1056 detects events based on data from a wide variety of sources, including analog input 1002, digital input 1004, I / O module 1006, vehicle controller 1008, engine controller 1010, inertial sensor 1012, GPS 1014, camera 1016, route / crew manifest component 1024, weather component 1026, map component 1064, PTC / signal data 1066, and fuel data 1068, which may change based on the asset configuration. When the vehicle event detector 1056 detects an event, the detected asset event information is stored in the queuing repository 1058 and can optionally be presented to the crew of asset 1048 via the in-cab audio / graphical user interface (GUI) 1060.

[0090] When the location of asset 1048 indicates that signal 1082 has been crossed, excessive braking has occurred, and asset 1048 has stopped within close proximity of signal 1082, or that a speed limit has been applied by the signal pattern, the onboard data manager 1020 initiates outward camera image analysis to determine the meaning or pattern of signal 1082, as shown in Figure 20. Using state-of-the-art image processing techniques, the outward camera image can be analyzed by a previously trained neural network or artificial intelligence component to decode the effects of the signal pattern and operating rules. In this exemplary implementation, the analysis and / or processing by the neural network or artificial intelligence component is performed in the back office. In another embodiment, the analysis and / or processing by the neural network or artificial intelligence component is performed on asset 1048. The output of the signal pattern decoding is combined with other sensor data to determine whether asset 1048 is significantly violating the signal indication by occupying the rail track (which could lead to a train-to-train collision in this exemplary implementation) or operating in an unsafe manner to achieve signal compliance. If asset 1048 is found to be non-compliant, an electronic alert is stored in the back office and, after the railway business rules are associated with the signaling and asset operation, is delivered to users who have subscribed to receive such alerts. These alerts can then be mined directly via the database or by using a website graphical user interface or web client 1042 provided to the user.

[0091] Furthermore, an audible alert can be added to the driver's cab of asset 1048, which will alert the driver of an imminent signal violation, an impending bad situation in which the driver may respond more quickly if the driver is distracted or otherwise not paying attention to a truck obstruction, stop signal, and / or asset 1048 is speeding in a zone where the signal requires a lower speed limit.

[0092] The automated signal monitoring and alerting system 1080 is also enhanced to automatically perform video analysis to determine the meaning of a signal whenever a monitored asset crosses a signal, to determine the meaning of a signal whenever an asset is subjected to excessive braking force and stops within a predefined distance, and to monitor asset speed to determine whether the asset is moving at a speed faster than the approved speed as determined by the signal pattern. The video analysis is performed on asset 1048 to reduce the delay between the actual event and electronic notifications to the user and / or subscribers. The functionality of the automated signal monitoring and alerting system 1080 is enhanced to enable automatic inbound and outbound video downloads upon alert to enhance the user experience and reduce the work required to investigate events. The functionality of the automated signal monitoring and alerting system 1080 is also enhanced to provide real-time audible cues within non-compliant asset 1048 to alert crew members in cases of distraction or other reasons for not following safe operating practices with respect to signal rules and meanings.

[0093] In addition, the automatic signal monitoring and alerting system 1080 and / or the video analysis system 910 may receive location information from the asset owner, including the latitude and longitude coordinates of signals such as stop signals, traffic signals, speed limit signals, and / or object signals near trucks. The video analysis system 910 then determines whether the location information received from the asset owner is correct. If the location information is correct, the video analysis system 910 stores the information and does not recheck the location information for a predetermined period of time, such as checking the location information on a monthly basis. If the location information is incorrect, the video analysis system 910 determines the correct location information, reports the correct location information to the asset owner, stores the location information, and does not recheck the location information for a predetermined period of time, such as checking the location information on a monthly basis. Storing location information provides easier detection of signals such as stop signals, traffic signals, speed limit signals, and / or object signals near trucks.

[0094] The onboard data manager 1020 also transmits data to the queuing repository 1058. In near real-time mode, the onboard data manager 1020 stores the encoded data and any event information received from the data encoder 1022 in the crash-hardened memory module 1018 and the queuing repository 1058. In the eighth embodiment, the onboard data manager 1020 may optionally store the encoded data in a non-crash-hardened removable storage device. After five minutes of encoded data have accumulated in the queuing repository 1058, the onboard data manager 1020 stores the five minutes of encoded data in the remote data repository 1030 via a remote data manager 1032 in the data center 1050, through a wireless data link 1046 accessed via a wireless gateway / router 1072. In real-time mode, the onboard data manager 1020 stores encoded data and arbitrary event information received from the data encoder 1022 in a crash-hardened memory module 1018, optionally in a non-crash-hardened removable storage device of the eighth embodiment, and in a remote data repository 1030 via a remote data manager 1032 in the data center 1050, accessed through a wireless data link 1046 accessed via a wireless gateway / router 1072. The process of replicating data to the remote data repository 1030 requires a wireless data connection between asset 1048 and the data center 1050. The onboard data manager 1020 and the remote data manager 1032 can communicate via various wireless communication links such as Wi-Fi, cellular, satellite, and private wireless systems utilizing the wireless gateway / router 1072. The wireless data link 1046 could be, for example, a wireless local area network (WLAN), a wireless metropolitan area network (WMAN), a wireless wide area network (WWAN), a private wireless system, a mobile phone network, or any other means of transferring data from the DARS1000's data recorder 1054 to the DARS1000's remote data manager 1030 in this example.If a wireless data connection is unavailable, the data is stored in memory and queued in queuing repository 1058 until wireless connectivity is restored and the data replication process can resume.

[0095] In parallel with data recording, the data recorder 1054 continuously and autonomously replicates data to the remote data repository 1030. The replication process has two modes: real-time mode and near real-time mode. In real-time mode, data is replicated to the remote data repository 1030 every second. In near real-time mode, data is replicated to the remote data repository 1030 every 5 minutes. The rates used for near real-time mode and real-time mode are configurable, and the rate used for real-time mode can be adjusted to support high-resolution data by replicating data to the remote data repository 1030 every 0.10 seconds. When DARS 1000 is in near real-time mode, the onboard data manager 1020 queues data to the queuing repository 1058 before replicating the data to the remote data manager 1032. The onboard data manager 1020 also replicates the vehicle event detector information queued in the queuing repository 1058 to the remote data manager 1032. Under most conditions, near real-time mode is used during normal operation to improve the efficiency of the data replication process.

[0096] Real-time mode can be initiated based on an event that has occurred and been detected by the vehicle event detector 1056 installed on asset 1048, or by a request initiated from data center 1050. A typical real-time mode request initiated by data center 1050 is initiated when a remotely located user 1052 requests real-time information from web client 1042. Typical reasons for real-time mode occurring on asset 1048 are the detection of an event or incident by the vehicle event detector 1056, such as an operator initiating an emergency stop request, emergency braking activity, rapid acceleration or deceleration on any axle, or loss of input power to data recorder 1054. When transitioning from near real-time mode to real-time mode, all data that has not yet been replicated to the remote data repository 1030 is replicated and stored in the remote data repository 1030, and then live replication is initiated. The transition between near real-time mode and real-time mode typically occurs in less than 5 seconds. After a predetermined time has elapsed since an event or incident, after a predetermined period of inactivity, or when user 1052 no longer desires real-time information from asset 1048, data recorder 1054 returns to near real-time mode. A predetermined time required to initiate the transition is configurable and is typically set to 10 minutes.

[0097] When the data recorder 1054 is in real-time mode, the onboard data manager 1020 attempts to continuously empty its queue to the remote data manager 1032, store the data in the crash-hardened memory module 1018, optionally in the non-crash-hardened removable storage device of the eighth embodiment, and simultaneously transmit the data to the remote data manager 1032. The onboard data manager 1020 also transmits detected vehicle information queued in the queuing repository 1058 to the remote data manager 1032.

[0098] Upon receiving data replicated from the data recorder 1054, along with data from the map component 1064, route / crew manifest component 1024, and weather component 1026, the remote data manager 1032 stores the compressed data in the remote data repository 1030 within the DARS 1000 data center 1050. The remote data repository 1030 may be, for example, cloud-based data storage or any other suitable remote data storage. Once the data is received, the process is initiated to have the data decoder 1036 decode the recently replicated data to and from the remote data repository 1030 and send the decoded data to the remote event detector 1034. The remote data manager 1032 stores vehicle event information in the remote data repository 1030. When the remote event detector 1034 receives the encoded data, it processes the decoded data to determine whether an event of interest is found within the decoded data. Next, the encoded information is used by the remote event detector 1034 to detect events, incidents, or other predetermined conditions in the data occurring in asset 1048. When the remote event detector 1034 detects an event of interest from the encoded data, it stores the event information and supporting data in the remote data repository 1030. When the remote data manager 1032 receives the information from the remote event detector 1034, it stores that information in the remote data repository 1030.

[0099] A remotely located user 1052 can access information, including vehicle event detector information, related to a specific asset 1048 or multiple assets, using a standard web client 1042 such as a web browser, or, in this implementation, a virtual reality device (not shown) capable of displaying thumbnail images from a selected camera. The web client 1042 communicates the user 1052's information request to the web server 1040 via the network 1044 using common web standards, protocols, and techniques. The network 1044 could be, for example, the internet. The network 1044 could also be a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a virtual private network (VPN), a mobile phone network, or any other means of transferring data from the web server 1040 to the web client 1042 in this example. The web server 1040 requests the desired data from the data decoder 1036. In response to the request from the web server 1040, the data decoder 1036 obtains the requested data related to the specific asset 1048 or multiple assets from the remote data repository 1030. The data decoder 1036 decodes the requested data and sends the decoded data to the localizer 1038. Localization is the process of converting data into a format desired by the end user, for example, converting the data to the user's preferred language and units of measurement. The localizer 1038 identifies the profile settings set by user 1052 by accessing the web client 1042, and uses the profile settings to prepare the information being sent to the web client 1042 for presentation to user 1052, since the raw encoded data and detected event information are stored in the remote data repository 1030 using coordinated Coordinated Universal Time (UTC) and the International System of Units (SI units). The localizer 1038 converts the decoded data into a format desired by user 1052, such as user 1052's preferred language and units of measurement.The localizer 1038, upon request, sends the localized data to the web server 1040 in the user 1052's preferred format. The web server 1040 then sends the localized data for one or more assets to the web client 1042 for viewing and analysis, and provides playback and real-time display of standard video and 360-degree video. The web client 1042 can display, and the user 1052 can view data, video, and audio related to a single asset, or view data, video, and audio related to multiple assets simultaneously. The web client 1042 can also provide synchronized playback and real-time display of data along with multiple video and audio data from image measurement sources, standard video sources, 360-degree video sources, and / or other video sources, as well as / or distance measurement sources, located on, within, or near the asset, nearby assets, and / or remotely located sites.

[0100] Figure 21 is a flowchart showing a first illustrated embodiment of a process 1100 for determining signal compliance, according to one implementation of the present disclosure. After the DARS 1000 and camera 1016 are installed and connected to various sensors on asset 1048 such as analog input 1002, digital input 1004, I / O module 1006, vehicle controller 1008, engine controller 1010, inertial sensor 1012, Global Positioning System (GPS) 1014, camera 1016, positive train control (PTC) / signal data 1066, fuel data 1068, cellular transmit detector (not shown), internal drive data and any additional data signals, 1102, onboard data and / or event start video and / or still images from the various sensors are transmitted to the back office data center 1074 every 5 minutes, and camera images are loaded and stored on asset 1048 in a capacity exceeding 72 hours. The back office data center 1074 service continuously scans the data for trigger conditions. If the episode business logic trigger condition is not met, the workflow is canceled and the episode event is not logged. If asset 1048 has moved through track signal 1082 as referenced by the latitude and longitude coordinates of all signals stored in back office data center 1074, and / or if asset 1048 has stopped within a certain distance before signal 1082 and is using excessive braking force to stop before passing through and crossing signal 1082, the back office data center 1074 service scans the data to determine whether the train vehicle is in the leading, control, or first position of train asset 1048 in this illustrated embodiment. The back office data center 1074 uses a first artificial intelligence model to determine whether the train vehicle is in the leading, control, or first position within train asset 1048. If the train car is not in the leading, control, or first position within train asset 1048, the episode business logic trigger condition is not met, the workflow is canceled, and no episode event is logged 1108.If a train car is in the leading, control, or first position within train asset 1048, the back office data center 1074 requests video content 1118 from the leading, control, or first position locomotive, filmed for a short period before crossing signal 1082 and / or while asset 1048 is stopped. The retrieved video content is passed to and / or stored in the back office data center 1074 and passed along a second artificial intelligence model, which scans the video content to determine the signal 1082 configuration, such as the combination of colors of each signal lamp, and determines whether signal 1082 indicates a stop. The back office data center 1074 determines whether the signal 1082 configuration indicates that asset 1048 must stop and cannot pass signal 1082. If the signal 1082 pattern does not indicate that asset 1048 must stop and cannot pass signal 1082, the episode business logic trigger condition is not met, the workflow is canceled, and episode event 1108 is not logged. If asset 1048 must stop and cannot pass signal 1082, and the signal 1082 pattern indicates that a stop signal exists, the episode is triggered, stored in the back office data center 1074 database, and email 1124 is sent to previously selected users to be notified when such conditions exist.

[0101] For the sake of simplicity, process 1100 is shown and described as a series of steps. However, the steps according to this disclosure can be carried out in various orders and / or simultaneously. In addition, the steps according to this disclosure may be carried out in conjunction with other steps not shown and described herein. Furthermore, not all illustrated steps are required to carry out the method according to the disclosed subject matter.

[0102] Train engineers operating certain classes of mobile assets are required by federal regulations in many countries to undergo tests to verify their skills and abilities, and are recertified upon passing these tests as part of regulatory compliance for the geographical location in which the engineer operates. An example of this skills performance assessment in the United States is 49 CFR §240.127, which stipulates tests by the Federal Railroad Administration (FRA) for railroads operating on U.S. tracks. The stated purpose of the regulation is "to ensure that only qualified persons operate locomotives or trains." The regulation also sets minimum federal safety standards for the qualification, training, testing, certification, and monitoring of all locomotive engineers to which it applies. Railroads may issue certifications to train service engineers, locomotive service engineers, and student engineers.

[0103] As stated in 49 CFR §240, railways required to meet these standards must conduct carefully defined evaluations and various monitoring of engineers' train operation performance on an annual, quartan, and periodic (audit) basis. Currently, there are three methods typically used by railways to conduct engineer performance evaluations. The first is for the evaluator to board the locomotive alongside the crew members under performance skills evaluation and to ride along a designated train route. This method is labor-intensive and requires that a designated supervisor of the locomotive engineer be physically present in the locomotive cab throughout the entire train movement segment being monitored. The engineer being evaluated is also informed that he or she is being actively evaluated so that he or she can adjust the operation of the mobile asset to avoid errors.

[0104] The second evaluation method uses a train simulator, which works to reproduce the visual, audible, and sometimes even more physical characteristics of train operator behavior in response to physical inputs and train characteristics. However, this method does not provide an evaluation of actual tracks over a given distance.

[0105] A third method used to perform skill performance assessments involves acquiring some or all of the locomotive event recorder data, including, but not limited to, video image data from inward and outward cameras, external and internal audio, accelerometer and gyrometer data, fuel and weather data, train formation data, route information, and the movement rights of monitored passengers captured across specific train routes. Analysis is performed in real time, after the operation has taken place, or a combination of both. This third method requires less time and effort, has been proven to improve the accuracy of the assessment, and can be performed remotely.

[0106] Locomotive and train-based simulators have created the ability to perform recertification in an environment that limits physical risks and enhances safety while engineers are being evaluated for performance. However, no automated systems or platforms are known that have been developed to reduce the time required to read and assimilate relevant segments of data in an efficient and simple manner. The Engineer Recertification Assistant of this Disclosure requires little more than prior knowledge of key geographical locations to read data on combinations of critical train operation signals that can be monitored to automatically indicate insufficient train performance, start / end times, and the locomotive of interest. Users of the Engineer Recertification Assistant simply press a button, and hours of manual work are automated and presented in a highly efficient format on a secure web-based portal and / or platform.

[0107] The improved engineer evaluation assistant described herein is an enhanced version of the third method described above, providing a more efficient and faster way to perform the activities required for engineer evaluation in a unified user experience across the entire desired train route. The engineer recertification assistant of this disclosure is an integrated online tool that significantly improves the engineer evaluation process by streamlining the activities required for evaluation into a unified user experience and increasing the productivity and accuracy of the engineer evaluation process. The engineer recertification assistant also provides a unified experience for engineer evaluation by providing two-way integration between the railway engineer evaluation portal and applicants. The engineer recertification assistant of this disclosure improved data collection by 10%, data organization by 25%, report generation by 30%, data analysis by 35%, and data analysis by over 50%, as shown in Figure 24.

[0108] The engineer evaluation methods and systems described here can be used in several ways to improve the efficiency of performance evaluation and engineer recertification.

[0109] Firstly, after determining the correct boarding for crew evaluation (using locomotive ID, train ID, date timestamp, subdivision basis, or a combination thereof), railway personnel can simply press a button while logged into a secure portal, and the method and system can be used to automatically return video data from both inbound and outbound cameras for a range of scenarios related to the operation of locomotives, trains, and along-railway assets. Examples of scenarios are listed below. The ability to automate the video and event recorder capture process for train performance characteristics, the geographical location of areas of interest, and specific operating areas of interest can save a considerable amount of time and effort that is usually spent manually determining start and end times to request and retrieve video data. A further advantage of this disclosure is the ability to coordinate time-synchronized event recorder and geographic location data with video footage, enabling a comprehensive view of the locomotive cab and surroundings during critical evaluation periods.

[0110] Some examples of useful timeframes for analyzing engineer performance include: a. When a train passes near a trackside signal, especially one that is in a less-than-clear form (something other than "all clear to proceed"), b. Zones with temporary speed restrictions that are not otherwise indicated by trackside signaling and that require evaluation for safety-critical behaviors. These zones can be activated to enhance safety around work sites, such as for drivers performing nearby track maintenance on adjacent tracks. c. Level crossings where pedestrian and vehicle traffic is present. d. Interference and other excessive train forces typically found in stations and within station premises that may exhibit unsafe behavior. e. Braking scenarios that are both operational and safety-oriented, and f. Train operation behavior that results in excessive or unsafe forces.

[0111] Secondly, while railway staff evaluate engineers' performance, they use the same secure web portal platform to perform various tasks related to reporting on engineer performance. Examples of available functions include: a. Create online notebooks to capture annotations and comments regarding engineers' performance. b. Sharing the entire ride, including comments, with other officers, c. Summarize the skills performance ride and results in a reporting format used for regulatory submission or performance discussions with engineers.

[0112] Thirdly, in addition to data collection and reporting, the methods and systems described in this disclosure create additional checks to monitor engineer performance for any exceptions by comparing their performance to test criteria defined in regulatory compliance documentation. An example of rail compliance documentation is FRA 49 CFR §240.127. This replaces the need for personnel to manually scan information (data, video, audio, etc.) from the entire ride to find these exceptions. Instead, algorithms are used to automatically identify these exceptions in the form of “events” and present them via a web portal as follows: a. Leveraging a range of business algorithms and / or rules (from algorithms and linear heuristic models to advanced machine learning models) to monitor performance, create real-time events for checking these, and identify exceptions. b. Integrate with additional data sources as needed to collect input for developing these algorithms. Examples include train control event logs and / or train dispatch management system logs. c. These results will be sent as real-time alerts to your email inbox or as text alerts to your browser. Furthermore, these exceptions will be summarized in a report, providing customers with integrated results to edit, review, and share with other rail users.

[0113] Real-time events are presented to railway personnel to evaluate engineer performance. An example of such an event is a train overspeed event, which identifies when an engineer is operating a train beyond the approved track speed, thereby violating standards for train handling. Railway personnel can review these events to determine whether the engineer's performance was satisfactory or unsatisfactory. Other indicators and icons include geotagging of trackside assets such as signals and level crossings.

[0114] Furthermore, the results of real-time events are converted into satisfactory or unsatisfactory scores for engineer performance using a combination of artificial intelligence (AI) and other algorithmic techniques. The system includes the ability to use algorithms to recommend authentication or deauthentication, and to become a fully automated system that deauthenticates any detected overall non-compliance by AI.

[0115] The disclosed methods and systems offer, among other advantages, the following: a. Push-button readout of dozens or hundreds of inward and outward camera videos with a predetermined duration. b. Easy and efficient grouping and visualization of key videos associated with engineer recertification train segments. c. Clear identification of critical locations along the train route, engineer recertification of critical train operation characteristics associated with the train route. d. The ability to capture critical train operation events and operational performance by identifying critical times for analyzing and reporting engineer performance using time-synchronized machine learning and event recorder signal analysis. For example, machine learning is used to detect when a mobile phone is used in the driver's cab, and then the event data recorder is used to filter the results of the machine learning model to show only locomotives of interest to the railway, such as locomotives that were moving and / or locomotives that were in the leading position when the mobile phone was used. The purpose of the machine learning model is to provide image classification and object detection results. The purpose of the event data recorder signals is to filter those results only if they are relevant to the railway's safety plan and / or operating rules. e. Ability to perform various tasks related to reporting on engineer performance using a web portal platform.

[0116] Figures 22 and 23 include several illustrative screenshots illustrating some of the concepts described above. Figure 22 shows that an engineer re-authentication button is added to an existing page within the secure web portal. Figure 23 shows an existing page enhanced with predefined events for engineer re-authentication, such as signal crossings. Recorded videos are also displayed on this page for easy reading and viewing. Indicators and icons, such as geotagging of railway assets like signals, crossings, and speed zones, are displayed in the DARS viewer.

[0117] The Engineer Recertification Assistant in this Disclosure includes a system and methods aimed at successfully conducting engineer assessments remotely by reducing the administrative time spent collecting and assembling information to successfully manage annual, quartan, or skill performance audits in accordance with FRA 49 CFR §240.127. As shown in Figure 24, the Engineer Recertification Assistant controls costs by successfully conducting several engineer assessments remotely, thereby enabling RFEs to better identify engineers at risk, giving RFEs more time to focus on correcting the behavior of engineers at risk, providing post-mortem monitoring and / or on-boarding for engineers at risk, increasing the number of field certifications that meet 49 CFR §240.127, and driving higher levels of safety.

[0118] Referring to Figure 25, the target process 1300 of the first illustrated embodiment of the process performed by the Engineer Recertification Assistant 1320 of this Disclosure includes five steps performed by the RFE and applicant characteristics to enable the corresponding steps of the RFE. The Engineer Recertification Assistant 1320 is an artificial intelligence (AI) implementation that utilizes both video and motion data using real-time data acquisition and recording systems such as DARS 100, 200, 800, 900, and 1000, analyzes the video and motion data using video content analysis systems such as video analysis system 910, and reports the video and motion data on a web-based viewer such as web client 826. The Engineer Recertification Assistant 1320 then combines this data with train and crew data to enable the railway company to quickly assess information that would lead to the certification or decertification of a crew member operating a train in a given region and route. The data collected and integrated by the Engineer Recertification Assistant 1320 allows a human to then allow the AI ​​to examine selected events and points along the route to assess potential inappropriate and / or unsafe violations of train scanning and operation rules. Alternatively, the AI ​​itself can review selected events and points along the route to determine potential inappropriate and / or unsafe violations of train operation and operation rules, determine an evaluation score, recommend certification or decertification of an engineer or crew member, or directly certify or decertify an engineer or crew member for any overall non-compliance detected.

[0119] The RFE initiates the evaluation process by selecting engineer 1302 to audit. In response, system 1320 provides the customer with a simple user interface to search for all train rides completed by that engineer within the last 12 months. As shown in Figure 26, the RFE can perform an on-demand audit by selecting an engineer and time range to verify that all engineers are listed on the train operation summary page or that the customer can define an audit schedule. The user interface 826 then displays results 1304 for that engineer over the last 12 months, including the trains and subdibs that the engineer operated. As shown in Figure 27, system 1320 automatically downloads videos of events of interest, including but not limited to trackside signals, temporary speed zones, level crossings, PTC initialization, yard entry and / or exit, alerts and / or train operation exceptions, such as physical interference, throttle adjustments, sudden braking, and mobile phone downloads. For example, the user interface 826 in Figure 27 displays an automatically downloaded video that includes 1) a 120-second video before the trackside signal and a 30-second video after the trackside signal, 2) a thumbnail showing the trackside signal when the train is passing, and 3) the trackside signal icon in the DARS view.

[0120] Next, the RFE selects a train / sub for the engineer audit 1306. System 1320 provides the capability to automatically download a 60-mile / 2-hour ride and six additional episodes to detect exceptions to the customer's engineer evaluation report (EER) rules. For example, as shown in Figure 28, the customer's EER rules include nine sections, and the Engineer Recertification Assistant 1320 of this disclosure covers eight of those sections. System 1320 then generates a completion email 1308 to the RFE for video and / or exceptions. The RFE and / or customer can review the audit results in the DARS viewer 826 and / or operator scorecard, which also allows the RFE to directly add notes regarding engineer exceptions to the DARS viewer 826, and the operator scorecard 1310 documents all exceptions. As shown in Figure 29, System 1320 is integrated into the Engineer Evaluation System or ERAD, which is the website where the customer performs engineer evaluations today, and includes one-click approval or rejection success evaluation. The Engineer Evaluation System 1320 includes several features, including, but not limited to, 1) the ability to right-click a purple bar to add comments about a performance evaluation ride, and to add comments about exceptions, such as, for example, "On MP433.42, the engineer observed not following sterile cab rules XX.XX while the train was under restricted speed limits"; 2) an icon appears to indicate that a comment has been made for the Engineer and / or RFE review; 3) the ability for the user to toggle comments on or off, as can also be done for episodes; 4) the ability to combine all comments about the ride into an operator scorecard document with a shared link, so that the user does not need to take screenshots of the DARS viewer 826; and 5) the ability to summarize ride comments and / or exceptions into a report to provide a one-stop shop for customers to edit, review, and share with other users.As shown in Figure 30, the operator scorecard document compiles all alerts, RFE comments, edited scores, and supporting data for the EER. The operator scorecard document can replace the EER if the EER is in the correct format and / or structure. Process 1300 is then repeated as needed.

[0121] Screenshots of the DARS viewer 826 from a live demonstration of the Engineer Recertification Assistant 1320 in this disclosure are shown in Figures 31 and 32. The screenshot in Figure 31 shows 1) videos automatically downloaded 2 minutes and 30 seconds before and after the trackside signal, 2) thumbnails showing the trackside signal as the train passes, and 3) the trackside signal icon. The screenshot in Figure 32 shows the RFE user determining the assets and time range for which they want to evaluate the engineer on the DVR video download page.

[0122] Figure 33 is a flowchart showing a second illustrated embodiment of process 1400 performed by the Engineer Recertification Assistant 1320 of this Disclosure. As described above with respect to process 1300, the Engineer Recertification Assistant 1320 is an artificial intelligence (AI) implementation that utilizes both video and motion data using real-time data acquisition and recording systems such as DARS 100, 200, 800, 900, and 1000, analyzes the video and motion data using video content analysis systems such as video analysis system 910, and reports the video and motion data on a web-based viewer such as web client 826. Process 1400 includes three workstreams when using computer-based data and video to authenticate engineers: a data acquisition and compilation workstream 1402, a data analysis workstream 1404, and a summary and conclusion reporting workstream 1406. The collection workstream 1402 identifies mobile assets using at least one camera, such as cameras 116, 216, 802, 940, and 1016, as well as at least one onboard data recorder, such as data recorders 154, 254, 808, 902, and 1054, which are installed on and connected to various sensors 1408 as described above, such as GPS, speed, and acceleration. The collection workstream 1402 may also obtain data from additional data sources, such as PTC event logs and / or network dispatch systems, to collect inputs for monitoring performance. The data collected from these mobile assets may include DARS data 1410, which includes event data recorder data, accelerometer data, gyroscope data, fuel amount data, microphone and in- and / or out-of-facing camera data that is sent to and stored in the back office, and microphone and in- and / or out-of-facing camera data that is onboard and stored in the mobile asset with a capacity of at least 72 hours. Data collected from external data sources is integrated into platform 1412 to enable additional monitoring of crew performance, compared with train operating rules, track access restrictions, weather conditions, and other factors.

[0123] The analysis workstream 1404 continuously and / or upon request includes back-office services and scans DARS data and camera data for critical events and regulatory requirement-based operational performance 1414. The user interface secure portal 826 allows the user to initiate the analysis by requesting an engineer re-authentication requirement 1416. For the determined geographical segment, the analysis workstream 1404 performs an analysis of all operational, performance, and behavioral characteristics 1418 related to specified government regulatory requirements for authentication and / or de-authentication of mobile asset engineers and / or operators at specified dates and times for a given crew.

[0124] The summary and conclusion workstream 1406 includes a user interface secure portal 826 that displays relevant information and results in a single view 1420, which may include critical geographical operating zones, critical operating areas such as work zones, algorithm-based regulation-based alerts, and AI-based regulation-based alerts. The user interface secure portal 826 allows users to add comments to specific events and / or periods for others 1422 to review. The summary and conclusion workstream 1406 provides the ability to issue summarized reports of crew operational performance 1424 for review and record-keeping. In some cases, skill performance assessments provide automated score-based recommendations for operator certification and / or decertification 1426.

[0125] For the sake of simplicity, processes 1300 and 1400 are presented and described as a series of steps. However, the steps according to this disclosure can be performed in various orders and / or simultaneously. In addition, the steps according to this disclosure may be performed in conjunction with other steps not presented and described herein. Furthermore, not all illustrated steps are required to carry out the method according to the disclosed subject matter.

[0126] An accelerometer-based mobile asset data recorder and transmitter of an embodiment of the present invention used on a locomotive involves the operational integration of nine components. The components are an event recorder similar to a black box on an airplane, a locomotive digital video recorder, a fuel level sensor, fuel level sensor software, a wireless processing unit, an inertial navigation sensor board, firmware, system software, and a system encompassing these components. The inertial navigation sensor board includes a 3-axis digital gyroscope, a 3-axis digital magnetometer, a 3-axis digital accelerometer, and a microcontroller. The gyroscope is used to measure the angular acceleration and angular deceleration of the asset, the magnetometer is used to measure the magnetic field, the accelerometer is used to measure linear acceleration and linear deceleration, and the microcontroller is used to process the data and communicate between the sensors and the wireless processing unit.

[0127] The mobile asset data recorder and transmitter perform seven functions: automatic orientation, automatic compass calibration, fuel compensation based on pitch and roll, emergency braking based on impact detection, detection of rough operating conditions, engine operation detection, and inertial navigation (dead reckoning).

[0128] Automatic collision detection can alert appropriate personnel when emergency braking is applied and immediately determine whether the collision coincides with a braking event. Mobile asset data recorders and transmitters provide immediate notification of collision severity, including display of locomotive derailment or overturning events.

[0129] Rough operating condition detection reduces losses due to rough switches and train operations. It provides alerts and summary reports when high-energy shocks are detected during switch operations. It also detects excessive slack action, allowing supervisors to continuously assess and improve train operations. This reduces cargo and equipment damage by identifying unsafe trends and enabling users to take immediate corrective action. Continuous monitoring of track conditions and on-road monitoring of vibration levels alerts track maintenance personnel to the exact location of rough tracks or switches that may require inspection and repair.

[0130] Accelerometer-based engine operation detection may be used as a backup source to reduce fuel costs by eliminating excessive idle when the engine operation signal is not yet accessible from other onboard systems. It also improves road fuel accuracy by compensating for locomotive tilt due to gradients and high altitudes.

[0131] Fuel compensation based on pitch and roll improves fuel reporting accuracy. This invention provides a simple, universal, and non-intrusive method for determining whether an engine is running while a locomotive is stopped. Improved accuracy provides enhanced real-time business intelligence to support strategic initiatives such as smart fuel delivery, combustion rate analysis, fuel adjustment, and emissions monitoring.

[0132] Inertial navigation, or dead reckoning, enhances positioning accuracy. It extends the high-precision differential GPS of the radio processing unit with sophisticated dead reckoning when inside store buildings, stations, tunnels, or any location where GPS signals are unavailable. This provides highly accurate station arrival and departure times, and precise positioning and locomotive orientation within store areas increases operational efficiency by improving store planning and workflow.

[0133] The mobile asset data recorder and transmitter system and its components of the present invention are shown in Figure 39. The mobile asset data recorder and transmitter system 1200 consists of 10 interrelated components: an event data recorder 1238, a locomotive digital video recorder (DVR) 1252, a fuel level sensor 1210, fuel level sensor software 1212, a WPU 1202, an inertial navigation sensor board 1214, a Global Positioning System (GPS) 1206, firmware 1224, system software 1226, and the system 1200 itself. Installing the WPU 1202 on an asset such as a locomotive consists of mounting the WPU 1202 and connecting it externally to the event data recorder 1238, the locomotive digital video recorder 1252, and any additional available condition sensing devices.

[0134] The event data recorder 1238 is an onboard data logging device for the locomotive, similar to a black box on an airplane. A typical event data recorder 1238 consists of digital and analog inputs, as well as pressure switches and pressure transducers that record data from various onboard devices such as throttle position, wheel speed, and emergency brake application. The WPU 1202 receives and processes data from the event data recorder 1238 once per second via an external serial connection.

[0135] The locomotive's digital video recorder (DVR) 1252 is an onboard video recording device, similar to a television DVR. The DVR 1252 is equipped with a forward-facing camera and microphone. The camera is mounted in an orientation that records what the engineer is seeing. The WPU 1202 accesses the locomotive's DVR 1252 via an external Ethernet connection to download video from the hard drive before, during, and after events.

[0136] The fuel level sensor 1210 is a sensor used to measure the amount of fuel inside the fuel tank. The fuel level sensor 1210 used in this invention is an ultrasonic level sensor that uses ultrasonic sound waves to determine the distance between the sensor head and the fuel level. The sensor 1210 is mounted on top of the fuel tank with known dimensions and mounting location. The WPU 1202 accesses this data via an external serial connection.

[0137] The fuel level sensor software 1212 takes the distance from the fuel level to the sensor 1210, which has the fuel tank geometry, and converts this data into a steady-state fuel amount. This is achieved by applying mathematical filtering to reduce noise from tank sloshing and ultrasonic behavior. Software 1226 also uses a smart algorithm to determine refueling and fuel droplet events.

[0138] The illustrated embodiment of the WPU1202 is a highly rugged onboard computer running Windows XP, embedded particularly for industrial applications. It has many different features that can be installed to customize the product to meet specific customer needs. The WPU1202 has the ability to communicate with a wide variety of onboard systems, including but not limited to vehicle control systems, event data recorders, DVRs, fuel level sensors, and engine controllers. The WPU1202 has the ability to communicate via a wide variety of protocols, including but not limited to RS232, RS422, RS485, CAN bus, LAN, WiFi, cellular, and satellite.

[0139] The inertial navigation sensor board (board) 1214 is a hardware upgrade for the WPU1202. It is installed internally and communicates with the WPU1202 via an internal serial port. Board 1214 consists of four components: a 3-axis gyroscope 1216, a 3-axis magnetometer 1215, a 3-axis accelerometer 1220, and a microcontroller 1222. The gyroscope 1216 is used to measure angular acceleration, the magnetometer 1215 is used to measure magnetic fields, the accelerometer 1220 is used to measure linear acceleration and deceleration, and the microcontroller 1222 is used to process data and communicate between the sensor and the WPU1202.

[0140] Firmware 1224 runs on the microcontroller 1222 of board 1214. Firmware 1224 constantly calculates pitch and roll using 3-axis acceleration data 1220. By comparing the 3-axis acceleration data with programmatically defined thresholds and durations, firmware 1224 can determine if a trigger event has occurred, and if so, sends a trigger event message to WPU 1202. Every second, firmware 1224 sends a periodic data message to WPU 1202 containing a predefined set of values. This data is used, but is not limited to, determining the direction of travel, internal ambient temperature, and angular acceleration.

[0141] System software 1226 is an application that runs on WPU 1202. This application communicates directly with GPS 1206 and board 1214 to collect relevant data. In addition to this data, system software 1226, like all other applications on WPU 1202, collects data from other software applications using standard inter-process communication protocols. These other software applications run on WPU 1202 and communicate with other devices (such as DVR 1252 and event data recorder 1238) that are physically connected to WPU 1202. By using all the collected data, system software 1226 can determine whether a particular event has occurred by comparing the data with predefined thresholds and durations.

[0142] System 1200 consists of a board 1214, a WPU 1202 with firmware 1224 and system software 1226 installed, an event data recorder 1238, a DVR 1252, and a fuel level sensor 1210. System software 1226 runs on the WPU 1202 and constantly corrects the fuel level and checks event messages from the board 1214 or the event data recorder 1238 to take action.

[0143] The mobile asset data recorder and transmitter system 1200 (Figure 39) performs seven functions: automatic orientation, automatic compass calibration, emergency braking upon impact detection, fuel compensation based on pitch and roll, detection of rough operating conditions, engine operation detection, and inertial navigation (dead dead navigation). Each of these seven functions takes into account the signal generated by the 3-axis accelerometer 1220.

[0144] Automatic orientation is used to correlate the axes of the WPU1202 to the locomotive axes so that the values ​​measured by the sensors correspond to the locomotive axes. This process is achieved by software 1226 and firmware 1224. Due to the different electronic environments on the locomotives, the compass needs to be calibrated for each locomotive. The software uses the GPS 1206 of the WPU1202 (Figures 38 and 39) to determine the locomotive's direction of travel. Measurements are then taken from the magnetometer 1215 and stored in the corresponding positions of the array. The array consists of 360 positions, one for each direction of travel. Using these values, the WPU1202 software 1226 can correct the locomotive's own magnetic field and detect only the changes due to the Earth's magnetic field.

[0145] Figure 34 shows a flowchart of the method application for emergency braking by collision detection. The WPU 1202 (Figure 39) software 1226 (Figure 39) sends initialization commands to firmware 1224 (Figure 39) to establish acceleration durations in each axis (Adx, Ady, Adz) 1234 used to trigger events. These durations are stored on the device embodying the system 1200. The WPU 1202 software 2226 also sends initialization commands to firmware 1224 to establish acceleration thresholds in each axis (Atx, Aty, Atz) 1236 used to trigger events. These durations are stored on the device embodying the system 1200 (Figure 39). The microcontroller 1222 (Figure 39) extracts raw 3-axis acceleration (Ax, Ay, Az) 1240 data from the accelerometer 1220 at a rate of 100 Hz. A low-pass filter 1244 is applied to the raw acceleration values ​​(Ax, Ay, Az) 1240, resulting in filtered acceleration values ​​(Afx, Afy, Afz) 1244. The board 1214 (Figure 39) axis of the filtered acceleration values ​​(Afx, Afy, Afz) 1244 is converted to the asset axis (Af'x, Af'y, Af'z) 1248. The board 1214 value of the raw value (Ax, Ay, Az) 1240 is converted to the asset axis (A'x, A'y, A'z) 1246. The filtered values ​​(Af'x, Af'y, Af'z)1248 of the asset axis are added to the established thresholds (Atx, Aty, Atz)1236 for each axis, and this added threshold (Af'tx, Af'ty, Af'tz)1250 is then continuously compared with the raw acceleration 1251 on the asset axis (A'x, A'y, A'z)1246. When the raw value (A'x, A'y, A'z)1246 exceeds the threshold 1250 on one or more axes, the timer 1253 is activated. When the raw value 1246 no longer exceeds the threshold 1250 on a particular axis 1256, the duration during which the raw value 1246 exceeded the threshold 1250 is evaluated to determine whether the duration exceeds the duration specified for that axis (Adx, Ady, Adz)1234.If the event duration 1254 is longer than the established duration (Adx, Ady, Adz) 1234, a trigger event 1255 is stored, including details about which axis, the event duration, and the time of the trigger event. In parallel with this monitoring, the onboard software 1226 (Figure 39) receives periodic data messages 1256 from the onboard event data recorder 1238, which monitors the real-time status of various input sensors. The onboard software 1226 monitors the periodic data messages 1256 and detects when the periodic data messages 1256 indicate that an emergency brake application discrete signal 1257 has occurred. The onboard software 1226 stores the time when the emergency brake application event 1258 occurred. If the onboard software 1226 stores either the trigger event 1255 or the emergency brake time 1258, the onboard system software 1226 checks the timestamp of each event to see if the two most recent events 1259 recorded from the trigger event 1255 or the emergency brake application 1258 are close together. If it is detected that event 1259 occurred in very close proximity, the onboard software 1226 triggers the emergency braking application via a collision alert 1260, requests a digital video recorder download 1261 covering the time of the event from the onboard DVR 1252, and requests a data log file 1262 covering the time of the event from the event data recorder 1238. The onboard software 1226 receives the downloaded video 1263 covering the time of the event and the data log file 1264 covering the time of the event and sends both to the back office 1265 / 1266.

[0146] The user receives an alert indicating the actual force of the collision and whether the collision resulted in a capsize or derailment. This, combined with immediate access to GPS location, video, and event recorder information, allows the user to accurately relay the severity and scope of the incident to the first responder as they en route to the incident.

[0147] Figure 35 shows a flowchart of a method application for fuel compensation using accelerometer-based pitch and roll. WPU1202 (Figure 39) software 1226 (Figure 39) extracts raw 3-axis acceleration data (Ax, Ay, Az) 1240 from accelerometer 1220 at a rate of 100 Hz. A low-pass filter 1244 is applied to the raw data (Ax, Ay, Az) 1240, resulting in filtered acceleration values ​​(Afx, Afy, Afz) 1242. The board 1214 (Figure 39) axes of the filtered values ​​(Afx, Afy, Afz) 1242 are converted to asset axes (Af'x, Af'y, Af'z) 1248. The asset pitch 1267 is the arctangent of the asset's filtered x-axis and asset's filtered z-axis.

[0148]

number

[0149]

number

[0150] The distance 1269 in front of the center is combined with the tangent of the asset's pitch 1267 to obtain a first fuel distance adjustment. The distance 1270 to the left of the center is combined with the tangent of the asset's roll 1268 to obtain a second fuel distance adjustment. The first and second fuel distance adjustments are combined to provide a single fuel distance adjustment 1271. An onboard distance level sensor records the distance from the top of the tank to the fuel level present in the onboard fuel tank. The raw distance 1272 from the fuel sensor 1273 to the fuel is combined with the distance adjustment 1271 to produce an adjusted distance 1274. The adjusted distance 1274 is combined with a previously defined fuel tank geometric tank profile 1275 that maps the distance to the fuel value to the fuel volume 1276. This is adjusted as the asset moves over various terrains where the pitch 1267 and roll 1268 are changing, resulting in a final fuel volume 1277 that compensates for the movement of liquid in the tank of the operating mobile asset.

[0151] Figure 36 shows a flowchart of a method application for detecting potentially rough operating conditions using an accelerometer. The WPU 1202 (Figure 39) software 1226 (Figure 28) sends initialization commands to firmware 1224 (Figure 39) to establish acceleration durations for each axis (Adx, Ady, Adz) 1234 used to trigger events. These durations are stored on the device. The software 1226 also sends initialization commands to firmware 1224 to establish acceleration thresholds for each axis (Atx, Aty, Atz) 1236 used to trigger events. These durations are stored on the device. The microcontroller 1222 (Figure 39) extracts raw 3-axis acceleration data (Ax, Ay, Az) 1240 from the accelerometer 1220 at a rate of 100 Hz. A low-pass filter 1244 is applied to the raw acceleration value 1240, resulting in filtered acceleration values ​​(Afx, Afy, Afz) 1242. The board 1214 (Figure 39) axes of the filtered values ​​1242 are converted to asset axes (Af'x, Af'y, Af'z) 1248, and the board 1214 axes of the raw values ​​1240 are converted to asset axes (A'x, A'y, A'z) 1246. The filtered asset axes (Af'x, Af'y, Af'z) 1248 are added to the thresholds (Atx, Aty, Atz) 1236 established for each axis, and these added thresholds (Af'tx, Af'ty, Af'tz) 1250 are then continuously compared with the raw accelerations 1251 on the asset axes (A'x, A'y, A'z) 1246. When the raw value 1246 exceeds the threshold 1250 on one or more axes, timer 1253 is activated. When the raw value 1246 no longer exceeds the threshold 1250 on a particular axis, the duration during which the raw value 1246 exceeded the threshold 1250 is evaluated to determine if it exceeds the duration specified for that axis (Adx, Ady, Adz) 1234. If the event duration is longer than the duration (Adx, Ady, Adz) 1234 established for that axis, trigger event 1255 is stored, containing details about which axis, the event duration, and the time of the trigger event.

[0152] In parallel with this monitoring, the onboard software 1226 (Figure 39) monitors asset speed via periodic messages from the onboard event data logger 1238 (Figure 34) and / or from the onboard GPS device 1206 (Figures 38 and 39). The onboard software 1226 monitors asset speed 1278 and detects when the asset speed exceeds a specified value 1279. If the speed 1278 exceeds the specified value 1279 and a stored trigger event 1255 occurs simultaneously 1280, the onboard system software 1226 checks which axis triggered the event. If event 1281 is triggered on the z axis, the system logs a potential track problem alert 1282. If the event is triggered on the x or y axis, the system logs an operator error alert 1283. If either a potential track problem alert 1282 or an operator error alert 1283 is triggered, the onboard software 1226 requests a digital video recorder download 1261 from the onboard DVR 1252 covering the time of the event. The onboard software 1226 receives the downloaded video 1263 and sends it to the back office 1265.

[0153] Users can now use the normal operation of their mobile assets to accurately locate and be alerted in real time areas where their assets are encountering rough operating environments such as poor tracks / switches, rough seas, and poor roads. As soon as a rough operating environment is identified, users receive alerts, still or video images, and critical operating black box data. Remediation teams can respond to the exact location of poor roads or tracks. Sea routes can be adjusted to avoid barren currents or choppy waters. The effectiveness of any remediation or rerouting can be verified when an asset equipped with the following mobile asset data recorder and transmitter system crosses any previously flagged area.

[0154] Figure 37 shows a flowchart of a method application for engine operation detection using an accelerometer. The WPU 1202 (Figure 39) software 1226 (Figure 39) sends initialization commands to firmware 1224 (Figure 28) to establish the activity / inactivity durations for each axis (A1dx, A1dy, A1dz) 1284 used to trigger events. These durations are stored on the device. The WPU 1202 (Figure 39) software 1226 (Figure 39) sends initialization commands to firmware 1224 (Figure 39) to establish activity / inactivity thresholds for each axis (A1tx, A1ty, A1tz) 1285 used to trigger events. These durations are stored on the device. The microcontroller 1222 (Figure 39) extracts raw 3-axis acceleration data (Ax, Ay, Az) 1240 from the accelerometer 1242 at a rate of 100 Hz. A low-pass filter 1244 is applied to the raw acceleration values ​​(Ax, Ay, Az) 1240, resulting in filtered acceleration values ​​(Afx, Afy, Afz) 1246. The board 1214 (Figure 39) axis of the filtered values ​​1246 is converted to asset axis (Af'z, Af'y, Af'z) 1248, and the board 1214 axis of the raw values ​​1240 is converted to asset axis (A'x, A'y, A'z) 1249. The filtered values ​​(Af'x, Af'y, Af'z)1248 for the asset axis are added to the established activity / inactivity thresholds (A1tx, A1ty, A1tz)1285 for each axis, and this added threshold (Af'1tx, Af'1ty, Af'1tz)1286 is then continuously compared to the raw acceleration on the asset axis (A'x, A'y, A'z)1249. When the raw value 1249 exceeds the threshold 1246 on one or more axes, a timer 1287 is activated. When the raw value 1249 no longer exceeds the activity / inactivity threshold 1246 on a particular axis, the duration for which the raw value 1249 exceeded the threshold 1286 is evaluated to determine whether it exceeds the specified duration for that axis (A1dx, A1dy, A1dz)1284.If the event duration is longer than the established duration (A1dx, A1dy, A1dz) for that axis, a trigger activity / inactivity event is stored, including details for that axis, the event duration, and the time the event was triggered. The engine operating status is updated when the activity / inactivity event is triggered.

[0155] Figure 38 shows a flowchart of the method application for inertial navigation (dead dead navigation). Microcontroller 1222 (Figure 39) extracts raw 3-axis acceleration data (Ax, Ay, Az) 1240 from accelerometer 1242 at a rate of 100 Hz. A low-pass filter 1244 is applied to the raw acceleration values ​​(Ax, Ay, Az) 1240, resulting in filtered acceleration values ​​(Afx, Afy, Afz) 1246. The board 1214 (Figure 39) axes of the filtered values ​​1246 are converted to asset axes (Af'x, Af'y, Af'z) 1248, 1249. The asset pitch 1267 is the arctangent of the asset's filtered x-axis and asset's filtered z-axis.

[0156]

number

[0157]

number

[0158] The user receives precise departure and arrival alerts and logging in environments where GPS signals are blocked or partially blocked by overhangs and canopies. This system 1200 (Figure 39) allows the user to define a virtual “running wire” even in areas where GPS devices become useless due to RF signal loss or interference. Inertial navigation capabilities automate operator performance against schedule matrices by alerting and recording the precise time when assets cross the departure and arrival virtual “running wire” when GPS signals cannot calculate accurate location data.

[0159] As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or as is evident from the context, “X includes A or B” is intended to mean any of the natural inclusive substitutions. That is, if X includes A, X includes B, or X includes both A and B, “X includes A or B” is satisfied under any of the above examples. In addition, “X includes at least one of A and B” is intended to mean any of the natural inclusive substitutions. That is, if X includes A, X includes B, or X includes both A and B, “X includes at least one of A and B” is satisfied under any of the above examples. The articles "a" and "an" used in this application and the attached claims should generally be interpreted as meaning "one or more" unless otherwise specified or unless the context makes it clear that they refer to a singular form. Furthermore, the use of the terms "implementation" or "one implementation" throughout the application is not intended to mean the same embodiment, aspect, or implementation unless otherwise stated.

[0160] While this disclosure has been described in relation to specific embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments, but rather is intended to encompass a variety of modifications and equivalent structures included in the appended claims, and the claims should be given the broadest possible interpretation to include all such modifications and equivalent structures as permitted under the law.

Claims

1. A method for automating the assessment of the operational performance of a designated locomotive operator based on video data and operational data of a locomotive, wherein the method is performed by an onboard computer system and a remote computer system located away from the locomotive. The onboard computer system receives a request from a user of the web portal, including the identification information of the designated locomotive operator and a specified time range, via a web portal hosted on a server located away from the locomotive. Using the data acquisition and recording system installed on the locomotive, the operation data of the locomotive, data related to the designated locomotive operator, and data related to the designated time range are received. The aforementioned data is obtained from a first data source mounted on the locomotive and a second data source located away from the locomotive, based on at least one signal. The first data source comprises at least one camera and at least one data recorder that provide the video data. The artificial intelligence component of the video analysis system of the mounted computer system processes the video data and the motion data in order to generate processing data. The video analysis system analyzes the operational performance of the designated locomotive operator by comparing the processed data with train and crew data and predefined safety operation rules and / or regulatory compliance standards for the safe operation of the locomotive, and generates output data indicating whether the designated locomotive operator complies with or does not comply with the predefined safety operation rules and / or regulatory compliance standards. The remote computer system displays web-viewable content, including (i) video data derived from the processing data and (ii) output data indicating compliance or non-compliance of the designated locomotive operator, on a display device via the web portal. method.

2. The aforementioned at least one camera comprises at least one 360-degree camera and / or at least one fixed camera. Each of the at least one 360-degree camera and the at least one fixed camera is positioned inside the locomotive, on the locomotive, and / or near the locomotive. Each of the at least one 360-degree camera and the at least one fixed camera is either facing inward toward the interior of the locomotive or outward toward the exterior of the locomotive. Furthermore, audio data is received from at least one microphone located inside, on, and / or near the locomotive. The method according to claim 1.

3. The first data source mounted on the locomotive comprises at least one video recorder located inside, on, and / or near the locomotive, at least one sound recorder located inside, on, and / or near the locomotive, at least one accelerometer, at least one gyroscope, and at least one magnetometer. The method according to claim 1.

4. The data further includes at least one of the following: event data recorder data, accelerometer data, gyroscope data, fuel data, positive train control event log, network dispatch system data, weather data, train formation data, crew data, time data, movement authority data for the designated movement course of the locomotive, microphone data, inward-facing 360-degree camera data captured by an inward-facing 360-degree camera directed towards the interior of the locomotive, outward-facing 360-degree camera data captured by an outward-facing 360-degree camera directed towards the exterior of the locomotive, fixed camera data captured by an inward-facing fixed camera installed facing the interior of the locomotive, and fixed camera data captured by an outward-facing fixed camera installed facing the exterior of the locomotive. The method according to claim 1.

5. moreover, The data manager of the data acquisition and recording system mounted on the locomotive stores the data and / or the processed data in at least one of the memory components of a remote computer located away from the locomotive and at least one of the at least one local memory component of the data acquisition and recording system mounted on the locomotive. The data manager of the data acquisition and recording system mounted on the locomotive transmits the data and / or the processed data to the remote computer system via a wireless data link at a configurable predetermined rate. The remote computer system stores the data and / or the processed data in a remote data repository. The method according to claim 1.

6. moreover, In order to detect violations of the predefined safety operation rules and / or regulatory compliance standards for the safe operation of the locomotive, the remote computer system continuously monitors at least one of the data and / or processing data based on data related to the operation of the locomotive, the data related to the designated locomotive operator, and the data related to the designated time range. The method according to claim 1.

7. The data acquisition and recording system receives the data from at least one of a first data source mounted on the locomotive and a second data source located away from the locomotive, via a wireless data link, a wired data link, or both. The method according to claim 1.

8. moreover, The onboard computer system coordinates the video data captured by the at least one camera with the data received from the data acquisition and recording system and geographic location data. The data received from the aforementioned data acquisition and recording system includes first time information, The aforementioned geographic location data includes geographic location information and second time information, The coordination includes synchronizing the video data, the data received from the data acquisition and recording system, and the geographic location data based on the first time information and the second time information. The method according to claim 1.

9. The display of the aforementioned web-viewable content is as follows: The geographical location information of the locomotive based on geographical location data, and The system includes displaying an alert generated by the remote computer system when the processing data indicates a violation of the predefined safety operation rules, the regulatory compliance standards, or both of the locomotive's safe operation, The alert is generated based on an analysis performed by the artificial intelligence component of the video analysis system using the video data, the motion data, and the geographic location data. The method according to claim 1.

10. moreover, Based on the analysis of at least one of the operation data related to the operation of the locomotive, the data related to the designated locomotive operator, and the data related to the designated time range, the remote computer system receives user comments related to the event identified by the onboard computer system via the web portal. The remote computer system displays the user comments along with the processing data via the web portal. The method according to claim 1.

11. moreover, Generate a summary report of the performance of the aforementioned designated locomotive operator. The remote computer system displays the summary report via the web portal. The method according to claim 1.

12. moreover, Based on an analysis of the video data, the motion data, and the processing data relating to the predefined safety operation rules, regulatory compliance standards, or both for the safe operation of the locomotive, the onboard computer system determines a performance score for the designated locomotive operator using the artificial intelligence component of the video analysis system. The onboard computer system and the remote computer system determine, based on the performance score, whether the designated locomotive operator has met the predefined safety operating rules, regulatory compliance standards, or both for the safe operation of the locomotive. Based on the determination, a recommendation is generated for the authentication or deauthentication of the designated locomotive operator. The method according to claim 1.

13. A system for automating the assessment of the operational performance of a designated locomotive operator based on video data and operational data of a locomotive, wherein the system is: The computer system mounted on the locomotive and the remote computer system located away from the locomotive, The aforementioned remote computer system is configured to provide a web portal, The onboard computer system is adapted to receive requests from users via the web portal, including the identification information of the designated locomotive operator and a specified time range. A data acquisition and recording system mounted on the locomotive, comprising at least one data recorder, The data acquisition and recording system is adapted to receive the operation data of the locomotive, the data related to the designated locomotive operator, and the data related to the designated time range. The aforementioned data is obtained from at least one first data source mounted on the locomotive and at least one second data source located away from the locomotive, based on at least one signal. The at least one first data source comprises at least one camera and at least one data recorder of the data acquisition and recording system that provides the video data. A video analysis system equipped with artificial intelligence components, The aforementioned video analysis system is The aforementioned data is processed to generate processed data, The aforementioned processing data is compared with train and crew data, as well as predefined safety operation rules and / or regulatory compliance standards for the safe operation of the locomotive, to analyze the operational performance of the designated locomotive operator. The designated locomotive operator is adapted to generate output data indicating compliance or non-compliance with the predefined safety operating rules and / or regulatory compliance standards. The remote computer system is further adapted to display web-viewable content, including video data derived from the processing data and output data indicating compliance or non-compliance of the designated locomotive operator, on a display device via the web portal. system.

14. The aforementioned at least one camera comprises at least one 360-degree camera and / or at least one fixed camera. Each of the at least one 360-degree camera and the at least one fixed camera is positioned inside the locomotive, on the locomotive, and / or near the locomotive. Each of the at least one 360-degree camera and the at least one fixed camera is either facing inward toward the interior of the locomotive or outward toward the exterior of the locomotive. Furthermore, the system includes at least one microphone located inside, on, and / or near the locomotive. The system according to claim 13.

15. The first data source mounted on the locomotive comprises at least one video recorder located inside, on, and / or near the locomotive, at least one sound recorder located inside, on, and / or near the locomotive, at least one accelerometer, at least one gyroscope, and at least one magnetometer. The system according to claim 13.

16. The data further includes at least one of the following: event data recorder data, accelerometer data, gyroscope data, fuel data, positive train control event log, network dispatch system data, weather data, train formation data, crew data, time data, movement authority data for the designated movement course of the locomotive, microphone data, inward-facing 360-degree camera data captured by an inward-facing 360-degree camera directed towards the interior of the locomotive, outward-facing 360-degree camera data captured by an outward-facing 360-degree camera directed towards the exterior of the locomotive, fixed camera data captured by an inward-facing fixed camera installed facing the interior of the locomotive, and fixed camera data captured by an outward-facing fixed camera installed facing the exterior of the locomotive. The system according to claim 13.

17. Furthermore, the locomotive is equipped with a data manager for the data acquisition and recording system, The aforementioned data manager, The aforementioned data and / or the processing data are stored in at least one of the following: a memory component of a remote computer located away from the locomotive, and the data recorder of the data acquisition and recording system mounted on the locomotive. The data and / or the processed data are adapted to be transmitted to the remote computer system via a wireless data link at a configurable predetermined rate. The remote computer system is adapted to store the data and / or the processed data in a remote data repository. The system according to claim 13.

18. The aforementioned remote computer system further, The operation data and / or processing data are received from the data acquisition and recording system. The operation data and / or processing data are adapted to continuously monitor the operation data and / or processing data in order to detect any violation of the predefined safety operation rules and / or regulatory compliance standards for the safe operation of the locomotive. The system according to claim 13.

19. The data acquisition and recording system is further adapted to receive the data from at least one of a first data source mounted on the locomotive and a second data source located away from the locomotive, via a wireless data link, a wired data link, or both. The system according to claim 13.

20. The data acquisition and recording system and the video analysis system are adapted to coordinate the video data captured by the at least one camera with data received from the data acquisition and recording system and geographic location data. The data received from the aforementioned data acquisition and recording system includes first time information, The aforementioned geographic location data includes geographic location information and second time information, The coordination includes synchronizing the video data, the data received from the data acquisition and recording system, and the geographic location data based on the first time information and the second time information. The system according to claim 13.

21. The video analysis system is further adapted to analyze the data and / or the processed data to generate analysis data based on designated government regulatory requirements for the certification or decertification of the designated locomotive operator. The analysis data includes the operational data, performance data, and / or behavioral characteristics related to the locomotive and the designated locomotive operator, and related to a predetermined geographical segment. The aforementioned analysis data is related to the specified time range. The system according to claim 13.

22. The remote computer system is further adapted to display the processed data on a display device via the web portal. The processed data is The geographical location information of the locomotive based on geographical location data, and The system includes an alert generated by the remote computer system when the processing data indicates a violation of the predefined safety operation rules, the regulatory compliance standards, or both of the locomotive's safe operation, The alert is generated based on an analysis performed by the artificial intelligence component of the video analysis system using the video data, the motion data, and the geographic location data. The system according to claim 13.

23. The aforementioned remote computer system further, User comments related to an event identified based on the analysis of at least one of the operation data related to the operation of the locomotive, the data related to the designated locomotive operator, and the data related to the designated time range are received via the web portal. The user comments, along with the processing data, are adapted to be displayed on a display device via the web portal. The system according to claim 13.

24. The aforementioned data acquisition and recording system further includes: Generate a summary report of the performance of the aforementioned designated locomotive operator. The web portal is adapted to display the summary report. The system according to claim 13.

25. The aforementioned onboard computer system and the aforementioned remote computer system further, Based on an analysis of the video data, the motion data, and the processing data based on the predefined safety operation rules and / or regulatory compliance standards, a performance score for the designated locomotive operator is determined using the artificial intelligence component of the video analysis system. Based on the performance score and the determination of whether the predefined safety operating rules and / or regulatory compliance criteria are met, the system is adapted to generate recommendations for the certification or decertification of the designated locomotive operator. The system according to claim 13.

26. A method for automating the assessment of the operational performance of a designated locomotive operator based on the analysis of video data and operational data of a locomotive, and for certifying or decertifying the designated locomotive operator, wherein the method is performed by a computer system. The aforementioned computer system receives user requests, including the identification information of the designated locomotive operator and the specified time range, via a web portal. The data acquisition and recording system installed on the locomotive receives operational data related to the operation of the locomotive, data related to the designated locomotive operator, and data related to the designated time range. The aforementioned data is obtained from a first data source mounted on the locomotive and a second data source located away from the locomotive, based on at least one signal. The first data source comprises at least one camera and at least one data recorder that provide the video data. The artificial intelligence component of the video analysis system of the computer system processes the video data and the motion data in order to generate processing data. The artificial intelligence component of the video analysis system reviews the selected event and second data related to points along the locomotive's travel path. In order for the designated locomotive operator to determine whether or not the locomotive complies with predefined safety operation rules and / or regulatory compliance standards for safe operation, the artificial intelligence component analyzes the processing data and additional operational data related to the selected events and the points along the locomotive's travel path, Based on the determination of compliance or non-compliance, the designated locomotive operator shall be certified or decertified. method.

27. moreover, Using the video analysis system, the processing data and the additional operational data related to the selected events and the points along the locomotive's travel path are analyzed based on the designated government regulatory requirements for the certification or decertification of the designated locomotive operator to generate analysis data. The analysis data includes the operational data, performance data, and / or behavioral characteristics related to the locomotive and the designated locomotive operator, and related to a predetermined geographical segment. The aforementioned analysis data is related to the specified time range. The method according to claim 26.

28. moreover, Using the video analysis system, the data and / or the processed data are analyzed to generate analysis data based on the designated government regulatory requirements for the certification or decertification of the designated locomotive operator. The analysis data includes the operation data, performance data, and / or behavioral characteristics related to the locomotive's operation data and / or the designated locomotive operator, and related to a predetermined geographical segment. The aforementioned analysis data is related to the specified time range. The method according to claim 26.

29. moreover, Using the artificial intelligence component of the video analysis system, based on the determination of compliance or non-compliance, recommend the authentication or deauthentication of the designated locomotive operator. The method according to claim 26.

30. moreover, The computer system determines, based on an analysis of the video data, the operation data, and the processing data, whether the locomotive meets the predefined safety operation rules and / or regulatory compliance standards for safe operation, using the artificial intelligence component of the video analysis system. Based on the above determination, the designated locomotive operator is authenticated or deauthenticated. The method according to claim 26.

31. A system for automating the assessment of the operational performance of a designated locomotive operator based on the analysis of video data and motion data of a locomotive, and for certifying or decertifying the designated locomotive operator, wherein the system comprises: A computer system configured to include a web portal, The computer system is adapted to receive user requests, including the identification information of the designated locomotive operator and a specified time range, via a web portal. A data acquisition and recording system mounted on the aforementioned locomotive, The data acquisition and recording system is adapted to receive the operation data related to the operation of the locomotive, the data related to the designated locomotive operator, and the data related to the designated time range. The aforementioned data is obtained from a first data source mounted on the locomotive and a second data source located away from the locomotive, based on at least one signal. The first data source comprises at least one camera and at least one data recorder that provide the video data. A video analysis system equipped with artificial intelligence components, The aforementioned video analysis system is The aforementioned data is processed into processed data, The processing data and additional operational data related to selected events and points along the locomotive's travel path are adapted to allow the designated locomotive operator to analyze the processing data and additional operational data related to selected events and points along the locomotive's travel path in order to determine whether the locomotive operator complies with or does not comply with predefined safety operation rules and / or regulatory compliance standards for the safe operation of the locomotive. The computer system is further adapted to certify or decertify the designated locomotive operator based on a determination of compliance or non-compliance. system.

32. The aforementioned video analysis system further, Based on the designated government regulatory requirements for the certification or decertification of the designated locomotive operator, the processing data and the additional operation data related to the selected events and the points along the locomotive's movement path are analyzed to generate analysis data. The analysis data includes the operation data, performance data, and / or behavioral characteristics related to the locomotive's operation data and / or the designated locomotive operator, and related to a predetermined geographical segment. The aforementioned analysis data is related to the specified time range. The system according to claim 31.

33. The artificial intelligence component of the video analysis system is further adapted to recommend the authentication or deauthentication of the designated locomotive operator based on the determination of compliance or non-compliance. The system according to claim 31.

34. The aforementioned computer system further, The computer system determines, based on an analysis of the video data, the operation data, and the processing data, whether the locomotive meets the predefined safety operation rules and / or regulatory compliance standards for safe operation, using the artificial intelligence component of the video analysis system. Based on the above determination, the designated locomotive operator is authenticated or deauthenticated. The system according to claim 31.

Images