Video content analysis system and method for transportation systems

Abstract

It will provide for faster emergency response, verify the effectiveness of repairs and rerouting, and monitor crew performance and safety. The truck detection and infrastructure monitoring system includes a reinforcement learning component, an object detection and localization component, and an obstacle detection component, and analyzes video data, vehicle data, weather data, and path / manifest data to determine internal and / or external conditions associated with the asset. A data acquisition and recording system uploads the data, internal and / or external condition information, object detection information, object location information, and obstacle detection information to a remote memory module to provide real-time streaming video data to a remote user. The remote user can view the data in various view modes via a web browser or a virtual reality device.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to the extent permitted by law to U.S. Provisional Application No. 62 / 337,225, filed May 16, 2016, which claims priority to U.S. Provisional Application No. 62 / 337,227, filed May 16, 2016, which claims priority to U.S. Provisional Application No. 62 / 337,228, filed May 16, 2016, and is a continuation-in-part of U.S. Non-provisional Application No. 15 / 595,650, filed May 15, 2017, and a continuation-in-part of U.S. Non-provisional Application No. 15 / 595,689, filed May 15, 2017, the contents of which are incorporated herein by reference in their entireties.

[0002] The present disclosure relates to devices for use on high-value moving assets, and in particular to real-time data acquisition and recording systems for use on high-value moving assets. [Background technology]

[0003] High-value mobile assets, such as locomotives, aircraft, mass transit systems, mining equipment, transportable medical equipment, cargo, ships, and military vessels, typically employ onboard data acquisition and recording "black box" and / or "event recorder" systems. Data acquisition and recording systems, such as event data recorders and flight data recorders, log various system parameters used for incident investigation, crew performance evaluation, fuel consumption analysis, maintenance planning, and predictive diagnostics. Typical data acquisition and recording systems include digital and analog inputs as well as pressure switches and pressure transducers to record data from various onboard sensor devices. Recorded data can include parameters such as speed, distance traveled, location, fuel level, engine revolutions per minute (RPM), fluid levels, driver controls, pressure, current and forecast weather and ambient conditions, and more. In addition to basic event and operational data, video and audio event / data recording capabilities are also deployed on many of these same mobile assets. Data is typically extracted from data recorders when they are recovered after an asset-related incident necessitating an investigation. However, situations may arise where a data recorder cannot be recovered or the data is otherwise unavailable, in which case the data, including event and operational data, video data, and audio data, captured by the data acquisition and recording system is needed quickly, regardless of whether the data acquisition and recording system or physical access to that data is available. Summary of the Invention [Means for solving the problem]

[0004] The present disclosure generally relates to real-time data acquisition and recording systems for use with high-value mobile assets. The teachings herein can provide real-time or near-real-time access to and video content analysis of video data related to high-value mobile assets. One implementation of a method for processing data from a mobile asset described herein includes: using a video analytics system onboard the mobile asset to receive data based on at least one data signal from at least one data source onboard the mobile asset and at least one data source remote from the mobile asset; using a reinforcement learning component of the video analytics system to process the data into processed data; using the video analytics system to transmit at least one of the data and the processed data to a data recorder onboard the mobile asset; using a data encoder of the data recorder to encode a record including a bitstream based on the processed data; and using an onboard data manager of the data recorder to transmit at least one of the data and the processed data to the data recorder. At least one local memory component and storing the information at a configurable first predetermined frequency.

[0005] One implementation of a system for analyzing video content described herein includes at least one of at least one 360-degree camera, at least one fixed camera, and at least one microphone; and a video analysis system onboard a mobile asset, the video analysis system including a reinforcement learning component, an object detection and localization component, and an obstacle detection component, and configured to receive data based on at least one data signal from at least one of the at least one 360-degree camera, the at least one fixed camera, and the at least one microphone, wherein the reinforcement learning component is configured to process the data into processed data, the object detection and localization component is configured to determine object detection data and object position data for a first object based on the processed data, and the obstacle detection component is configured to determine the obstacle detection data based on at least one of the processed data, the object detection data, and the object position data.

[0006] Variations on these and other aspects of the disclosure are described in further detail below. [Brief explanation of the drawings]

[0007] This description will be made with reference to the accompanying drawings, in which like reference numerals refer to like parts in the several drawings. [Figure 1] 1 illustrates a field installation of an exemplary real-time data acquisition and recording system in accordance with an implementation of the present disclosure. [Figure 2A] FIG. 1 illustrates an example track detection in accordance with implementations of the present disclosure. [Figure 2B] 10A-10C illustrate exemplary track detection and switch detection in accordance with implementations of the present disclosure. [Figure 2C] 10A-10C illustrate exemplary track detection, track counting, and signal detection according to implementations of the present disclosure. [Figure 3] 1 is a flow diagram illustrating a process for determining the internal state of a mobile asset according to an implementation of the present disclosure. [Figure 4] 1 is a flow diagram illustrating a process for determining object detection and obstacle detection occurring outside of a moving asset according to an implementation of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0008] The real-time data acquisition and recording system and video analytics system described herein provide remotely located users with real-time or near-real-time access to a wide range of data, including event and operational data, video data, and audio data, related to high-value assets. The data acquisition and recording system records data related to the asset before, during, and after an incident and streams the data to a remote data repository and remotely located users. The data is streamed to the remote data repository in real time or near-real time. This essentially eliminates the need to locate and download a "black box" to investigate an incident involving the asset by streaming information to the remote data repository in real time and making the information available at least until a catastrophic event occurs. The DARS performs video analytics on video data recorded for mobile assets to determine, for example, cab occupancy and track detection. The remotely located user accesses a standard web browser, which navigates the user to the desired data related to the selected asset and allows the user to view the data. This eliminates the need to interact with the data capture and recording system on the asset to request the download of specific data, locate and transfer the file, and use a dedicated application to view that data.

[0009] DARS provides remote users with access to video data and video analysis performed by the video analytics system by streaming data to a remote data repository and remote users before, during, and after an incident. This eliminates the need for users to manually download, extract, and play video to review video data and determine cab occupancy for crew members or unauthorized individuals at the time of the incident, track detection, investigation, or other target. Furthermore, the video analytics system processes image and video data in real time to provide cab occupancy status determination, track detection, and lead and trail unit determination, thereby ensuring that the correct data is always available to users. For example, real-time image processing enhances railroad safety by ensuring that a locomotive designated as a trailing locomotive is not leading. Previous systems provided locomotive locations within a train by using the train configuration function in the dispatch system. This dispatch system information is sometimes out of date because it is not updated in real time and can be replaced by the responsible crew if deemed necessary.

[0010] Prior to the disclosed system, inspectors and / or asset managers had to manually inspect truck conditions, manually verify whether the vehicle was in a leading or trailing position, manually survey the location of each object of interest, manually create a database of the geographic locations of the objects of interest, periodically conduct manual field surveys of each object of interest to verify their location and identify any changes in their geographic location from the original survey, manually update the database if the objects of interest have changed location due to repairs or additional infrastructure development since the original database was created, select and download the desired data from a digital video recorder or data recorder, inspect the downloaded data or footage offline, check the truck for obstructions, and have the vehicle operator physically inspect the truck for obstructions and / or switch changes. The disclosed system eliminates the need for users to perform these steps, allowing users to simply use a common web browser to navigate to the desired data. Asset owners and operators can automate and improve the efficiency and safety of their mobile assets in real time, proactively monitor truck conditions, and receive real-time alert information. The disclosed system eliminates the need for asset owners and operators to download data from data recorders to monitor truck conditions and investigate accidents. As a preventative safety system, DARS can assist operators in checking for obstacles, sending alerts in real time and / or storing the information offline for remote monitoring and storage. Both current and historical truck detection information can be stored in real time in a remote data repository, facilitating users' viewing of the information as needed. Remote users can access a common web browser to navigate to desired data related to a selected asset, allowing them to view and analyze asset operational efficiency and safety in real time or near real time.

[0011] The disclosed system can be used to continuously monitor objects of interest and identify them in real time if they have moved or been damaged, become blocked by foliage, or have malfunctioned and require maintenance. DARS utilizes video, image, and / or audio information to detect and identify various infrastructure objects, such as railroad tracks, within the video, and has the capability to follow the track as the mobile asset progresses, periodically creating, auditing, and updating a database of objects of interest along with their geographic location. DARS can automatically inspect the condition of the track, including counting the number of tracks present, identifying the current track the mobile asset is following, detecting existing obstructions or defects such as washed-out ballast, broken tracks, out-of-gauge tracks, misaligned switches, transfer switches, track flooding, and snow accumulation, and scheduling preventative maintenance that will not result in a catastrophic event. DARS can also detect track switches and track changes. DARS can also detect changes in the location of data, such as whether an object is missing, obstructed, and / or present where it should be. Truck detection, infrastructure diagnostic information, and / or infrastructure monitoring information can be displayed to the user using a standard web client, such as a web browser, eliminating the need to download files from the data recorder and use specialized application software or other external applications to view the information, as required by previous systems. This process can be extended to automatically create, audit, and / or update a database with the geographic location of objects of interest to ensure compliance with federal regulations. The disclosed system utilizes cameras previously installed to comply with federal regulations to perform various tasks that previously required human interaction, specialized vehicles, and / or alternate equipment. DARS allows these tasks to be performed automatically as mobile assets travel throughout their territory as part of their normal revenue service and daily operations.DARS can save countless hours of manual labor by utilizing normal vehicle operation and previously installed cameras to perform tasks that previously required manual intervention. DARS can also perform tasks previously performed using specialized vehicles. This eliminates the need to close sections of track to inspect and locate track and objects of interest, which often results in lost revenue service and the purchase and maintenance of expensive equipment. DARS also reduces the amount of time people must be near the tracks, reducing accidents overall and the potential loss of life.

[0012] Data may include, but is not limited to, measured analog and frequency parameters such as speed, pressure, temperature, current, voltage, and acceleration originating from and / or near the moving asset; measured Boolean data such as switch position, actuator position, warning light intensity, actuator commands; position, speed, and altitude information from a Global Positioning System (GPS) and additional data from a Geographic Information System (GIS) such as latitude and longitude of various objects of interest; internally generated information such as the regulatory speed limit of a moving asset given its current location; train control status and operational data generated by systems such as Positive Train Control (PTC); and vehicle information such as speed, acceleration, and position, including that received from a GPS. The data may include data obtained from any combination of the above sources, including vehicle and inertial parameters, GIS data such as latitude and longitude of various objects of interest, video and image information from at least one camera positioned at various locations within, on, or near the mobile asset, audio information from at least one microphone positioned at various locations within, on, or near the mobile asset, information regarding the mobile asset's operational plan, such as route, schedule, and cargo manifest information, transmitted from a data center to the mobile asset, information regarding environmental conditions, such as current and forecasted weather, in the area in which the mobile asset is currently operating or will be operating, and additional data, video, and audio analysis and analytics.

[0013] "Track" may include, but is not limited to, railroad rails and sleepers used to transport locomotives and / or trains. "Objects of interest" may include, but are not limited to, various infrastructure objects installed and maintained within the vicinity of the tracks that can be identified using reinforcement learning of camera images and video of the asset. Reinforcement learning utilizes a previously labeled dataset defined as "training" data to enable remote and autonomous identification of objects within the field of view of cameras within, on, or near a mobile asset. DARS may or may not require human interaction at all stages of implementation. Examples of implementation stages include, but are not limited to, labeling the training dataset required for reinforcement learning. Objects include, but are not limited to, tracks, track centers, running signs, signals, crossing gates, switches, intersections, and text signs. "Video analytics" refers to any unambiguous information gathered by analyzing video and / or images recorded from at least one camera within, on, or near a mobile asset. Examples include, but are not limited to, tracking of target objects, geographic location of objects, movement of obstacles, distance between target objects and mobile assets, misalignment, etc. Video analytics systems can also be used in any mobile asset, residential area, space, or room that includes a surveillance camera to enhance video surveillance. In mobile assets, the video analytics system provides detection of spontaneous events occupying the vehicle interior economically and efficiently for remote users.

[0014] FIG. 1 illustrates a field installation of a first embodiment of an exemplary real-time data acquisition and recording system (DARS) 100 in which aspects of the present disclosure can be implemented. The DARS 100 is a system that distributes real-time, video, and audio information from a data recorder 102 on a mobile asset 164 through a data center 166 to remotely located end users 168. The data recorder 102 is installed in a vehicle, i.e., the mobile asset 164, and communicates with any number of different information sources via any combination of wired and / or wireless data links 142, such as a wireless gateway / router (not shown). The data recorder 102 collects video, audio, and other data or information via the onboard data link 142 from a wide variety of sources, which may vary widely based on the asset's configuration. The data recorder 102 includes local memory components within the asset 164, such as a crash-protected memory module 104, an onboard data manager 106, and a data encoder 108. In a second embodiment, the data recorder 102 can also include a removable storage device (not shown) without crash protection. An exemplary crash protection memory module 104 can be, for example, a shock-resistant event recorder memory module that complies with Federal Regulations and Federal Railroad Administrative Regulations, a crash survivable memory unit that complies with Federal Regulations and Federal Aviation Administration regulations, a crash protection memory module that complies with applicable Code of Federal Regulations, or other suitable hardened memory known in the art. The wired and / or wireless data link can include any one or combination of discrete signal inputs, standard or proprietary Ethernet, serial connections, and wireless connections.

[0015] DARS 100 further comprises a video analytics system 110 that includes a track detection and infrastructure monitoring component 114. The track detection and infrastructure monitoring component 114 comprises a reinforcement learning component 124 or other neural network or artificial intelligence component, an object detection and localization component 126, and an obstacle detection component 128. In this implementation, live video data is captured by at least one camera 140 mounted within, on, or near the asset 164. The camera 140 is positioned at an appropriate height and angle to capture video data in and around the asset 164 to obtain a sufficient amount of view for further processing. Live video and image data is captured by the camera 140 in front of and / or around the asset 164 and is fed to the track detection and infrastructure monitoring component 114 for analysis. The track detection and infrastructure monitoring component 114 of the video analytics system 110 processes each frame of the live video and image data to detect the presence of railroad tracks and any objects. Camera position parameters such as height, angle, shift, focal length, field of view, etc. can be fed to either the truck detection and infrastructure monitoring component 114 or the camera 140 and configured to enable the video analytics system 110 to detect and identify the camera position and parameters.

[0016] To identify situations, such as cab occupancy detection, the video analytics system 110 uses a reinforcement learning component 124 and / or other artificial intelligence and learning algorithms to evaluate asset data 134, such as video data from cameras 140, speed, GPS data, and inertial sensor data, data from a weather component 136, and data from a route / occupant, manifest, and GIS component 138. Cab occupancy detection is susceptible to environmental noise, such as light reflecting off clouds or sunlight filtering through buildings and trees as the asset moves. To address environmental noise, the reinforcement learning component 124, object detection and localization component 126, obstacle detection component, asset component 134 data, such as speed, GPS data, and inertial sensor data, data from the weather component 136, and other learning algorithms are combined to form a determination of interior and / or exterior situations involving the mobile asset 164. The track detection and infrastructure monitoring component 114 may also include a facial recognition system adapted to enable authorized access to the locomotive as part of a locomotive security system, a fatigue detection component adapted to monitor crew alertness, and an activity detection component to detect unauthorized activity such as smoking.

[0017] Reinforcement learning of the truck using reinforcement learning component 124 utilizes various information obtained from successive frames of video and / or images, as well as additional information received from data center 166 and vehicle data component 134, including inertial sensor data and GPS data, to determine training data. Object detection and localization component 126 utilizes the training data received from reinforcement learning component 124 and specific information about the moving asset 164 and track, such as track width and curvature, sleeper position, and vehicle speed, which distinguish the track, signs, and signals from other objects, to determine object detection data. Obstacle detection component 128 utilizes the object detection data received from object detection and localization component 126, additional information from weather component 136, route / occupant manifest data, and GIS data component 138, and vehicle data component 134, including inertial sensor data and GPS data, to improve accuracy and determine obstacle detection data. Mobile asset data from vehicle data component 134 includes, but is not limited to, speed, position, acceleration, yaw / pitch rate, and railroad crossings. Additional information received and utilized from data center 166 includes, but is not limited to, day / night details and geographic location of mobile asset 164.

[0018] The infrastructure object of interest, information processed by the track detection and infrastructure monitoring component 114, diagnostic and monitoring information, is transmitted over on-board data link 142 to a data encoder 108 in the data recorder 102 for encoding. The data recorder 102 stores the encoded data in a crash-protected memory module 104, and optionally also in any non-crash-protected removable storage device, and transmits the encoded information over a wireless data link 144 to a remote data manager 146 in a data center 166. The remote data manager 146 stores the encoded data in a remote data repository 148 in the data center 166.

[0019] The video analytics system 110 uses a reinforcement learning component 124 or other artificial intelligence, object detection and localization component 126, and obstacle detection component 128, as well as other image processing algorithms, to process and evaluate camera images and video data from cameras 140 in real time to determine obstacle detection 128 or object detection 126, such as a truck ahead of an asset 164. The truck detection and infrastructure monitoring component 114 uses the processed video data in conjunction with asset component 134 data, which may include speed, GPS data, and inertial sensor data, weather component 136 data, route / occupant manifest, and GIS component 138 data, to identify external situational features, such as leading and trailing moving assets, in real time. For example, when processing image and video data for track detection, the video analytics system 110 automatically configures the camera 140 parameters required for track detection, detects run-through switches, counts the number of trucks, detects additional trucks beside the asset 164, identifies which track the asset 164 is currently traveling on, detects track irregularities, detects track washouts such as detecting water near the track within a specified range, and detects missing ramps or tracks. The accuracy of object detection depends on the lighting conditions in and around the asset 164. DARS 100 can adapt to various lighting conditions using additional data collected on the asset 164 and from the data center 166. DARS100 has been improved to work in different lighting conditions, work in different weather conditions, detect more target objects, automatically integrate with existing database systems to create, audit and update data, detect multiple tracks, work consistently with curved tracks, detect any obstacles, detect any defects in the track that could pose a safety hazard, and work with low-cost embedded systems.

[0020] Interior and / or exterior situation determinations from video analytics system 110, such as cab occupancy, object detection and location such as truck detection, and obstacle detection, are provided to data recorder 102 via on-board data link 142, along with any data from a vehicle management system (VMS) or digital video recorder component 132. Data recorder 102 stores the interior and / or exterior situation determinations, object detection and location component 126 data, and obstacle detection component 128 data in crash-protected memory module 104, and optionally in a second embodiment on a non-crash-protected removable storage device, and in a remote data repository 148 via remote data manager 146 located in data center 166. Web server 158 provides the interior and / or exterior situation determinations, object detection and location component 126 information, and obstacle detection component 128 information to a remote user 168 via web client 162 upon request.

[0021] The data encoder 108 encodes at least a minimum data set, typically defined by a regulatory agency. The data encoder 108 receives video, image, and audio data from any cameras 140, video analytics system 110, and video management system 132 and compresses, encodes, and time-synchronizes the data to facilitate efficient real-time transmission and replication to a remote data repository 148. The data encoder 108 transmits the encoded data to the onboard data manager 106, which then transmits the encoded video, image, and audio data to the remote data manager 146 via a remote data manager 146 located in a data center 166 in response to an on-demand request by a user 168 or in response to specific operating conditions observed on the asset 164. The onboard data manager 106 and the remote data manager 146 work in conjunction to manage the data replication process. The remote data manager 146 in the data center 166 can manage the replication of data from multiple assets 164.

[0022] Onboard Data Manager106 The onboard data manager determines whether to queue or immediately transmit detected events, interior and / or exterior situation determinations, object detection and location, and / or obstacle detection based on the prioritization of detected events. For example, in normal operating conditions, detecting an obstacle on the track is much more urgent than detecting whether someone is in the cab of the asset 164. 106 The onboard data manager 106 also transmits data to a queue repository (not shown). In near real-time mode, the onboard data manager stores the encoded data and any event information received from the data encoder 108 in the collision-protected memory module 104 and the queue repository. After five minutes of encoded data have accumulated in the queue repository, the onboard data manager 106 stores the five minutes of encoded data in the remote data repository 148 via the wireless data link 144 to the remote data manager 146 in the data center 166. In real-time mode, the onboard data manager 106 stores the encoded data and any event information received from the data encoder 108 over a wireless data link 144 via a remote data manager 146 in a data center 166 in a collision-protected memory module 104 and a remote data repository 148 .

[0023] In this implementation, the onboard data manager 106 transmits video data, audio data, interior and / or exterior condition determinations, object detection and location information, obstacle detection information, and any other data or event information over a wireless data link 144 to a remote data repository 148 via a remote data manager 146 in a data center 166. The wireless data link 144 may be, for example, a wireless local area network (WLAN), a wireless metropolitan area network (WMAN), a wireless wide area network (WWAN), a wireless virtual private network (WVPN), a cellular network, or any other means of transferring data, in this example, from the data recorder 102 to the remote data manager 146. The process of remotely retrieving data from the asset 164 requires a wireless connection between the asset 164 and the data center 166. If a wireless data connection is unavailable, the data is stored and queued until a wireless connection is restored.

[0024] Concurrent with data recording, the data recorder 102 continuously and autonomously replicates the data to the remote data repository 148. This replication process has two modes: real-time mode and near real-time mode. In real-time mode, data is replicated to the remote data repository 148 every second. In near real-time mode, data is replicated to the remote data repository 148 every five minutes. The frequency used for near real-time mode is configurable, and the frequency used for real-time mode can be adjusted to support high-resolution data by replicating data to the remote data repository 148 every 0.10 seconds. Near real-time mode is used during normal operation under most conditions to improve the efficiency of the data replication process.

[0025] Real-time mode can be initiated based on an event occurring at the asset 164 or by a request initiated from the data center 166. A typical data center 166-initiated request in real-time mode is initiated when a remote user 168 requests real-time information from the web client 162. A typical reason for an asset 164 to be the source of real-time mode is the detection of an event or incident related to the asset 164, such as an operator-initiated emergency stop request, an emergency braking maneuver, a sudden acceleration or deceleration in any axis, or a loss of input power to the data recorder 102. When transitioning from near real-time mode to real-time mode, all data not already replicated to the remote data repository 148 is replicated and stored in the remote data repository 148, after which live replication begins. The transition from near real-time mode to real-time mode typically occurs in less than five seconds. After a predetermined time has passed since the event or incident, a predetermined period of inactivity, or if the user 168 no longer desires real-time information from the asset 164, the data recorder 102 reverts to near real-time mode. The predetermined time required to initiate this transition is configurable and is typically set to 10 minutes.

[0026] When the data recorder 102 is in real-time mode, the on-board data manager 106 continuously attempts to empty its queue to the remote data manager 146, storing the data in the collision-protected memory module 140, and optionally in the second embodiment, a removable storage device without collision protection, and simultaneously transmitting the data to the remote data manager 146.

[0027] Upon receiving video data, audio data, interior and / or exterior condition determinations, object detection and location, obstacle detection information, and any other data or information to be replicated from the data recorder 102, the remote data manager 146 stores the data received from the onboard data manager 106, such as encoded data and detected event data, in a remote data repository 148 in the data center 166. The remote data repository 148 may be, for example, a cloud-based data store or other suitable remote data store. Once this data is received, a process begins in which the recently replicated data from the remote data repository 148 is decoded by a data decoder 154 and sent to the track / object detection / location component 150, which checks the stored data for additional "post-processed" events. In this implementation, the track / object detection / location component 150 includes an object / obstacle detection component that identifies the interior and / or exterior condition determinations, object detection and location information, and obstacle detection information. Upon detecting interior and / or exterior information, object detection and location, and / or obstacle detection information, the track / object detection / location component 150 stores the information in a remote data repository 148 .

[0028] A remotely located user 168 can access any other information stored in the remote data repository 148, including video data, audio data, interior and / or exterior condition determinations, object detection and location information, obstacle detection information, truck information, asset information, and cab occupancy information for a specified asset 164 or multiple assets, using a standard web client 162, such as a web browser, or, in this implementation, a virtual reality device (not shown), which can display thumbnail images from a selected camera. The web client 162 communicates the user's 168 information request to the web server 158 over a network 160 using common web standards, protocols, and techniques. The network 160 may be, for example, the Internet. The network 160 may also be a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a virtual private network (VPN), a cellular network, or any other means of transferring data from the web server 158 to the web client 162 in this example. When the web server 158 requests desired data from the remote data repository 148, the data decoder 154 retrieves the requested data for the specified asset 164 from the remote data repository 148 in response to the request from the web server 158. The data decoder 154 decodes the requested data and sends the decoded data to the localizer 156. Once the raw encoded data and detected track / object detection / location information are stored in the remote data repository 148 using Coordinated Universal Time (UTC) and the International System of Units (SI units), the localizer 156 accesses the web client 162 to identify profile settings established by the user 168 and uses the profile settings to prepare information to send to the web client 162 for presentation to the user 168. The localizer 156 converts the decoded data into a format desired by the user 168, such as the user's 168's preferred units of measurement and language. The localizer 156 sends the localized data in the user's 168's preferred format to the web server 158 as requested.The web server 158 then transmits the localized data to the web client 162 for viewing and analysis, providing playback and real-time display of standard and 360-degree video along with interior and / or exterior condition determinations, object detection and location information, and obstacle detection information, such as track information and the information shown in Figures 2A, 2B, and 2C.

[0029] The web client 162 is enhanced with a software application that provides 360-degree video playback in a variety of different modes: the user 168 can select the mode in which the software application presents the video playback, such as fisheye view, dewarped view, panoramic view, double panoramic view, and quad view.

[0030] 3 is a flow diagram illustrating a process 300 for determining the internal state of an asset 164 according to one implementation of the present disclosure. The video analytics system 110 receives data signals from various input components 302, such as cameras 140 located on, in, or near the asset 164, a vehicle data component 134, a weather component 136, and a route / manifest and GIS component 138. The video analytics system 110 processes the data signals using a reinforcement learning component 304 to determine an internal state 306, such as cab occupancy.

[0031] 4 is a flow diagram illustrating a process 400 for determining object detection / localization and obstacle detection occurring outside of an asset 164, according to one implementation of the disclosure. The video analytics system 110 receives data signals from various input components 402, such as cameras 140 located on, within, or near the asset 164, a vehicle data component 134, a weather component 136, and a path / manifest and GIS component 138. The video analytics system 110 processes the data signals using a reinforcement learning component 124, an object detection / localization component 126, and an obstacle detection component 128, 404, to determine obstacle detection 406 and object detection and location 408, such as the presence of a truck.

[0032] For ease of explanation, process 300 and process 400 are shown and described as a series of steps. However, steps in accordance with the present disclosure may be performed in various orders and / or in parallel. Also, steps in accordance with the present disclosure may be performed with other steps not shown and described herein. Furthermore, not all illustrated steps may be required to practice a method in accordance with the disclosed subject matter.

[0033] Although the present disclosure has been described in connection with particular embodiments, the present disclosure is not limited to the disclosed embodiments, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is to be accorded the broadest interpretation so as to encompass all modifications and equivalent arrangements permitted by law.

Claims

1. 1. A method for processing data relating to the operation of a mobile asset, comprising: one or more data sources on-board the mobile asset, the data sources having fixed cameras and / or 360-degree cameras, the data received by a video analytics system on-board the mobile asset comprising live video and / or image data relating to the mobile asset; and / or one or more remote mobile asset data sources remote from the mobile asset, the remote mobile asset data sources having a weather component, a cargo manifest, a route / crew manifest, and / or geographic information system information received from a geographic information system (GIS), and wherein remote mobile asset data is received from the remote mobile asset data sources related to the operation of the mobile asset; receiving the data based on a data signal from the mobile asset using a video analytics system onboard the mobile asset; processing the live video and / or image data and / or the remote mobile asset data into processed data using a reinforcement learning and / or artificial intelligence component of the video analytics system; transmitting the processed data using the video analytics system to a data recorder onboard the mobile asset; encoding the processed data into encoded data using a data encoder of the data recorder; storing the encoded data using an on-board data manager of the data recorder to a remote data repository at a first predetermined frequency, configurable to be greater than 0 seconds and less than or equal to 5 minutes; A method for providing

2. the remote mobile asset data received from the remote mobile asset data source includes global positioning system data and inertial sensor data; The method of claim 1.

3. The data received by the video analysis system comprises: Video information received from the fixed camera and / or the 360-degree camera; and / or audio information received from a microphone; having The method of claim 1.

4. using an object detection and location component of the video analysis system to identify first object detection data and / or first object location data for a first object based on the processed data; Using the object detection and location component, the first object detection data; and / or the first object position data under a first condition that the first object detection data is identified; and, the first object position data under a second condition that the first object position data is identified; identifying an internal condition for the mobile asset based on the The method of claim 1 further comprising:

5. using an obstacle detection component of the video analysis system to identify obstacle detection data based on the processed data and / or the first object detection data in a third condition where the first object detection data is identified and / or the first object position data in a fourth condition where the first object position data is identified; using the obstacle detection component to identify an external condition related to the mobile asset based on the obstacle detection data, the external condition including object detection and / or obstacle detection occurring outside of the mobile asset; The method of claim 4 further comprising:

6. The obstacle detection data is a fifth condition being the first object position data; and / or a sixth condition that the first object position data is identified; and / or the first object detection data under a seventh condition that the first object detection data is identified; Based on the obstacle detection data corresponds to second object position data and second object detection data for a second object; The method of claim 5.

7. the external condition includes truck detection under a condition in which the moving asset includes a train; The method of claim 5.

8. receiving the external condition and / or the obstacle detection data using the data recorder; encoding, using the data encoder of the data recorder, the second encoded data based on the external conditions and / or the obstacle detection data; storing the external conditions and / or the obstacle detection data and / or the second coded data in the at least one remote data repository at the configurable first predetermined frequency using the on-board data manager of the data recorder; The method of claim 7 further comprising:

9. Using the data recorder, said internal conditions, and / or the first object detection data under a third condition that the first object detection data is identified; and / or the first object position data under a fourth condition that the first object position data is identified; receiving the using the data encoder of the data recorder, said internal conditions, and / or the first object detection data under the third condition that the first object detection data is identified; and / or the first object position data under the fourth condition that the first object position data is identified; encoding the second encoded data based on the using the on-board data manager of the data recorder; said internal conditions, and / or the first object detection data under the third condition that the first object detection data is identified; and / or the first object position data and the second encoded data under the fourth condition that the first object position data is identified; at said configurable first predetermined frequency in said at least one remote data repository; The method of claim 4 further comprising:

10. receiving the external condition and / or the obstacle detection data using the data recorder; encoding, using the data encoder of the data recorder, the second encoded data based on the external conditions and / or the obstacle detection data; storing the external conditions and / or the obstacle detection data and / or the second coded data in the remote data repository at the configurable first predetermined frequency using the on-board data manager of the data recorder; The method of claim 5 further comprising:

11. transmitting the encoded data to the remote data manager via a wireless data link at a configurable first predetermined frequency using the onboard data manager, wherein the encoded data is continuously sent to the remote data manager at a second predetermined frequency between 0 and 1 seconds; storing the first record in the remote data repository using the remote data manager; The method of claim 1 further comprising:

12. 1. A system for analyzing video content, comprising: one or more 360-degree cameras and one or more fixed cameras; a video analytics system onboard a mobile asset, the video analytics system including a reinforcement learning and / or artificial intelligence component, an object detection and location component, and an obstacle detection component, and configured to receive data based on data signals from the one or more 360-degree cameras and / or the one or more fixed cameras; Equipped with one of the reinforcement learning and artificial intelligence components configured to process the data into processed data; the object detection and location component is configured to determine object detection data and object location data for a first object based on the processed data; the obstacle detection component is configured to determine obstacle detection data based on the processed data and / or the object detection data and / or the object position data; a data recorder onboard the mobile asset, the data recorder including a data encoder, an onboard data manager, one or more local memory components, and one or more remote data repositories, the data recorder configured to store the processed data, the object detection data, the object position data, and the obstacle detection data in the one or more remote data repositories at a first predetermined frequency, the first predetermined frequency being configurable to be greater than 0 seconds and less than or equal to 5 minutes; Further provided with system.

13. a vehicle data component onboard the mobile asset, the vehicle data component having a global positioning system (GPS) and an inertial sensor, the vehicle data component configured to transmit global positioning system data received from the global positioning system (GPS) and / or inertial sensor data received from the inertial sensor to the video analytics system; a weather component configured to transmit current weather information and / or forecasted weather information to the video analytics system; a route / occupant manifest and geographic information system (GIS) component configured to transmit route information and / or occupant information and / or route / occupant manifest information and / or GIS information to the video analytics system; Furthermore, the reinforcement learning and / or artificial intelligence component is configured to process the data into processed data using the global positioning system data and / or the inertial sensor data and / or the current weather information and / or the forecasted weather information and / or the route information and / or the occupant information and / or the route / occupant manifest information and / or the GIS information. The system of claim 12.

14. the data recorder onboard the mobile asset, the data recorder including a local memory and / or an onboard data manager and / or a data encoder, the data recorder configured to receive the processed data and / or the object detection data and / or the object position data and / or the obstacle detection data from the video analysis system; the data encoder configured to encode records based on at least one of the processed data, the object detection data, the object position data, and the obstacle detection data; the onboard data manager configured to store the records and / or the processed data and / or the object detection data and / or the object position data and / or the obstacle detection data in the one or more remote data repositories at a configurable first predetermined frequency; The system of claim 12 further comprising:

15. a remote data manager remote from the mobile asset, configured to receive first records from the on-board data manager over a wireless data link at the configurable first predetermined frequency, the remote data manager receiving the first records continuously at a second predetermined frequency between 0 and 1 seconds; the one or more remote data repositories configured to store the records received from the remote data manager; The system of claim 14 further comprising:

16. a data decoder remote from the mobile asset, the data decoder configured to receive the records from the one or more remote data repositories and decode the records; an external monitoring component remote from the mobile asset, the external monitoring component configured to identify the object detection data and / or the object position data and / or the obstacle detection data; The system of claim 15 further comprising:

17. a web client including a display device; a web server in wireless communication with the web client and configured to receive a request including specified data and a particular view mode regarding the mobile asset; a localization component in wireless communication with the web server, the localization component configured to receive from the data decoder the designated data based on the records and / or the processed data and / or the object detection data and / or the object position data and / or the obstacle detection data, and to modify the designated data based on a user-specified time setting and a user-specified unit of measure setting specified by a remote user; further comprising the web server is configured to receive the specified data; The system of claim 16 , wherein the display device is configured to display the specified data in the particular view mode.

18. a digital video recorder onboard the mobile asset, the digital video recorder configured to receive data based on data signals from the one or more 360-degree cameras and / or the one or more fixed cameras; Furthermore, the digital video recorder transmitting the data to the data encoder of the data recorder onboard the mobile asset; The system of claim 12.

Images