Systems and methods for bandwidth reduction and communication of visual events - Patents.com

Abstract

The present invention provides a system and method that employs data queuing as well as reduced bandwidth and stored data volume to detect and continuously communicate important and / or relevant vessel visual information from the vessel (or other remote vehicle / location) to shore within the available bandwidth of a typical available satellite link without losing useful / high priority information. The system and method supports remote configuration and management from shore to the vessel over the same communication channel, but in the reverse direction. Additionally, the system and method facilitates logical abstraction of continuous visibility into on-board activities, on-board actions, and on-board status, such that a ground / shore-based station manager / reviewer / auditor can use a ground user interface / dashboard to review, understand, and synthesize such information at a glance related to events that have recently occurred on a commercial cargo vessel at sea, without the need to review multiple channels of video over hours / days / weeks.

Description

[Technical field]

[0001] The present invention relates to a system and method for communicating visual data and associated events over a reduced bandwidth communication link, and a user interface for viewing and manipulating such data and events. [Background technology]

[0002] International shipping is an important part of the world economy. Ocean-going commercial cargo ships are employed to carry virtually all goods and materials between ports and between countries. The current approach to cargo transportation employs intermodal cargo containers that are loaded and unloaded from the deck of the ship and carried in a stacked configuration. Cargo is also transported in bulk carriers (e.g., grain) and liquid tankers (e.g., oil). Cargo ship operations can be dangerous and safety concerns are ever-present. Similarly, passenger ships, with their precious cargo of human beings, are just as concerned, if not more, about the safety of their operation and the adherence of crew and passengers to rules and regulations. Knowing the current status of the ship, crew, and cargo can go a long way in ensuring a safe and efficient operation.

[0003] In many areas of commercial and / or governmental activities, visual monitoring (manual and automated) and other status sensors are employed to ensure safe and compliant operation. However, these approaches involve the generation and transmission of large amounts of data to local or remote locations where such data is stored and / or analyzed by management personnel. However, unlike most land-based (i.e., wired, fiber, or broadband wireless) communication links, transmitting useful data (e.g., visual information) from the ship to shore is often much more difficult. It can be assumed that 10 channels of raw video data is the minimum number required to provide visibility of the ship, requiring an uplink speed of 5 Mb / s per HD video channel, or 50 Mb / s overall. Conversely, a typical satellite link to a ship is 1 / 200 or 1 / 400 the size and transmits only 256 Kb / s or 128 Kb / s.

[0004] It would be desirable to provide a system and method that allows continuous visibility of on-board activities, on-board behavior, and on-board status of ocean-going commercial vessels (cargo, fishing, industrial, passenger). Further, it would be desirable for the transmitted visual data and associated status to be accessible via an interface that assists users in operating, organizing, and acting on such information. Summary of the Invention [Means for solving the problem]

[0005] The present invention overcomes the shortcomings of the prior art by providing a system and method for detecting important and / or relevant vessel visual information and continuously communicating from the vessel (or other remote vehicle / location) to shore within the available bandwidth of a typical available satellite link without losing useful / high priority information, employing data queuing as well as bandwidth and data storage reduction. The system and method supports remote configuration and management from shore to the vessel, but in the reverse direction, over the same communication channel. Furthermore, the system and method facilitates a logical summary of continuous visibility into on-board activities, on-board actions, and on-board status, so that a land / shore manager / reviewer / auditor can at a glance review, understand, and synthesize information related to recent events occurring on a commercial cargo vessel at sea, without the need to review (for example) 10+ channels of video over hours / days / weeks, using a ground computer graphical user interface (GUI) that can be defined in a dashboard.

[0006] In an exemplary embodiment, a system and method is provided for storage and bandwidth reduction of visual events associated with activities. At least one visual event detector is provided and may include at least one camera that captures images of the activities. At least one processor receives the images and a visual event detection process operates at least in part on the processor to process the images. A queuing process identifies visual events from the at least one visual event detector and stores the visual events. A communication process enables a low bandwidth communication channel adapted to transmit the stored visual events to a remote location in a reduced bandwidth and queued format. Illustratively, the visual events may include activities related to at least one of bridge operations, cargo operations, maintenance operations, safety operations, and screening and security related to the vessel. The visual events may be based on software and hardware based event detectors that identify predetermined characteristics in one or more images and determine a status based on these characteristics. The visual events may provide data primitives from which user information is derived, such that these data primitives may include at least one of statistics, pattern matches, metrics, and time. The stored visual events may be communicated over the communication channel based on a priority set by a user, with higher priority events from the stored visual events being communicated first in the queue. The user interface may display at least one of information of bridge operations, cargo operations, maintenance operations, safety operations, and screening and security on the base station in one or more predefined presentation formats. The user interface may include an input that allows a user to set the priority. The user interface may include an input that allows adding or setting configuration parameters of the event detector. The user information may also include at least one of a fleet report for a plurality of vessels associated with the user interface, a vessel report, an occurrence of a discrete event, and an occurrence of a composite event including a plurality of discrete events.The information can be displayed in the form of a dashboard that includes one or more visual windows, alerts, and numerical data.

[0007] In an exemplary embodiment, a user interface is provided that is disposed on the base station and receives visual events over a communication channel from a remote object, moving or stationary, the object including at least one visual event detector having at least one camera that captures images of activity. At least one processor is provided that receives the images, and a visual event detection process operates at least in part on the processor to process the images. The communication process enables a communication channel adapted to transmit stored visual events to the base station in a queue format. The user interface includes a dashboard for displaying information related to the object. The user interface enables configuration of at least one of (a) a priority of visual events communicated over the communication channel, and (b) configuration of characteristics related to the visual events. The communication channel can be a reduced bandwidth wireless communication channel. The object can be an ocean-going vessel, and the visual events can relate to at least one of safety, maintenance, cargo handling, bridge operations, and screening and security related to the vessel. A dashboard display on the user interface can provide at least one of configuration, status, and performance information related to the vessel. A dashboard display on the user interface can provide at least one of configuration, status, and performance information related to each vessel of the vessel fleet. Illustratively, the information includes one or more forms of displayed sensor data having at least one of labels, highlights, comments, and annotations visually attached thereto. The information processed and / or displayed by the interface may include workflows established between one or more users or one or more vessels. A communication channel may be arranged to transmit control data from the user interface to one or more vessels, and to receive feedback via the user interface relating to operations on board the vessel regarding the control data.

[0008] In an exemplary embodiment, a method is provided for storing and bandwidth reducing visual events associated with an activity. The method includes acquiring images of the activity with at least one camera and performing visual event detection with a processor. Visual events from the at least one visual event detector are identified and stored in a queue based on predetermined parameters. Communication with a remote location is via a low bandwidth communication channel adapted to transmit the stored visual events to the remote location in a reduced bandwidth and queued format.

[0009] The following description of the invention refers to the accompanying drawings, in which: [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an overview of a system and associated method for acquiring, transmitting, analyzing, and reporting visual and other sensor information over a communication link typically characterized by reduced bandwidth, in accordance with an exemplary embodiment.

[0011] [Figure 1A] 2 is a block diagram showing data and operations employed by the system and method of FIG. 1;

[0012] [Figure 1B] FIG. 2 illustrates the acquisition of images and other data for anticipated event detection according to the system and method of FIG. 1;

[0013] [Diagram 2] FIG. 2 is a flow diagram illustrating detection and reporting of visual events and associated data by the processors and processes of the system and method of FIG. 1.

[0014] [Diagram 3] FIG. 2 is a flow diagram illustrating detection of visual events using the processor and processes of the system and method of FIG. 1 in an example bridge routine of an ocean-going commercial cargo ship.

[0015] [Figure 4] FIG. 2 is a flow diagram illustrating detection of visual events using the processor and processes of the system and method of FIG. 1 in the example of the performance of a safety patrol by personnel (crew) of an ocean-going commercial cargo ship.

[0016] [Diagram 5] FIG. 2 is a flow diagram illustrating detection of visual events using the processors and processes of the system and method of FIG. 1 in the example of performing cargo handling activities on an ocean-going commercial cargo ship.

[0017] [Figure 6] FIG. 2 illustrates an example graphical user interface (GUI) displayed by the processor and associated processes in the system and method of FIG. 1 , depicting an example dashboard of vessel fleet activity and status.

[0018] [Figure 7] FIG. 2 illustrates an exemplary GUI displayed by the processor and associated processes in the system and method of FIG. 1, depicting an exemplary dashboard of shipboard activities including current status and performance.

[0019] [Figure 8] FIG. 2 illustrates an exemplary GUI displayed by the processor and associated processes in the system and method of FIG. 1, depicting another exemplary dashboard of shipboard activities including current status and performance.

[0020] [Figure 9] FIG. 2 illustrates an example GUI displayed by the processor and associated processes in the system and method of FIG. 1, depicting an example dashboard for tracking cargo handling procedures on a ship.

[0021] [Figure 10]FIG. 2 illustrates an example GUI displayed by the processor and associated processes in the system and method of FIG. 1 , depicting an example dashboard for tracking maintenance activities on a ship.

[0022] [Figure 11] FIG. 2 illustrates an exemplary GUI displayed by the processor and associated processes in the system and method of FIG. 1 , depicting an exemplary dashboard for on-board bridge activity / action tracking.

[0023] [Figure 12] FIG. 2 illustrates an example GUI displayed by the processor and associated processes in the system and method of FIG. 1 , depicting an example dashboard for safety status and procedure tracking on a ship.

[0024] [Figure 13] FIG. 2 illustrates an example GUI displayed by the processor and associated processes in the system and method of FIG. 1, depicting an example dashboard for cargo handling procedure reporting on a ship.

[0025] [Figure 14] FIG. 2 illustrates an example GUI displayed by the processor and associated processes in the system and method of FIG. 1, depicting an example dashboard for ship maintenance activity reporting.

[0026] [Figure 15] FIG. 2 illustrates an exemplary GUI displayed by the processor and associated processes in the system and method of FIG. 1 , depicting an exemplary dashboard for on-board bridge activity / action reporting;

[0027] [Figure 16] FIG. 2 illustrates an example GUI displayed by the processor and associated processes in the system and method of FIG. 1, depicting an example dashboard for shipboard safety status and procedure reporting. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] I. System Overview

[0029] The arrangement 100 shown in Figures 1 and 1A tracks and reports visual and other events generated by on-board visual sensors creating video data streams, visually detects on-board events based on those video data streams, aggregates those on-board visual detections, prioritizes and queues the aggregated event detections, combines the aggregated events to reduce the bandwidth of the video data streams, transmits the events to shore over a low bandwidth communication channel, reports the events to a shore user interface, and further aggregates events from multiple vessels and multiple time periods into fleet-wide aggregates that can present information over time. The systems and methods described herein further provide the ability to configure and set the above-mentioned systems to select or not select events to present to reduce confusion for personnel viewing the dashboard, and to set priorities for communicating certain events or classes of events over a reduced bandwidth communication channel such that the most important events are communicated at the expense of less important events.

[0030] The arrangement 100 of FIG. 1 specifically represents a shipboard location 110, which includes a camera (visual sensor) array 112 including a plurality of discrete cameras 118 (and / or other suitable environmental / event driven sensors), which are connected to a wired and / or wireless communication link (e.g., part of a TCP / IP LAN or other protocol driven data transmission network 116) via one or more switches, routers 114, etc. Image (and other) data from the (camera) sensors 118 is transmitted over the network 116. It should be noted that the cameras may provide analog or other forms of image data to a remote receiver that generates digitized data packets for use by the network 116. The cameras 118 may include conventional machine vision cameras or sensors that operate to collect raw video or digital image data, which may be based on two-dimensional (2D) and / or three-dimensional (3D) imaging. Furthermore, the image information may be grayscale (monochrome), color, and / or near-visible (e.g., infrared (IR)). Similarly, other forms of event-based cameras may be employed.

[0031] It should be noted that data, as used herein, can include both direct feeds from suitable sensors and data feeds from other data sources that may aggregate various information, telemetry, etc. For example, location and / or direction information can be obtained from a navigation system (e.g., GPS) or other systems (e.g., via an API) through an associated data processing device (e.g., computer) networked with the system server 130. Similarly, crew members can input information through a suitable user interface. The interface can request specific inputs, such as logging in or out of a shift, providing health information, etc., or the interface can retrieve information entered by crew members during normal operations and determine when crew members are entering data in the normal course of ship operations, for example to ensure that proper procedures are being implemented in a timely manner.

[0032] The shipboard location 110 may further include a local image / other data recorder 120. The recorder may be a standalone unit or may be part of a broader computer server configuration 130 with appropriate processors, data storage, and network interfaces. The server 130 may use appropriate software to perform generalized or specialized ship to ship operations of the systems and methods described herein. The server 130 may communicate with a workstation or other computing device 132, which may include a suitable display (e.g., touch screen) 134 and other components to provide a graphical user interface (GUI). The GUI provides the shipboard user with a local dashboard for viewing and controlling operations of the event data generated by the sensors 118, as described further below. It should be noted that displaying and manipulating the data may include, but is not limited to, augmenting the displayed data (e.g., images, videos, etc.) with labels, comments, flags, highlights, etc.

[0033] The information processed and / or displayed by the interface may include a workflow established between one or more users or one or more vessels. Such a workflow may be a business process where information is transferred from user to user (onshore or at sea interacting with the application via a GUI) for action according to business procedures / rules / policies. The automation of this workflow is commonly referred to as "Robotic Process Automation".

[0034] Processes 150 that perform dashboards and other data processing operations in the systems and methods may be executed in whole or in part on the on-board server 130 and / or using a remote computing (server) platform 140 that is part of a ground based or other generally fixed location (base station 142) with sufficient computing / bandwidth resources. The processes generally include computational processes 152 that process sensor data into meaningful events. This may include machine vision algorithms and similar procedures. Data processing processes 154 may be used to derive events and associated status based on the events, e.g., crew and equipment movements, cargo operations, etc. Information processes 156 may be used to drive one or more vessel dashboards and provide both status and operation of the data to on-board and base station users.

[0035] Data communication between the ship (or other remote location) 110 and the base station 142 occurs over one or more low bandwidth radio channels and may be facilitated by satellite uplink / downlink 160, or other transmission modalities, such as long wavelength airborne transmission. Additionally, other forms of wireless communication may be employed, such as mesh networks and / or underwater communications (e.g., acoustic-based long range communications and / or VUF). It should be noted that if the ship is located near a high bandwidth channel on land or is physically connected by wires while in port, the systems and methods described herein may be adapted to utilize that high bandwidth channel to transmit all previously untransmitted low priority events, alerts, and / or image-based information.

[0036] The (land-based) base station environment 140 communicates via a suitable secure and / or encrypted link (e.g. LAN or WAN (Internet)) 162 with a user workstation 170, which may include a computing device having a suitable GUI configuration, defining a user dashboard 172 enabling the user to monitor and operate one or more vessels in the fleet for which he is responsible and manages.

[0037] With further reference to FIG. 1A, the data processed by the system is shown in further detail. The data acquired in the shipboard environment 110 and provided to the server 130 can include multiple possible detected visual (and other sensor-based) events. These events can be generated by the operation of software and / or hardware-based detectors that analyze visual images and / or time sequences of images acquired by the cameras. With further reference to FIG. IB, visual detection is facilitated by multiple 2D and / or 3D camera assemblies represented as cameras 180 and 182 using ambient or secondary illumination sources 183 (visible and / or IR). The camera assemblies image a scene 184 located on the ship (for example). The scene can relate to maritime events, personnel safety and / or cargo, among others. The images are sent as image data to an event detection server or processor 186, which also receives input from a plan or program 187 characterizing the events and event detection, and from a clock 188 that verifies the timeline and timestamp of the received images. The event detection server or processor 186 may also receive input from a GPS receiver 189 to stamp the location of the vessel at the time of the event, and may also receive input from architectural drawings 190 of the vessel (mapping on-board locations on various decks) to stamp the location of sensors within the vessel that sent the input. The event server / processor 186 may have one or more types and / or architectures of processors, including, but not limited to, a central processing unit (CPU, e.g., one or more processing cores and associated computation units), a graphical processing unit (GPU operating on a SIMD or similar configuration), a tensor processing unit (TPU), and / or a field programmable gate array (FPGA with a generalized or customized architecture).

[0038] 1A , a base station dashboard 172 is established for each vessel and / or fleet and communicates with the on-board server 130 via a communication link 160, typically in a reduced bandwidth and possibly intermittent manner performing data transfer operations. The link 160 transmits event and status updates 162 from the on-board server 130 to the dashboard 172, and event priorities, camera settings and vision system parameters 164 from the dashboard 172 to the on-board server. More specifically, the dashboard allows display and manipulation of event reports and logs 173, alarm reports and logs 174, event priorities 175, etc., camera setup 176, vision system task selection and setup 177 related to event detection, etc. The shipboard server 130 has various functional modules, including visual event bandwidth reduction 132 to facilitate transmission over link 160, alarm and status polling and queuing 133 to determine when an alarm or various status items has occurred and transmit them with the appropriate priority, priority setting 134 to select reporting and transmission priorities, and data storage 135 to hold images and other associated data from a given period of time.

[0039] In various embodiments, the bandwidth of the communication link between the vessel and the base station may be limited in usage by an external system, such as a router / link QoS (Quality of Service) setting, or by an internal system (edge ​​server 130), for example to 15% (for example) of the total available communication bandwidth. This bandwidth limitation may be based on a variety of factors, including, but not limited to, time of day and / or communication satellite usage cost schedule. A suitable set of instructions may be programmed into the server using conventional or custom control processes. The specific settings of such bandwidth control may also be instructed by the user via the GUI.

[0040] II. Vision Detector

[0041] As shown in FIG. IB, the various imaged events are determined from the acquired image data using suitable processes / algorithms 188 executed by the processor 186. These may include classical algorithms that are part of conventional vision systems available from (for example) Keyence, Cognex, MVTec, or HIKVision. Alternatively, the classical vision system could be based on open source such as OpenCV. Such classical vision systems may include various vision system tools including, but not limited to, edge finders, blob analyzers, pattern recognition tools, etc. The processor 186 may also employ machine learning or deep learning algorithms, which may be custom built or commercially available from various sources, employing suitable deep learning frameworks such as caffe, tensorflow, torch, keras, and / or OpenCV. The network could be a Mask R-CNN or a Yolov3 detector. See also URL addresses. https: / / engineer.dena.com / posts / 2019.05 / survey-of-cutting-edge-computer-vision-papers-human-recognition / on the World Wide Web

[0042] 1A, visual detectors relate to marine events 191, crew safety actions and events 192, maintenance actions and events 193, cargo status and related events 194, and / or non-visual alarms such as smoke, fire, and / or toxic gas detection by appropriate sensors. By way of non-limiting example, some specific detected events and associated detectors relate to: (a) Personnel are at their stations at the scheduled times and report their stations, start times, end times, and elapsed time. (b) Personnel enter the location at the scheduled time and report the location, start time, end time, and elapsed time. (c) Personnel pass through locations at scheduled times and report the location, start time, end time, and elapsed time. (d) Personnel perform scheduled activities at scheduled times and locations and report the location, start time, end time, and duration. Activities may include (for example) surveillance, monitoring, installation, connecting or disconnecting hoses, crane operations, rope tying, etc. (e) Personnel run, slip, trip, fall, lie down, use handrails or fail to use handrails at scheduled times and in scheduled locations and report the location, start time, end time, and duration. (f) Personnel performing scheduled activities at scheduled times, in scheduled locations, with or without protective equipment, including (for example) helmets, gloves, shoes / boots, hearing protection, safety goggles, life jackets, gas masks, welding masks, or other protective gear, reporting the location, start time, end time, and duration. (g) Doors open and close at their locations at the scheduled times, reporting the location, start time, end time, and elapsed time. (h) The presence of objects at locations at scheduled times, reporting their location, start time, end time, and elapsed time. Objects may include (for example) walkways, hoses, tools, ropes, cranes, boilers, pumps, connectors, solids, liquids, small boats, and / or other unknown items. (i) Normal work activities are performed using at least one of the following: engines, cylinders, hoses, tools, ropes, cranes, boilers, and / or pumps. (j) Necessary maintenance activities are performed on engines, cylinders, boilers, cranes, steering mechanisms, HVAC, electrical, pipes / plumbing, and / or other systems.

[0043] It should be noted that the above list of examples (a-j) is only a portion of the wide range of possible tasks that may be the basis for detectors according to exemplary embodiments herein. Those skilled in the art should understand that other detectable events are expressly contemplated, including human-to-human, human-to-machine, or machine-to-machine interactions.

[0044] In operation, the visual detector of expected events takes as input the detection results of one or more vision systems on the ship. The result could be detection, non-detection, or anomaly at the time of the expected event according to the plan. Multiple events or detections could be combined into a single higher-level event. For example, a maintenance procedure, a cargo handling activity, or an inspection round could result from combining multiple events or detections. It is noted that each visual event is associated with a specific image or video sequence of a specific (or multiple) vision system camera 118, 180, 182 at a specific time and at a known location within the ship. The associated video may or may not be optionally transmitted with each event or alert. If a video is transmitted with the event or alert, it may be useful for post-event verification of the event or alert. It is noteworthy that the discrete images and / or short-time video frame sequences are actually a small portion of the video stream, resulting in a significant reduction in the bandwidth required for transmission compared to the entire video sequence over a low-bandwidth link. Furthermore, in addition to compacting the video by reducing to a few images or short duration sequences, the system can reduce the size of the image by cropping the image to the critical or meaningful image locations required by the detector, or by reducing the resolution from approximately the equivalent of high definition (HD) resolution to standard definition (SD) resolution or below standard resolution.

[0045] In addition to reducing bandwidth by identifying events by the vision system and cropping such images if appropriate, the number of image frames can be reduced by increasing the time interval between frames. Furthermore, using the above procedure, bandwidth can be further reduced by cropping and spacing around events (all on-board server side) using commercially available or customized lossy or lossless image compression techniques. Such techniques can include, but are not limited to, Discrete Cosine Transform (DCT), Run Length Encoding (RLE), Predictive Coding, and / or Lempel-Ziv-Welch (LZW).

[0046] Images or video sequences that are not associated with a visual event may be stored for a period of time on board the ship.

[0047] The on-board server establishes a priority for the transmission of processed visual events, usually based on settings provided by the user operating the onshore (base station) dashboard. The on-board server buffers these events in a queue in storage where they can be ordered based on priority. Priority can be set based on various factors. For example, personnel safety and / or ship safety may be given the highest priority, while maintenance may be placed last in the priority order, typically mapping to the urgency of such matters. As an example, all events in the highest priority queue are sent first, followed by lower priority events. If a new event arrives on the ship with a higher priority, this new high priority event is sent before the lower priority events. It is assumed that if the high priority events use all the available bandwidth, the lower priority events are dropped. The on-board server receives an acknowledgement from the on-shore base station server to confirm that the event has been received and acknowledged on shore before marking the on-board event as sent. Multiple events may be sent before receiving an acknowledgement (or lack of receipt). A lack of response may stall the queue or require a retransmission of the event before sending all subsequent events in the priority queue on the server. The land-based server interface can configure or select visual event detectors over the communications link. In addition to visual events, the system can send non-visual events such as fire and smoke alarm signals.

[0048] III. Detection Flow

[0049] As shown in FIG. 2, a generalized detection flow operational procedure 200 is shown for use in implementing the system. The operation can be characterized in three stages or segments: computation 210, generation of data primitives 220, and information creation 230 and presentation to the user via the land-based dashboard 240. Alternatively, some or all of the functionality described herein can be implemented by the user via the on-board dashboard, which affects programming of the local server and / or the base station server. The on-board dashboard can also function as a passive terminal that transmits instructions back to the base station interface via a communication link, and such instructions can be executed through the base station. The computation stage 210 includes performing measurements 212 using sensors and visual detection 213. These generate a set of metrics 222 that are displayed to the user as discrete events 232. The computation stage 210 provides pattern matching 224 according to the time series of events 226 using event ordering (prioritization) 214, filtering (by trimming, compression, etc.), and event qualification 216 based on rules 217. This data is presented as composite events 234. These composite events 234 can include scenarios such as maintenance actions successfully performed or the occurrence of a safety violation. The computation stage 210 can aggregate the visual and other events 218 to derive statistics 228, such as the number of safety violations over a given time interval. These statistics 228 can be presented to users on shore as individual vessel reports 236 and fleet reports 238, providing the users with valuable information regarding the behavior and performance of various factors related to the aggregated events.

[0050] 3 illustrates a detection flow procedure 300 for an example bridge routine for one or more vessels in a fleet. In a computation stage 310, sample detectors 312 provided by vision and other detectors detect (for example) personnel crossing or stopping at a location, personnel operating equipment, personnel walking, personnel sitting, personnel not moving (stationary), personnel staring at a location, personnel wearing earphones at a location, and / or lights off. In an associated data primitive generation stage 320, detected sample metrics 322 are provided, including (for example) start and end times, duration, number of participants, bridge stations visited, protocol steps performed, and non-compliance with protocols. Event samples 324 can include the names of participants identified to perform a shift, when the shift started, whether a given participant's shift was longer or shorter than normal, understaffing and / or overstaffing / unauthorized personnel on the bridge. In an exemplary information stage, a sample report 332 is created which may include (for example) shift times over time, shift participation (number of people), statistics of time operating equipment, distribution, e.g., number of shifts x hours, and location graphs (e.g., heat maps) which may be based on month, week, day, etc. In the information stage 330, the sample report 332 may be presented as a vessel report 334 and a fleet report. The detected sample metrics 322 and event samples 324 may be presented to the user as discrete events 338 and composite events 339.

[0051] 4 illustrates a detection flow procedure 400 for an example safety patrol for one or more vessels in a fleet. In a computation stage 410, sample detectors 412 provided by the vision and other detectors include (for example) location of an event, personnel operating equipment, personnel stopping at a location, personnel walking or staring at a location, helmets, personnel wearing life jackets or other protective equipment, and / or personnel holding safety tools such as fire extinguishers, flashlights, etc. In a data primitives stage 420, detected sample metrics can include (for example) start or end time of an event, duration, number of participants, stations visited, protocol steps performed, and / or safety patrol protective equipment (PPE) employed. The event samples 424 may include safety patrols not performed for a predefined number of hours, patrols taking X% longer or shorter than normal, patrols performed by X people, patrols started X minutes late, patrols performed without required PPE, and / or patrols completed in X minutes. The information stage 432 provides sample reports 432 based on events, including duration over time, participation, adherence to safety protocols, station time, distribution (e.g., number of patrols x duration), and / or graphs / heat maps based on month, day, week, etc., including vessel reports 434 and fleet reports 436. The information stage 430 also reports discrete events 438 and composite events 439 based on the detected sample events 422 and event samples 424.

[0052] FIG. 5 illustrates a detection flow procedure 500 for an example of a cargo handling operation of one or more vessels in a fleet. In the computation stage 510, sample detectors 512 may include pipes connecting, pipes disconnecting, personnel operating equipment, personnel standing, personnel arriving or departing, personnel wearing helmets, gloves, goggles, and / or other PPE. The data primitives stage 520 provides detected sample metrics 522, including start and end times, duration, number of personnel participating, protocol steps performed, and / or PPE employed in the task. Event samples 524 may include task completed in X minutes, task completion was X% longer or shorter than normal, task performed by a person, and / or task performed without (without) X type of PPE. In the information stage 530, sample reports 532 may include duration over time, participation, adherence to protocol, location / log, allocation (e.g., number of trainings x duration), and / or non-adherence to normal / standard operation. These may be presented as a vessel report 534 or a fleet report 536. The detected sample metrics 522 and event samples 524 are reported as discrete events 538 and composite events 539.

[0053] Other exemplary detection flows may be provided as needed to generate desired information regarding activities of interest by vessel personnel and systems, such detection flows employing associated detector types, parameters, etc.

[0054] IV. User Interface Dashboard

[0055] A. Status and Performance

[0056] FIG. 6 shows an exemplary Fleet Dashboard 600 display that can provide all key information to the user on a single display screen (which may be a single or multiple physical displays in a console arrangement). The exemplary display 600 can include multiple windows and numeric fields that provide an interactive experience to the user. An overall fleet status window 610 is displayed with, for example, alarms (e.g., red circles), cautions (e.g., yellow triangles), and cleared alarm status (e.g., green squares). Clicking can drill down to other specific windows. Similar status windows are provided across the top for cargo procedures 612, maintenance activities 614, bridge actions 616, and safety activities 618. A world (or selectable smaller region) map 620 is provided with vessels in the fleet flagged. Location can be determined using GPS or other suitable positioning technology. A list of recent activities 622 is also provided in the center of the display 600. A visual window 630 allows events of interest (e.g., recent events) to be selected and displayed. In this case, the events show locations on the ship and highlight people or other relevant activity 632. A filmstrip 634 is provided (e.g. above or below window 630) to allow events to be selected, for example in chronological order. Button 636 can be used to play an overview of the events and / or another play / pause button can be used to play the visual events. Horizontal time based (for example) shaded or color coded graphs (e.g. heat maps) can be provided to show status and / or activity at a fleet level for each of the cargo, maintenance, bridge operations and safety categories.

[0057] FIG. 7 illustrates a vessel status and performance display 700, selectable from the main fleet dashboard 600 or another mechanism. An exemplary status box 710 at the top right of the display 700 allows for the selection of a vessel from a menu 702 and includes the selected vessel name or identifier along with the number of alarms 712, warnings 714, and cleared alarms 716, respectively, and status conditions for cargo 720, maintenance 722, bridge actions 724, and safety precautions 726. A map 730 shows the location of the vessel. The 3D projection can be oriented and zoomed, and shows the location of events with identifiers 742 associated with sensors / cameras, and highlights 744 of locations with alerts / alarms. Selecting a particular sensor / camera displays status data of current and past activity in a central list 750. An image display window 760 shows images and sequences associated with the sensor / camera, which can be manipulated using the appropriate buttons. A graph or heat map 760 shows activity and status for various categories associated with the selected vessel.

[0058] 8 illustrates an additional view of a shipboard dashboard display 800 based on the above-described display 700 and the exemplary information provided therein. Accordingly, similar exemplary display elements are similarly numbered. The camera array window 740 of the display 700 has been switched to display a recent activity window 840 showing the activity (e.g., maintenance, cargo, bridge, or safety), date, time, and duration.

[0059] B. Event Tracking

[0060] 9-12 show respective displays 900, 1000, 1100 and 1200 accessible via an activity tracking side menu 902, which allow a user to track various activities with respect to multiple vessels. FIG. 9 in particular shows a display 900 allowing tracking of cargo operations. It includes filter boxes allowing selection of a tracking date range 910, an associated vessel 912 and a cargo terminal 914. An activity box 920 with vessel, terminal, time and duration is shown. Furthermore, the duration of various vessel cargo handling activities is shown in a comparative bar graph 930. A cargo procedure visual review setting 940 is also provided. The duration of a particular cargo handling procedure (e.g. gangway operation, hose connection, etc.) is shown in a segment window 950. Visual events related to the cargo handling activities can be displayed in window 960. A bar graph (histogram) 970 also shows the comparative duration over a series of days or weeks.

[0061] Figure 10 shows a maintenance activity tracking display 1000. The display includes a time filter 1010 and a map 1012 showing a selected vessel graphic and the locations being tracked thereon. An activity window 1020 shows the area of ​​interest and the start time, end time and duration. A time log 1030 shows comparative area activity over time as a series of bars. The location of the activity is shown on the vessel graphic in window 1040. A visual activity segments window 1050 is provided to allow selection of visual maintenance events for display in window 1060.

[0062] FIG. 11 shows a bridge behavior / activity tracking display 1100 including associated date range and vessel name filters 1110. For a given date range and time block, a chart 1112 is shown which can be shaded or color coded based on the level of activity. Activity can be selected for number of personnel or personnel movement by selection switch 1114. A map 1120 showing the bridge plan / layout of the vessel is represented by a heat map which is color coded or shaded based on the level of movement or personnel density (as selected above). This heat map can be projected onto a three-dimensional image from a particular bridge sensor / camera as shown in windows 1130 and 1132. An activity segment window 1140 is also provided which allows selection of visual events displayed in window 1150. A particular sensor / camera can be selected for display with switch 1152.

[0063] FIG. 12 shows a safety tracking display 1200 as a vessel map with associated time filters 1210, which allows selection of safety patrols and points of interest in specific areas, such as upper deck (and port, starboard, etc.), engine room, etc. An activity window 1220 provides area, safety patrol route, time, and duration information. A bar graph 1230 shows activity in a selected area (such as upper deck or engine room) over time. A visual review 1240 shows a map of a selected view (e.g. upper deck) and sub-area (e.g. port, starboard) of the vessel, as well as patrols and associated sensor / camera locations. A segment window 1250 also allows selection of visual events to display in window 1260.

[0064] C. Reports

[0065] 13-16 show displays for reports related to cargo procedures 1300, maintenance activities 1400, bridge actions / activities 1500, and safety 1600, respectively. A side menu 1302 allows for the selection of reports. Such reports can be created to provide users and managers with follow-on data to establish baselines and allow decisions to be made regarding personnel management, regulations, and operations. Referring to the display 1300 of FIG. 13, selection boxes allow for filtering based on a specific terminal location 1310, date range 1312, and vessel 1314. Total activities 1320 are listed with associated alarm, warning, and cleared alarm statistics. Overall performance 1322 is also listed with associated bar graphs. Plot graphs 1324 and histograms 1326, 1328 can also be displayed to chart various performance parameters as shown. A historical baseline chart of performance is also displayed in window 1330. A comparative chart (shaded or color coded) of two or more vessels in the fleet is depicted in window 140. The historical baseline may also be displayed as a pie chart 1350. It should be noted that any acceptable statistical graphic or display may be provided in alternative embodiments for this and other reports, provided that appropriate statistics and information exist to generate such a report. These reports may be generated using appropriate computational techniques that should be apparent to one of ordinary skill in the art.

[0066] Figure 14 shows a display 1400 for a maintenance activity report, including filters for date range 1410, vessel 1412, and location (upper deck, engine room, etc.) 1414. A map 1416 of the associated vessel deck is displayed with deck highlights 1418 of areas of interest. An overall activity box 1420 reports alarm, warning, and cleared conditions. An overall performance box 1422 includes a bar indicating performance levels. A bar graph (histogram) 1424 shows performance by vessel. An activity performance window 1440 plots maintenance by hour against the date range, using a color-coded or shaded scale to indicate the level of activity.

[0067] FIG. 15 shows a display 1500 for generating reports on bridge actions / activity, which includes filters for a date range 1510 and at least one vessel 1512. The display provides a window showing bridge activity sequences with alarm, warning and cleared status. A window 1522 is also provided showing overall performance with indicator bars. A table 1524 may show comparative performance between vessels in the fleet and a plot 1526 of performance against time may be provided. Additionally, the current and past performance of each vessel may be provided in a bar graph (histogram) 1530. Also, a comparative plot 1540 of bridge activity against time may be drawn for multiple fleet vessels.

[0068] FIG. 16 shows a display 1600 for creating a safety report, including associated filters based on date range 1610, vessel 1612, and report type 1614 (e.g., safety patrols, PPE usage, etc.). An overall activity window 1620 (e.g., 7 days) lists alarm, warning, and cleared status. An overall performance window 1622 using indicator bars is also displayed. Baseline safety activity by minute is shown in a bar graph (histogram). Historical performance among the vessels in the fleet is also shown in a bar graph 1632. Hourly safety performance for a date range is displayed in window 1640 as a color-coded or shaded plot according to level.

[0069] V. Conclusion

[0070] It will be apparent that the above described system and method effectively provides a user with meaningful information regarding the status and performance of various activities essential for the safe and efficient operation of the vessel. This information includes associated visual information that can be highly useful in understanding the activities, providing evidence for follow-up analysis, etc. The system and method can operate to deliver meaningful visual and other information in near real-time over low bandwidth communication channels. A graphical user interface can be provided to control the system and method, set priorities, track activities, and report if necessary. The system and method can also be adapted to communicate over high bandwidth channels and / or with other types of remote locations (islands, underground / subterranean, Arctic and Antarctic bases, space stations, etc.).

[0071] Exemplary embodiments of the present invention have been described above in detail. Various modifications and additions can be made without departing from the spirit and scope of the present invention. Features of each of the various embodiments described above can be combined with features of other described embodiments as appropriate to provide a variety of combinations in related new embodiments. Moreover, while a number of separate embodiments of the apparatus and method of the present invention have been described above, what has been described herein is merely illustrative of the application of the principles of the present invention. For example, as used herein, the terms "process" and / or "processor" should be taken broadly to include various electronic hardware and / or software-based functions and components (and which may alternatively be referred to as functional "modules" or "elements"). Furthermore, the described processes or processors can be combined with other processes and / or processors or divided into various sub-processes or processors. Such sub-processes and / or sub-processors can be variously combined in accordance with the embodiments of the present invention. Similarly, it is expressly contemplated that any functions, processes and / or processors herein can be implemented using electronic hardware, software consisting of a non-transitory computer-readable medium of program instructions, or a combination of hardware and software. In addition, the various orientation and / or configuration terms (and grammatical variations thereof) used herein, such as "vertical", "horizontal", "up", "down", "bottom", "top", "side", "front", "rear", "left", "right", "forward", "rear", and the like, are used only as relative descriptions and not as absolute orientations / configurations with respect to a fixed coordinate space, such as the direction of gravity. Furthermore, when the terms "substantially" or "approximately" are used with respect to a given measurement, value, or characteristic, it refers to an amount (e.g., 1-5%) that is within the normal operating range to achieve a desired result, but includes some variation due to inherent inaccuracies and errors within the tolerances of the system. Thus, this description is intended to be taken only as an example and not otherwise limiting the scope of the invention.

[0072] The claims are set forth below.

Claims

1. 1. A system for activity related visual event storage and bandwidth reduction, comprising: at least one visual event detector having at least one camera for capturing images of the activity, at least one processor for receiving the images, and a visual event detection process running at least in part on the processor for processing the images; a queuing process that identifies the visual events from the at least one visual event detector and stores the visual events; a communication process enabling a low bandwidth communication channel adapted to transmit the stored visual events to a remote location with reduced bandwidth and in a queued format; Including, the visual events include activities related to at least one of bridge operations and safety operations on the vessel, and the visual events are based on at least one of a software-based and a hardware-based event detector that identifies predefined characteristics within one or more images and determines a status based on the characteristics; The above system.

2. The system of claim 1, wherein the visual events provide data primitives that are derived from user information, the data primitives including at least one of statistics, pattern matches, metrics, and time.

3. 3. The system of claim 2, wherein the stored visual events are communicated over the reduced bandwidth communication channel based on a priority set by a user, with higher priority events from the stored visual events being communicated first in a queue.

4. The system of claim 3 , further comprising a user interface for displaying the at least one of bridge operation and safety operation information on the base station in one or more predefined presentation formats.

5. The system described in claim 4, wherein the visual events further include activities related to at least one of loading and unloading operations, maintenance operations, and inspection and security, and the system has a user interface that displays information about at least one of loading and unloading operations, maintenance operations, and inspection and security in one or more predetermined presentation formats on the base station.

6. The system of claim 4, wherein the safety operations include at least one of personnel operating equipment, personnel standing at a location, personnel walking or staring at a location, personnel wearing helmets, life jackets or other protective equipment, and personnel with safety tools.

7. The system of claim 4, further comprising a server on board the vessel that interacts with the communication process and receives at least one of the vessel's position and direction information navigation systems.

8. The system of claim 4 , wherein the user interface includes an input that allows a user to set priorities.

9. The system of claim 4, wherein the visual event detector detects small boats at a specific location, start time, and end time.

10. 5. The system of claim 4, wherein the user information includes at least one of a fleet report for a plurality of vessels associated with the user interface, a vessel report, an occurrence of a discrete event, and an occurrence of a composite event including a plurality of discrete events.

11. The system of claim 10 , wherein the user information is displayed in a dashboard format including one or more visual windows, alerts, and numerical data.

12. The system of claim 11, wherein the base station comprises a land server that receives the stored visual events through the communication process, processes the stored visual events, and displays information related to the visual events in a predetermined format on a remote user interface.

13. The system of claim 12, wherein the user information is displayed in a graphical format with respect to the vessels and vessel fleet.

14. The system of claim 13, wherein the user information includes statistics regarding the vessel and vessel fleet.

15. The system of claim 14, wherein the statistics are displayed as graphs over time.

16. The system of claim 14, wherein the user interface includes a selector for accessing statistics regarding each of the ships and ship fleets.

17. The system of claim 4, wherein the one or more predetermined presentation formats include a graph versus time.

18. A user interface operated by a base station processor located at a base station and receiving visual events from a remote object, whether moving or stationary, via a communication channel. the object includes at least one camera for capturing images of the activity, at least one object processor for receiving the images, and at least one visual event detector having a visual event detection process running at least in part on the object processor for processing the images, and a communications processor enabling a communications channel adapted to transmit stored visual events in a queue to the base station; the user interface includes a dashboard display for displaying information related to the object; the visual events include activities related to at least one of bridge operations and safety operations on the vessel, and the visual events are based on at least one of a software based event detector and a hardware based event detector that identifies predetermined characteristics within one or more of the images and determines a status based on the characteristics. The above user interface.

19. 20. The user interface of claim 18, wherein the communication channel is a reduced bandwidth wireless communication channel.

20. 20. The user interface of claim 19, wherein the object is an ocean-going vessel and the visual events relate to at least one of safety and bridge operations relating to the ocean-going vessel.

21. A user interface as described in claim 20, wherein the dashboard display on the user interface provides at least one of configuration, status, and performance information related to the ocean-going vessel.

22. The user interface of claim 20 , wherein the dashboard display on the user interface provides at least one of configuration, status, and performance information relating to each vessel in a vessel fleet.

23. The user interface of claim 20, wherein the dashboard display on the user interface provides at least one of configuration, status, and performance information related to a collective vessel fleet.

24. 20. The user interface of claim 19, wherein the information includes one or more forms of displayed sensor data having at least one of labels, highlights, comments, and annotations visually attached thereto.

25. The user interface of claim 19, wherein the visual event includes detection of a small boat at a particular location, start time and end time.

26. 1. A method for activity related visual event storage and bandwidth reduction, comprising: The method includes capturing images of the activity with at least one camera and performing visual event detection with a processor; The method further comprises identifying the visual events from a software-based event detector and a hardware-based event detector that identify predetermined characteristics in one or more of the images, determining a status based on the characteristics, and storing the visual events in a queue based on predetermined parameters, where the visual events include activities related to at least one of bridge operations and safety operations on an ocean-going vessel; and The method further includes communicating with a remote location over a low bandwidth communication channel adapted to transmit the stored visual events to the remote location with reduced bandwidth and in a queued format. The above method.

Images