System and method for risk assessment of marine vessels in response to marine visual events

Abstract

Systems and methods are provided for assessing marine vessel risk in response to automatically detected marine visual events. At least one marine visual event is detected by at least one camera on a vessel, which provides image data of the visual event to a processor. The visual event may be associated with at least one of safety, security, maintenance, crew actions, and cargo. A risk assessment score is generated in response to the detected visual event, and the risk assessment score is provided to a user in a desired format. Generating the risk assessment score may involve comparing the visual event to conforming or non-conforming model visual event data from data storage. Risk assessment scores may be aggregated from multiple events and / or fleets of vessels to generate an overall score for the vessel and fleet.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION This application relates to systems and methods for detecting and communicating visual data and related events in a transportation environment.

[0002] Background of the Invention International shipping is an important part of the global economy. Ocean-going commercial cargo ships are used to transport virtually all products and goods between ports and countries. Current cargo transportation methods utilize intermodal cargo containers, which are loaded and unloaded from the ship's deck and carried in stacked configurations. Cargo is also transported by bulk carriers (e.g., grain) or liquid tankers (e.g., oil). Commercial ship operations can be dangerous, and safety concerns are always present. Similarly, passenger ships, which transport valuable personnel, are equally, if not more, concerned about operational safety and crew and passenger compliance with regulations and laws. Awareness of the current condition of the ship, crew, and cargo can be extremely helpful in ensuring safe and efficient operation.

[0003] Commonly-owned U.S. Patent Application No. 17 / 175,364, filed February 12, 2021, by Naslavsky, Ilan et al., entitled "SYSTEM AND METHOD FOR BANDWIDTH REDUCTION AND COMMUNICATION OF VISUAL EVENTS," teaches a system and method for addressing bandwidth limitations in certain remote transportation environments, such as ships at sea, and is incorporated herein by reference for useful background information. While it is desirable in many areas of commercial and / or governmental activity to use visual and other condition sensors to enable visual monitoring (both manual and automated) to ensure safe and compliant operations, these approaches inevitably involve the generation and transmission of large amounts of data to local or remote locations, where such data is stored and / or analyzed by management personnel. Unlike most land-based (i.e., wired, fiber optic, or high-bandwidth wireless) communication links, transmitting useful data (e.g., visual information) from ship to shore is often much more difficult. The incorporated U.S. application teaches a system and method that enables continuous visibility into onboard activities, actions, and conditions of commercial vessels (cargo, fishing, industrial, and passenger ships) at sea, whereby transmitted visual data and associated conditions are made accessible via an interface that helps users manipulate, organize, and act on such information.

[0004] The ability to assign risk values / levels to events can help prioritize the severity of events and can be useful for various cost management activities, such as those related to insurance risk mitigation, which can help lower the rates charged to ship owners and operators. Traditionally, manual paper surveys have been used to collect information about ships and their operations. The results of these manual paper surveys have been used to create risk ratings for insurance (for example). Furthermore, highly skilled investigators and inspectors have been deployed to conduct these surveys (e.g., spending at least eight hours per year per ship). Given that automated event detection is contemplated by the above-incorporated applications, it would be desirable to automatically identify specific risks on a commercial vessel based on a marine visual event(s) detected on the vessel through a response(s) to such automatically detected event(s) on the vessel.

[0005] Summary of the Invention The present invention overcomes the shortcomings of the prior art by providing automated, real-time, near-real-time, and retrospective evidence-based reporting and assessment of risk in marine vessel operations, as well as automated support for risk assessments performed using other conventional techniques, including human / manual inspection of the vessel. Accordingly, a system and method for automatically assessing risk on a marine vessel in response to automatically detected marine visual events may include detecting at least one marine visual event with at least one camera mounted on the vessel, which provides image data of the visual event to a processor. The visual events may be associated with at least one of safety, security, maintenance, crew actions, and cargo. These visual events may be associated with broader categories, such as hull and machinery, cargo, and crew. A risk assessment score is generated in response to the detected visual event, optionally in broader categories, and provided to a user in a desired format. Generating the risk assessment score may involve comparing the visual event to compliant or non-compliant model visual event data from a data storage. Risk assessment scores can be aggregated from multiple events and / or fleets of vessels to generate an overall score for the vessel and fleet of vessels.

[0006] In an exemplary embodiment, a system and method are provided for assessing marine vessel risk in response to automatically detected marine-based visual events. The system and method detect at least one marine visual event from a plurality of marine-based visual events acquired by at least one camera on the vessel, which provides image data of the visual event to a processor. The visual event is associated with at least one of safety, security, maintenance, crew actions, and cargo. A risk assessment score is generated in response to the detected at least one visual event. The risk assessment score may be provided to a user in a desired format. By way of example, the risk assessment score may be generated by comparing the at least one visual event with data of conforming or non-conforming model visual events from a data storage, and establishing a score based on a level of match between the at least one visual event and the conforming or non-conforming model visual event. The comparison may be based on various processes, including neural networks and / or deep learning processes, running on a computer process. The risk can be related to at least one of (a) engine maintenance alerts, (b) cargo condition or operation, and (c) crew safety, security, and crew behavior (and / or other areas, such as areas related to the vessel and engine). Generating the risk assessment score can include comparing the at least one visual event to minimum standards associated with at least one of (a) the type of vessel or fleet of vessels, (b) cargo handling standards, and (c) safety standards (among other criteria apparent to those skilled in the art). Alternatively or additionally, generating the risk assessment score can include comparing the at least one visual event to relative standards associated with at least one of (a) the type of vessel or fleet of vessels, (b) cargo handling standards, and (c) safety standards (among other criteria apparent to those skilled in the art). The relative standards can be based on a predetermined number of standard deviation(s) from a mean value. Additional information can be provided to a user by the systems and methods related to a risk assessment score consistent with that provided in a vessel risk survey.A plurality of marine-based visual events may be acquired by a camera on each of a plurality of vessels in the fleet, each providing image data for the plurality of visual events. The plurality of visual events may be associated with at least one of safety, security, maintenance, crew actions, and cargo and used to generate a risk assessment score in response to the detected visual events. The system and method then correlates the risk assessment scores to provide an overall risk assessment for the fleet of vessels. The risk assessment is organized into at least one of safety, security, maintenance, crew actions, and cargo and displayed on a user interface. Additionally, a risk assessment profile for each individual vessel in the fleet may be displayed on the user interface based on a user selection of that particular vessel from a menu.

[0007] The following description of the invention refers to the accompanying drawings. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an overview of a system and associated method for acquiring, transmitting, analyzing, and reporting visual and other sensor information over a communication link, in accordance with an exemplary embodiment.

[0009] [Figure 1A] 2 is a block diagram illustrating data and operations utilized by the system and method of FIG. 1;

[0010] [Figure 1B] 2 illustrates the acquisition of images and other data related to anticipated event detection according to the system and method of FIG. 1.

[0011] [Figure 2] 2 is a flow diagram illustrating the detection and reporting of visual events and associated data by the processors and processes of the system and method of FIG. 1;

[0012] [Figure 3]2 is a flow diagram illustrating the detection of visual events using the processor and processes of the system and method of FIG. 1 in an example bridge routine of a commercial marine vessel.

[0013] [Figure 4] 2 is a flow diagram illustrating the detection of visual events using the processor and processes of the system and method of FIG. 1 in the example of a safety patrol by a crew member of a commercial marine vessel.

[0014] [Figure 5] 2 is a flow diagram illustrating the detection of visual events using the processors and processes of the system and method of FIG. 1 in an example of performing cargo handling activities on a commercial marine vessel.

[0015] [Figure 6] 1 is a flow diagram illustrating a general procedure for generating risk assessments and reports from events.

[0016] [Figure 7] FIG. 2 illustrates an exemplary GUI displayed by the processor and associated processes in the system and method of FIG. 1 , showing an exemplary dashboard of a vessel risk profile based on visual events.

[0017] [Figure 8] FIG. 2 illustrates an exemplary GUI displayed by the processor and associated processes in the system and method of FIG. 1 , showing an exemplary dashboard of vessel risk profiles from visual events and fleet risk assessments based on aggregation of fleet and industry risk data.

[0018] Detailed Description I. System Overview

[0019] 1 and 1A illustrate a configuration 100 for tracking and reporting visual and other events generated by onboard visual sensors that create video data streams, visually detecting events onboard based on those video data streams, aggregating those visual detections onboard, prioritizing and queuing the aggregated detections into events, combining the aggregated events to optionally reduce the bandwidth of the video data streams, transmitting the events to shore over a reduced-bandwidth communication channel, reporting the events to an onshore user interface, and aggregating events from multiple vessels and multiple time periods into fleet-wide aggregations that can display information over time. The systems and methods herein further provide the ability to configure and set the above system to select or deselect events for display and to prioritize the communication of specific events or classes of events to reduce confusion for those viewing the dashboard. Such communication may occur over a reduced-bandwidth communication channel, as needed, so that the most important events are communicated at the expense of less important events.

[0020] In FIG. 1 , configuration 100 specifically depicts a shipboard location 110 including a camera (visual sensor) array 112 including multiple discrete cameras 118 (and / or other suitable environmental / event-driven sensors) connected via wired and / or wireless communication links (e.g., part of a TCP / IP LAN or other protocol-driven data transmission network 116) via one or more switches, routers, etc. 114. Image (and other) data from the (camera) sensors 118 is transmitted over the network 116. It is noted that the cameras may provide analog or other formatted image data to a remote receiver that generates digitized data packets for use by the network 116. The cameras 118 may include conventional machine vision cameras or sensors that operate to collect raw video data or digital image data that may be based on two-dimensional (2D) and / or three-dimensional (3D) imaging. Furthermore, the image information may be grayscale (monochrome), color, and / or near-visible (e.g., infrared (IR)). Similarly, other types of event-based cameras may be utilized.

[0021] It is noted that data, as used herein, can include direct feeds from appropriate sensors and data feeds from other data sources that may aggregate various information, telemetry, etc. For example, location and / or direction information may be obtained from a navigation system (e.g., GPS) or other systems (e.g., via an API) via an associated data processing device (e.g., a computer) networked with the system's server 130. Similarly, crew members may input information through an appropriate user interface. The interface may require specific input, such as logging in or out of a shift or providing health information, or the interface may retrieve information that crew members enter during their normal duties (e.g., determining when crew members are entering data during normal operations on board to ensure appropriate procedures are followed in a timely manner).

[0022] The shipboard location 110 may further include a local image / other data recorder 120. The recorder may be a stand-alone unit or part of a broader computer server apparatus 130 equipped with appropriate processor(s), data storage, and network interfaces. The server 130 may employ appropriate software to perform general shipboard or specialized operations of the systems and methods herein. The server 130 communicates with a workstation or other computing device 132, which may include a suitable display (e.g., touch screen) 134 and other components that provide a graphical user interface (GUI). The GUI provides a shipboard user with a local dashboard for viewing and controlling the manipulation of event data generated by the sensors 118, as described further below. It is noted that displaying and manipulating data may include, but is not limited to, adding labels, comments, flags, highlights, etc. to displayed data (e.g., images, videos, etc.).

[0023] The information processed and / or displayed by the interface may include a workflow provided between one or more users or vessels. Such a workflow is a business process in which information is transferred from user to user (onshore or at sea interacting with an application via a GUI) for action according to business procedures, rules, and policies. This workflow automation can be implemented in a variety of ways, including through computer and network configurations, and in one embodiment may be referred to as "robotic process automation."

[0024] The process 150 that executes the dashboard and other data processing operations in the system and method may be executed, in whole or in part, using the onboard server 130 and / or using a remote computing (server) platform 140 that is part of a terrestrial or other generally fixed location (base 142) with sufficient computing / bandwidth resources. The process 150 generally includes a computational process 152 that processes sensor data into meaningful events, which may include machine vision algorithms and similar procedures. A data processing process 154 may be used to derive events and associated status based on the events (e.g., crew and equipment movements, cargo handling, etc.). An information process 156 may be used to drive one or more ship dashboards to provide both status and data manipulation to onboard and base users.

[0025] Data communication between the vessel (or other remote location) 110 and the base 142 occurs over one or more wireless channels, which may be facilitated by a satellite uplink / downlink 160 or another transmission method (e.g., long-wavelength radio transmission). Additionally, other forms of wireless communication, such as mesh networks and / or underwater communications (e.g., long-range acoustic and / or VLF), may also be utilized. Note that if the vessel is located near a land-based high-bandwidth channel or is physically wired while in port, the systems and methods herein may be adapted to utilize that high-bandwidth channel to transmit all low-priority events, alerts, and / or image-based information not previously transmitted.

[0026] The (shore) base server environment 140 communicates via a suitable, secure, and / or encrypted link (e.g., LAN or WAN (Internet)) 162 with a user workstation 170, which may include a computing device equipped with a suitable GUI configuration that defines a user dashboard 172 that enables the user to monitor and operate one or more vessels in the fleet of vessels for which the user is responsible and manages.

[0027] Further reference to FIG. 1A illustrates the data processed by the system in more detail. Data acquired in the vessel environment 110 and provided to the server 130 can include multiple possible detected visual events (and other sensor-based events). These events can be generated by the operation of software- and / or hardware-based detectors that analyze visual images and / or time series of images acquired by the cameras. Further reference to FIG. 1B, visual detection is facilitated by multiple 2D and / or 3D camera assemblies, shown as cameras 180 and 182, using ambient or secondary illumination sources 183 (visible and / or infrared). The camera assemblies capture images of a scene 184 located on the vessel (for example). The scene can relate to marine events, hull and machinery, crew safety, and / or cargo, among other subjects. The images are sent as image data to an event detection server or processor 186. The event detection server or processor 186 also receives input from a plan or program 187 that characterizes events and event detection, and a clock 188 that sets a timeline and timestamp for received images. The event detection server or processor 186 may also receive input from a GPS receiver 189 to stamp the location of the vessel at the time of the event, and from a vessel blueprint 190 (which maps the vessel's locations on various decks) to stamp the location of sensors within the vessel that sent the input. The event server / processor 186 may include processor(s) of one or more types and / or architectures, including, but not limited to, a central processing unit (CPU, e.g., one or more processing cores and associated computational units), a graphics processing unit (GPU, operating in a SIMD or similar configuration), a tensor processing unit (TPU), and / or a field programmable gate array (FPGA, having a general-purpose or customized architecture).

[0028] Referring again to FIG. 1A , a base dashboard 172 is established for each ship and / or fleet of ships and communicates with the shipboard server 130 via a communication link 160. The communication link 160 may perform intermittent data transfer operations, with reduced bandwidth as needed. The link 160 transmits event and status updates 162 from the shipboard server 130 to the dashboard 172, and event priorities, camera settings, and vision system parameters 164 from the dashboard 172 to the shipboard server. More specifically, the dashboard displays and enables operation of event reports and logs 173, alarm reports and logs 174, event priorities, etc. 175, camera settings 176, and vision system task selection and configuration related to event detection, etc. 177. The shipboard server 130 includes various functional modules, including visual event bandwidth reduction 132, which facilitates transmission over link 160; alarm and status polling and queuing 133, which determines when alarms or various status items occur and transmits them at the appropriate priority; priority setting 134, which selects the priority of reporting and transmission; and data storage 135, which maintains images and other related data from a predetermined period of time.

[0029] II. Visual Detection

[0030] As shown in FIG. 1B, various image events are determined from the acquired image data using appropriate processes / algorithms 188 executed by processor(s) 186. These may include classical algorithms that are part of conventional vision systems, such as those available from (for example) Keyence, Cognex Corporation, MVTec, or HIKVision. Alternatively, the classical vision system may be based on open source software such as OpenCV. Such classical vision systems may include various vision system tools, including, but not limited to, edge finders, blob analyzers, pattern recognition tools, and the like. The processor(s) 186 may also utilize machine learning or deep learning algorithms, either custom-built or commercially available from various sources, and may utilize appropriate deep learning frameworks such as Caffe, Tensorflow, Torch, Keras, and / or OpenCV. The network may be a Mask R-CNN or Yolov3 detector. See also the following URL address on the World Wide Web: https: / / engineer.dene.com / posts / 2019.05 / surver-of-cutting-edge-computer-vision-papers-human-recognition

[0031] 1A, visual detectors relate to marine events 191, vessel personnel safety actions and events 192, hull and machinery maintenance actions and events 193, cargo conditions and events related thereto 194, and / or non-visual alarms such as smoke, fire, and / or toxic gas detection by appropriate sensors. By way of non-limiting example, some specific detected events and associated detectors relate to the following: (a) A person is at their station at the scheduled time and reports their station, start time, finish time, and elapsed time. (b) A person enters a location at a scheduled time and reports the location, start time, end time, and elapsed time. (c) A person travels to a location at a scheduled time and reports the location, start time, end time, and elapsed time. (d) A person is performing a scheduled activity at a scheduled time at a scheduled location, reporting the location, start time, end time, and duration of the activity. Activities may include (for example) lookout, surveillance, installation, connecting or disconnecting hoses, operating a crane, and tying with ropes. (e) A person runs, slips, trips, falls, lies down, or uses or does not use a handrail in a location at a scheduled time, and reports the location, start time, end time, and elapsed time. (f) Whether or not a person is wearing protective equipment while performing a scheduled activity at a scheduled location at a scheduled time, and reporting the location, start time, end time, and elapsed time. Protective equipment may include (for example) hard hats, left or right gloves, left or right shoes / boots, hearing protection, safety goggles, life jackets, gas masks, welding masks, and other protective gear. (g) The door will open and close at the scheduled time and location and report the location, start time, end time, and elapsed time. (h) An object is present at a location at a scheduled time and reports the location, start time, end time, and elapsed time. Objects may include (for example) gangways, hoses, tools, ropes, cranes, boilers, pumps, connectors, solids, liquids, small boats, and / or other unknown items. (i) Its normal operating activities are carried out using at least one of the following: engines, cylinders, hoses, tools, ropes, cranes, boilers, and / or pumps. (j) Necessary maintenance activities are being performed on engines, cylinders, boilers, cranes, steering mechanisms, HVAC, electrical systems, piping / plumbing systems, and / or other systems.

[0032] It is noted that the above example items (a-j) are only a few of the wide range of possible interactions that may form the basis of detectors according to exemplary embodiments herein, and those skilled in the art will understand that other detectable events, including person-to-person, person-to-device, or device-to-device interactions, are expressly contemplated.

[0033] During operation, the predictive event vision detector receives as input the detection results of one or more vision systems on the vessel. The results may be detection, non-detection, or anomaly at the time of the predicted event according to the plan. Multiple events or detections may be combined into a single higher-level event. For example, combining multiple events or detections may result in a maintenance procedure, a cargo operation, or an inspection patrol. Note that each visual event is associated with a specific image or video sequence from a specific (or several) vision system camera(s) 118, 180, 182 at a specific time and at a known location within the vessel. The associated video may or may not be optionally transmitted with each event or alarm. If video is transmitted with an event or alarm, it may be useful for later verification of the event or alarm. In addition to compressing video by reducing it to a few images or a short sequence, the system can reduce the size of the image by cropping the image down to the critical or meaningful image locations required by the detector, or by reducing the resolution, for example, from the equivalent of high definition (HD) resolution to standard definition (SD) resolution or below standard resolution.

[0034] The onboard server establishes a transmission priority for processed visual events, which is typically based on settings provided by a user operating the onshore (base) dashboard. The onboard server buffers these events in a queue in storage, where they can be ordered based on priority. Priorities can be set based on various factors. For example, personnel safety and / or ship safety can have the highest priority, while maintenance can have the lowest priority, generally categorizing events according to the urgency of the matter involved. As an example, all of the highest priority events in the queue are transmitted first, followed by lower priority events. If a new higher priority event occurs onboard, the new higher priority event is transmitted before the lower priority events. It is contemplated that the lowest priority event may be interrupted if the higher priority events occupy all available bandwidth. The onboard server receives an acknowledgement from the onshore base server, confirming that the event has been received and acknowledged onshore, before marking the onboard event as transmitted. Multiple events may be transmitted before an acknowledgement is received (or not). Lack of acknowledgment potentially stalls the queue or requests a retransmission of the event before sending any subsequent events in the server's "priority" queue. The land-based server interface can configure or select visual event detectors over the communications link. In addition to visual events, the system can transmit non-visual events such as fire or smoke alarm signals.

[0035] III. Detection flow

[0036] As shown in FIG. 2, an exemplary operational flow 200 for a generalized detection flow used in implementing the system is illustrated. The operation can be characterized in three stages or segments: computation 210, data primitive generation 220, information creation 230, and presentation to a user via a land-based dashboard 240. Alternatively, some or all of the functionality herein can be performed by a user via a ship-based dashboard, which affects programming in at least one local or base server. The ship dashboard can also function as a passive terminal, sending commands back to a base interface via a communications link so that such commands can be executed via the base. The computation stage 210 includes performing sensor-based measurements 212 and visual detection 213, which generate a set of metrics 222 that are displayed to the user as separate events 232. The computation stage 210 performs pattern matching 224 on a time series of events 226 using event ordering (priority) 214, filtering (by pruning, compression, etc.), and event qualification 216 based on rules 217. This data is presented as composite events 234. These composite events 234 may include scenarios such as a successfully performed maintenance task or the occurrence of a safety violation. The computation stage 210 can aggregate visual and other events 218 and derive statistics 228 (e.g., number of safety violations over a period of time). These statistics 228 can be presented to a shore-based user as individual vessel reports 236 and fleet reports 238, providing the user with valuable information regarding behavior and performance in the context of various factors related to the event as a whole.

[0037] Figure 3 illustrates a detection flow procedure 300 for an example bridge routine for one or more vessels in a fleet. During the computation stage 310, sample detectors 312 provided by visual and other detectors include (for example) people crossing or stopping at a location, interacting with equipment, walking, sitting, motionless (stationary), gazing at a location, wearing earphones at a location, and / or lighting a fire. During the associated data primitive generation stage 320, sample detection metrics 322 are provided, including (for example) start and end times, duration, number of participants, bridge stations visited, protocol steps performed, and protocol non-compliance. Event samples 324 may include the name(s) of participants identified performing a shift, shift start time, whether a particular participant's shift was longer or shorter than normal, and understaffing and / or excessive / unauthorized personnel on the bridge. In an exemplary information stage, sample reports 332 are created that may include (for example) shift duration over a period of time, shift participation (headcount), equipment interaction time statistics, distribution (e.g., number of shifts x duration), and location graphs (e.g., heat maps) that may be based on month, week, day, etc. In the information stage 330, the sample reports 332 may be presented as vessel reports 334 and vessel fleet reports. Sample detection metrics 322 and sample of events 324 may be presented to the user as separate events 338 and composite events 339.

[0038] Figure 4 illustrates a detection flow procedure 400 for an example safety patrol of one or more vessels in a fleet. In the calculation phase 410, sample detectors 412 provided by visual and other detectors include (for example) event locations, people interacting with equipment, people stopping at a location, people walking or looking at a location, people wearing hard hats, life jackets, other protective gear, and / or people carrying safety tools such as fire extinguishers, flashlights, etc. In the data primitive phase 420, sample detection metrics can include (for example) event start or end times, duration, number of participants, stations visited, protocol steps performed, and / or patrol-specific protective equipment (PPE) utilized. Event samples 424 can include whether a safety patrol was not performed for a predetermined number of hours, patrols that took X% longer or shorter than normal, patrols performed by X personnel, patrols that started X minutes late, patrols performed without required PPE, and / or patrols completed in X minutes. The information stage 432 provides sample reports 432 based on events, including duration over time, participation, adherence to safety protocols, station time requirements, distribution (e.g., number of patrols x duration), and / or graphs / heat maps based on month, day, week, etc., vessel reports 434 and vessel fleet reports 436. The information stage 430 also reports distinct events 438 and composite events 439 based on sample detected events 422 and sample of events 424.

[0039] Figure 5 illustrates a detection flow procedure 500 for an example loading operation involving one or more vessels in a fleet. In the calculation stage 510, sample detectors 512 may include pipes connected, pipes disconnected, people interacting with equipment, people standing, people arriving or departing, and people wearing hard hats, gloves, goggles, and / or other PPE. The data primitive stage 520 provides sample detection metrics 522, including start and end times, duration, number of participating crew members, protocol steps performed, and / or PPE utilized in the operation(s). Sample events 524 may include operations completed in X minutes, operations completed X% longer or shorter than normal, operations performed by X crew members, and / or operations performed without X type of PPE (no PPE). In the information stage 530, sample reports 532 may include duration over time, participation, protocol adherence, location / log, distribution (e.g., number of drills x duration), and / or non-compliance vs. normal / standard practice. These may be presented as vessel reports 534 or vessel fleet reports 536. Sample detection metrics 522 and sample of events 524 are reported as separate events 538 and composite events 539.

[0040] Other exemplary detection flows may be provided as needed to generate desired information regarding vessel personnel and system activities of interest. Such detection flows utilize associated detector types, parameters, etc. Likewise, the mechanisms for performing detection may vary. In alternative configurations expressly contemplated herein, the event detector may be implemented, in part or in full, using appropriate deep learning software algorithms / persistent computer-readable program instructions implemented on shore-based and / or vessel-based processor(s). By way of non-limiting example, an implementation of a deep learning / artificial intelligence "hybrid" detector configuration is shown and described in commonly assigned U.S. Patent Application No. 17 / 873,053, filed July 25, 2022, and entitled "SYSTEM AND METHOD FOR AUTOMATIC DETECTION OF VISUAL EVENTS IN TRANSPORTATION ENVIRONMENTS," the teachings of which are expressly incorporated by reference as useful background information.

[0041] IV. Risk Assessment

[0042] A. Operation Process

[0043] In an exemplary embodiment, the systems and methods herein enable assessment of risk to a commercial vessel through response(s) to automatically detected maritime visual events on the vessel. Events are monitored and generated using the above-described configurations and equivalent implementations thereof. The generated and stored event data is used in real-time and near-real-time (e.g., with normal system transmission / processing latencies) to subsequently generate a risk profile for a vessel and fleet of vessels, along with information related to the risk profile (e.g., insurance rate information, recommended risk mitigation procedures, etc.).

[0044] Referring back to system configuration 100 of FIG. 1 , processing configuration 150 includes risk assessment process (processor) or module 157 and risk reporting process (processor) or module 158. The actual functionality of these modules can be configured in various manners and may be instantiated on shore-based server platform(s) 140, on ship-based server 130, or both. The processes / processors perform various functions based on the received event data. Referring to procedure 600 of FIG. 6 , the system provides one or more automatically generated events, along with associated information about the automatically detected marine visual events, to the risk assessment process (processor) in step 610. The events may be characterized by the following categories: crew behavior / navigation / management, crew safety, ship machinery, maintenance and housekeeping, ship environment and pollution prevention, and active cargo monitoring. These events may be further categorized to include hull and machinery, cargo, and crew. According to step 620, events may be evaluated as single instances or may be combined in a time-based (or other baseline) manner.

[0045] In step 630, one or a group of aggregated events is compared to examples of safe or unsafe conditions associated with a particular event or category of events using appropriate comparison metrics. The comparison can use, for example, traditional deep learning (and / or other artificial intelligence (AI)) techniques, where visual information of the event is matched with various images of high-, low-, or medium-risk scenarios retrieved from a local or cloud-based data store. These comparisons are then used to provide a risk assessment score (step 640) based on a scale that can be established for each type of event. The scale can include various factors and can be linear or nonlinear. For example, in the case of a partial PPE event by a crew member, failure to wear gloves may establish a low level of risk assessment score, while failure to wear a hard hat may establish a much higher risk assessment score (also referred to herein as a "risk score"). Note that training of the deep learning / AI system to recognize high-, medium-, and low-risk scenarios can be ongoing. As a user notices new and / or unique visual events across the fleet, they can use the interface configuration herein to add those events to the overall deep learning library of image data. In this way, the risk assessment profile can be continually refined and improved. Also, note that in FIG. 1, such library of risk-related image data and corresponding metrics regarding the level / magnitude of risk is shown as risk data 159. This data store 159 interacts with the processing configuration(s).

[0046] Determining the risk assessment score can be based on a variety of techniques, which can be applied in various ways depending on the type of event or other factors. For example, the score can be computationally based on comparing the acquired visual event to a minimum standard associated with the type of vessel or fleet of vessels. Alternatively, the risk assessment score can be based on comparing the acquired visual event to a relative standard (a number of compliant or non-compliant events) associated with the type of vessel or fleet of vessels. In particular, the relative standard can be based on a predetermined number of standard deviations from an average value (e.g., a value that deviates by two or more standard deviations is non-compliant). As a non-limiting example, if a cargo is detected at a certain point in time to be unstrapped, a single strap or a single instance may generate a first score. That score may be below the standard deviation for non-compliance. However, if multiple instances of missing steps or multiple missing straps in a single instance are detected, it may exceed one standard deviation for non-compliance. Alternatively, an absolute minimum standard may determine that missing two straps is always a non-compliant act, but occasionally allowing missing one strap.

[0047] The scores may then be aggregated / combined into a risk assessment value in step 650. More generally, the risk assessment can be a single score corresponding to a single detection event or multiple detection events at a single point in time, or it can be a composite score or array of scores derived by combining multiple detection events or by examining the overall statistics of a single detection event or multiple detected marine visual events over a period of time.

[0048] In addition to representing static conditions, the present system and method allows for automatic risk assessment while the vessel is in operation. For example, in addition to the question, "Do the hoses / manifolds / pipelines appear to be in good condition?", the present system and method provides further questions based on the observed conditions, such as, "If the hoses / manifolds / pipelines were observed in operation, would any leaks be visible?" Similarly, non-visual and detected conditions, such as operation of pumps, generators, engines, purifiers, etc., can be evaluated in addition to the visible static conditions. This evaluation of operational conditions can improve the overall assessment of risk compared to static assessments. These questions include broad categories of hull and machinery.

[0049] Below is an example of a dynamic and automated assessment based on automated visual events compared to a static human inspector, where the static inspection generally generates a list of deficiencies that allows the vessel to be classified as "standard", "below standard" or "above standard". These categories can also be divided into medium risk, high risk and low risk.

[0050] (a) Instead of static survey questions such as "Are crew members trained to use PPE?" and / or "Is PPE available and in good condition?", the present system and method automatically determines "Are crew members using PPE when in the machinery room, steering room, and on deck?" This is done by identifying and analyzing images of crew members and querying visual events that correlate with the various ship locations (i.e., camera locations) where the event images were acquired.

[0051] (b) In addition to survey questions such as "Have the water leak alarms been tested?", the present system and method uses images and sensors to determine "Is there a visible water leak during the voyage?"

[0052] (c) Instead of survey questions such as "Are systems in place for routine cleaning / maintenance and are adequate records of cleaning / maintenance kept?", the present system and method determines during the voyage "Are routine cleaning / maintenance being performed?"

[0053] (d) Instead of static survey questions based on the ship's logbook or documentation, such as "Have safety training been conducted and documented accordingly?", the present system and method determines that adequate safety training has been conducted through visual observation of the safety training.

[0054] (e) Instead of survey questions such as "Are safety procedures in place for working in cold rooms / confined spaces?", the system and method invention identifies that exposure limits are being met and that appropriate protective equipment is being used. In general, safety issues fall under the broad category of personnel.

[0055] B. Reporting and Display

[0056] The risk assessment module 157 derives data regarding risk for individual vessels and fleets of vessels. This data is provided to the risk reporting module 158, enabling shore-based and / or ship-based display of relevant information on suitable graphical user interface (GUI) screens instantiated on (for example) a conventional web browser-based computing platform (e.g., displays 134, 172) or another custom computing device. The platform provides various interface screens for reporting and manipulating event data, as outlined in the above-incorporated U.S. patent application Ser. No. 17 / 175,364. Figures 7 and 8 show exemplary GUI displays 700 and 800, respectively, directly related to reporting and manipulating vessel risk profiles and fleet risk assessments. Both displays 700 and 800 may be selected via appropriate tabs on the main interface screen or other menu-based configurations.

[0057] As shown in FIG. 7, display 700 is selected from a vessel risk profile tab 710, which appears alongside a vessel risk assessment selection tab 712 used to access display 800, described below. Vessel risk profile screen 700 includes a drop-down menu 714 for selecting a vessel within a fleet. A second drop-down menu 716 allows the user to select the time period for which the risk profile of the selected vessel is sought (in this example, year-to-date). Other time periods / intervals that should be apparent to those skilled in the art (such as this or last quarter, this or last month, last year, or a custom date range) can also be selected. Upon selection of a vessel and time period, its associated identification data is displayed in window 720, and a processor calculates an overall risk score 730 for that vessel within the applicable time period. The displayed overall vessel risk score can be based (for example) on a weighted combination of individual risk scores for that period based on the events and other detected conditions described above. In this example, the overall vessel risk score is based on a 100 scale, although other numerical and / or graphic scales can be utilized. A graph 732 of the overall vessel risk score over time, over a selected time period / interval, is also provided. In particular, the system accesses available data on the industry and relative peer groups (e.g., tankers operating on the same route and / or tankers within the same fleet) to provide a benchmark of the vessel's risk profile. The fleet benchmark 740 and industry peer group benchmark 742 are shown as sliding scales with respective indices 744 and 746 of the vessel in question.

[0058] A pane 750 containing multiple side-by-side tabs allows the user to further analyze the various categories / types of risk being analyzed. Exemplary categories shown in pane 750 include crew behavior 751, navigation 752, safety 753, equipment 754, maintenance 755, environment 756, and cargo 757. The number and type of categories may be modified based on the type of vessel, its mission, and / or industry standards for risk assessment. In this example, the safety tab 753 is open, showing the current safety risk score 760 and associated risk score graph 762 over a selected time period. A sliding scale of fleet benchmark 764 and industry peer group benchmark 766 in the safety category is also shown. More specific information used to construct a category's risk score is displayed in a selectable lower series of panes 770, 780, and 790. The number of panes in this area corresponds to the type of event being monitored for the risk category. In this safety example, the types include PPE Usage Policy Violations (pane 770, shown open in the illustrated example), Safety Round Performance (pane 780), and Dangerous Behavior by Crew (pane 790). Each pane may contain information specific to the type of event being monitored. Generally, the information is similar to that displayed in the safety pane 770. The information includes a risk score 771 and a graph of the score over a time period 772.It also includes sliding scales 773 and 774 with fleet and industry peer group benchmarks, showing the vessel's relative position along the benchmark scales 775 and 776, respectively. A scrolling list 777 of all violation (and compliance) events is shown. This list includes (a) the type of event (e.g., hardhat usage, goggle usage, etc.), (b) status (e.g., compliance, violation, etc.), (c) location on the vessel (typically corresponding to one or more cameras / sensors), and (d) timestamp. By clicking (for example) with a cursor or a screen touch on an entry in the list, the user can view a video clip of the event in a display window 778, with appropriate playback controls, including applicable audio.

[0059] As shown in FIG. 8, the Fleet Risk Assessment screen 800 is accessed via the Assessment Selection tab 712 described above. This screen allows the user to review data relating to all vessels in a fleet, aggregated together, to perform a fleet-wide risk assessment. The Fleet Risk Assessment screen 800 presents a drop-down menu 814 for selecting the fleet and time period 816 for which a risk assessment is desired. The overall risk score (in this example, on a 100 scale) 820 for the current date is shown, as well as a graph 822 of the risk score over the selected time period. The system accesses available data on the industry and the industry's relative peer group (e.g., tankers operating on the same trade route) to provide a benchmark of the fleet's risk. The industry benchmark 830 and peer group benchmark 832 are shown as sliding scales with indices 834 and 836, respectively, for the fleet in question. An ID list 840 and a vessel list 841 are provided to show the current vessels in the fleet being assessed. Bar graphs 850 and 851 show the fleet risk profile by vessel (two bar graphs are shown for two exemplary vessels) and the distribution of risk scores, respectively. Also displayed is a graph 854 showing the risk scores over the time period. In embodiments, by clicking or touching various vessel-specific information, information may be presented on screen 700 regarding that vessel's profile and / or other vessel-specific information.

[0060] A pane 860 containing multiple side-by-side tabs allows the user to analyze the various categories / types of risk analyzed for the fleet in more detail. Exemplary categories shown in pane 860 include crew behavior 861, navigation 862, safety 863, equipment 864, maintenance 865, environment 866, and cargo 867. The number and type of categories may be modified based on the type(s) of vessels in the fleet, their mission, and / or industry standards for risk assessment. In this example, the safety tab 863 is reopened, showing the current fleet safety risk score 870 and associated risk score graph 872 over the selected time period. A sliding scale of the industry safety benchmark 874 and industry peer group benchmark 876 in the safety category is also shown. The position of the fleet within each benchmark scale 874 and 876 is indicated by indicia 878 and 879, respectively.

[0061] V. Conclusion

[0062] As is apparent, the above-described system and method provide an effective and useful tool for assigning and processing risk for various automatically detected visual events. The system and method effectively replace, supplant, and / or improve upon existing static methods of single-point, manual (and even paper-based) condition assessments, often performed by surveyors or inspectors. Because risk assessments are performed automatically without the need for paid surveyors or inspectors, assessment costs may be lower than traditional manual assessments or may allow for shorter manual assessments. The system and method offers a further advantage over paid surveyors or inspectors in that paid surveyors or inspectors produce assessments that vary widely in skill level. Conversely, the system and method produces direct observations of condition that are significantly more repeatable than manual assessments.

[0063] The foregoing is a detailed description of exemplary embodiments of the present invention. Various modifications and additions may be made without departing from the spirit and scope of the present invention. Features of each of the various embodiments described above may be combined with features of other described embodiments, as appropriate, to provide numerous combinations of features in related new embodiments. Moreover, while the foregoing describes numerous separate embodiments of the apparatus and method of the present invention, what has been described herein is merely illustrative of the application of the principles of the present invention. For example, as used herein, the terms "process" and / or "processor" should be interpreted broadly to include various electronic hardware and / or software-based functions and components (and may alternatively be referred to as functional "modules" or "elements"). Furthermore, illustrated processes or processors may be combined with other processes and / or processors or divided into various sub-processes or sub-processors. Such sub-processes and / or sub-processors may be combined in various ways in accordance with embodiments of the present invention. Similarly, it is expressly contemplated that any functions, processes, and / or processors herein may be implemented using electronic hardware, software consisting of a non-transitory computer-readable medium of program instructions, or a combination of hardware and software. Furthermore, as used herein, various directional and orientation terms, such as "vertical," "horizontal," "up," "down," "bottom," "top," "side," "front," "rear," "left," "right," etc., are used only as relative conventions and not as absolute orientations / orientations relative to a fixed coordinate space, such as the direction of gravity. Furthermore, when the terms "substantially" or "about" are used with respect to a given measurement, value, or characteristic, it means an amount that is within a normal operating range to achieve a desired result, but includes some variation due to inherent imprecision and error within the system's tolerance (e.g., 1-5%). Therefore, this description is intended to be merely illustrative and is not intended to limit the scope of the present invention.

Claims

1. 1. A method for assessing marine vessel risk in response to automatically detected marine-based visual events, comprising: detecting at least one marine visual event from a plurality of marine-based visual events captured by at least one camera on the vessel, providing image data of the visual event to a processor, the visual event being associated with at least one of safety, security, maintenance, crew actions, and cargo; generating a risk assessment score in response to the detected at least one visual event; providing the risk assessment score to a user in a desired format.

2. 2. The method of claim 1, wherein generating the risk assessment score comprises comparing the at least one visual event with data of conforming or non-conforming model visual events from a data storage and establishing a score based on a level of match between the at least one visual event and the conforming or non-conforming model visual events.

3. The method of claim 2 , wherein the risk relates to at least one of (a) engine maintenance alerts, (b) cargo condition or operation, and (c) crew safety, security, and crew behavior.

4. 3. The method of claim 2, wherein generating the risk assessment score comprises comparing the at least one visual event to minimum standards associated with at least one of: (a) a type of vessel or fleet of vessels, (b) cargo handling standards, and (c) safety standards.

5. 3. The method of claim 2, wherein generating the risk assessment score comprises comparing the at least one visual event to relative criteria associated with at least one of: (a) a type of ship or fleet of ships, (b) cargo handling criteria, and (c) safety criteria.

6. The method of claim 5 , wherein the relative criteria is based on a predetermined number of standard deviations from a mean value.

7. The method of claim 1 , further comprising providing additional information to a user in association with the risk assessment score that matches that provided in a vessel risk survey.

8. 10. The method of claim 1, further comprising: detecting a plurality of marine-based visual events acquired by a camera on each of a plurality of vessels in a fleet of vessels, each providing image data for a plurality of visual events, the plurality of visual events being associated with at least one of safety, security, maintenance, crew actions, and cargo; generating a risk assessment score in response to the detected visual events; and correlating the risk assessment scores to provide an overall risk assessment for the fleet of vessels.

9. The method of claim 8 , wherein the risk assessment is organized into at least one of safety, security, maintenance, crew behavior, and cargo and displayed on a user interface.

10. 10. The method of claim 9, wherein risk assessment profiles for individual vessels in the fleet are displayed on the user interface based on user selection.

11. 1. A system for assessing marine vessel risk in response to automatically detected marine-based visual events, comprising: a camera configured to acquire at least one detected marine visual event on the vessel related to at least one of safety, security, maintenance, crew actions, and cargo, and to provide image data of the visual event; a processor that receives image data of the visual event and generates a risk assessment score in response to the at least one detected marine visual event; a user interface that displays information related to the risk assessment score to a user in a desired format.

12. 12. The system of claim 11, further comprising a comparison process that compares the at least one visual event with data of matching or non-matching model visual events from a data storage and establishes a score based on a level of match between the at least one visual event and the matching or non-matching model visual event.

13. 13. The system of claim 12, wherein the risks relate to at least one of (a) engine maintenance alerts, (b) cargo conditions or operations, and (c) crew safety, security, and crew behavior.

14. 13. The system of claim 12, wherein the risk assessment score is based on the comparison process comparing the at least one visual event to minimum standards associated with at least one of: (a) vessel or fleet type, (b) cargo handling standards, and (c) safety standards.

15. 13. The system of claim 12, wherein the risk assessment score is based on the comparison process comparing the at least one visual event to relative criteria associated with at least one of: (a) type of ship or fleet of ships, (b) cargo handling criteria, and (c) safety criteria.

16. The system of claim 15 , wherein the relative criteria is based on a predetermined number of standard deviations from a mean value.

17. 12. The system of claim 11, further comprising additional information provided to a user in association with the risk assessment score that matches that provided in a vessel risk survey.

18. 12. The system of claim 11, further comprising: a plurality of cameras on each vessel of a plurality of vessels in the fleet, each providing image data for a plurality of visual events, the plurality of visual events being associated with at least one of safety, security, maintenance, crew actions, and cargo; generating a risk assessment score in response to the detected visual events; and correlating the risk assessment scores to provide an overall risk assessment for the fleet.

19. 20. The system of claim 18, further comprising a user interface including the risk assessment, wherein the risk assessment is displayed according to categories including at least one of safety, security, maintenance, crew actions, and cargo.

20. 20. The system of claim 19, wherein the user interface displays a profile of the risk assessment for each individual vessel in the fleet of vessels and includes a selector on the user interface for selecting the individual vessel.

Images