Systems and methods for operating autonomous marine vessels

Abstract

A system for operating a plurality of autonomous vessels (101). A communication interface (108) enables communication between a remote control center (110) and each of the autonomous vessels (101). A fleet coordination layer (201) implemented in the remote control center (110) collects information regarding the operation of the autonomous vessels (101), performs risk assessments, and issues operational mode transition commands to the autonomous vessels (101) as needed. A vessel coordination layer (202) implemented on the autonomous vessels (101) controls the transitions between operational modes of the autonomous vessels (101). A vessel execution layer (203) implemented on the autonomous vessels (101) includes sensors (301), a perception, planning, and execution module (302), and actuators (303) that are used to control the movement of the autonomous vessels (101) according to a mission description, received sensor data, and operational mode commands from an operational mode management module (402). Corresponding methods are also disclosed.

Description

[Technical Field]

[0001] The present invention relates to the operation of autonomous vessels. In particular, the present invention relates to a system and associated method for operating an autonomous vessel that is in communication with a remote control center that can assist the autonomous vessel, control the autonomous vessel, or hand over control to a human operator. [Background technology]

[0002] Autonomy has been gaining significant attention in the transportation sector for many years now. The focus has primarily been on autonomous vehicles and has followed two main paths. One is the driver assistance systems being deployed in cars and trucks. These systems allow the driver to maintain control of the vehicle at all times but also introduce various possibilities for providing assistance to the driver in certain situations. Examples include anti-lock braking systems, adaptive cruise control, lane-keeping assist, parking assist, emergency braking systems, and the like. These systems receive input from sensors such as radar, lidar, cameras, and wheel sensors, and include image recognition and other processing capabilities to recognize traffic signs, other vehicles, pedestrians, lane markings, and the like. The actions performed by the systems range from simply warning the driver that the vehicle is about to deviate from its lane, through assistance where the system actively controls some aspects of steering or braking, to fully autonomous procedures such as automated parking. The various systems implemented in vehicles are primarily designed to interact with or assist the human driver, and do not interact with each other.

[0003] The other track focuses on fully autonomous vehicles, which have no driver and are not remotely controlled. The most famous example is a shuttle bus, which operates at low speeds and travels only along a predetermined route. In such fully autonomous vehicles, the various subsystems are designed to cooperate with each other, but one vehicle operates independently of the other vehicles, and there is no remote assistance or control from a common control system.

[0004] One area of ​​transportation that has not been explored much in terms of autonomy is maritime transport. While autopilots and various assistance systems for large ships have long been available, urban waterways are dominated by relatively large, manually operated ferries. Urban areas are growing worldwide. By 2050, 70% of the world's population is predicted to live in cities, a 58% increase from 4.3 billion today to 6.8 billion (World Bank, 2020). Humans tend to congregate near waterfronts, and today, 90% of urban areas are coastal (UN Habitat, 2022). There is also a global trend for cities to increasingly transform old industrial areas into attractive residential and commercial areas through their waterfronts. As a result of this development, current urban transport infrastructure is reaching its breaking point; there simply is not enough capacity on the roads to meet the growing demands for personal mobility and transport logistics (the latter consisting of both the cargo, food, materials, and other goods that enter cities and the waste that flows out again). This challenge is further compounded by the introduction of zero-emission zones and public demand for more sustainable transport solutions, a much-needed advancement given that cities already account for over 70% of global CO2 emissions (UN Environment Programme, 2022).

[0005] Today, urban waterways are often perceived as obstacles to transportation flow. Bridges and tunnels traverse them, but bridges and tunnels represent bottlenecks for road-based travel. However, these waterways hold great unrealized potential as part of the solution toward sustainable and livable cities. To achieve this, urban maritime transportation must become more flexible, better meet changing needs, and introduce a greater variety of vessel sizes. At the same time, to make such transportation scalable and economical, the need for human operators must be reduced or eliminated. This means that urban maritime transportation must undergo the same transition to autonomy currently being developed for land transportation. As a result, there is a need for the development of efficient systems for transportation on urban waterways that include safe, secure, and scalable autonomy for vessels operating on such waterways. Summary of the Invention

[0006] To address these needs, a system is provided for operating a plurality of autonomous vessels configured to serve a plurality of locations under the supervision of a remote control center. The system includes a communication interface configured to enable communication between the remote control center and each of the autonomous vessels. In accordance with the present invention, the system is implemented in several layers, as follows: a fleet coordination layer is implemented in a computer system associated with the remote control center; i.e., the computer system does not necessarily have to be located at the remote control center; instead, at least some of its components may be located remotely and accessible via a computer network. The fleet coordination layer includes at least one module configured to collect information regarding the operation of the autonomous vessels, perform a risk assessment based on the collected information, monitor the operating modes of the autonomous vessels, and issue operating mode transition instructions to the autonomous vessels based on the risk assessment.

[0007] The vessel coordination layer is implemented on the plurality of autonomous vessels, and includes an operational mode management module on each autonomous vessel configured to control transitions between operational modes of the autonomous vessel according to instructions received from at least one module in the fleet coordination layer. Furthermore, the vessel execution layer is implemented on the plurality of autonomous vessels, and includes sensors, a perception, planning, and execution module, and actuators, where the perception, planning, and execution module is configured to control motion of the autonomous vessel using the actuators according to the mission description, the received sensor data, and the operational mode commands from the operational mode management module. At least one module in the vessel coordination layer is configured to transmit information regarding the operational mode to a remote control center, and at least one module in the vessel execution layer is configured to transmit information regarding the sensor data to the remote control center. The transmitted information can then be included in information gathered by the at least one module in the fleet coordination layer.

[0008] In some embodiments, at least one module in the fleet coordination layer may be implemented using a monolithic architecture, with the at least one module being a module having a neural network trained using machine learning. The neural network may take the collected data as input and generate outputs related to risk assessments and operational mode transitions. The neural network may further be trained to receive additional information and generate additional outputs related to, for example, logistics and resource utilization. In other embodiments, the at least one module in the fleet coordination layer includes a networked online risk management module configured to collect information related to the operation of the autonomous vessels and perform risk assessments based on the collected information, and an operational mode coordination module configured to monitor the operational modes of the autonomous vessels, receive risk assessments from the networked online risk management module, and issue operational mode transition instructions to the autonomous vessels based on the received risk assessments.

[0009] Some embodiments of the present invention include additional modules in the fleet coordination layer. In particular, a fleet coordination and optimization module may be configured to receive risk assessment information from the risk management module, information regarding each vessel's operational mode from the vessel coordination layer, and information regarding sensor data from the vessel execution layer. This module may be configured to use the received information to plan optimization of the autonomous vessels' utilization and issue mission descriptions to the autonomous vessels based on the planned optimization. Correspondingly, a mission planning and re-planning module in the vessel coordination layer may be configured to receive mission descriptions from the fleet coordination layer and situational awareness information from the vessel execution layer, plan or re-plan mission execution based on the received information, and issue instructions to the perception, planning, and execution module.

[0010] In some embodiments, the vessel coordination layer may include a self-diagnostic module configured to receive operational information from the vessel execution layer and perform diagnostics based on the received information, and an online risk management module configured to receive operational information from the vessel execution layer, diagnostic information from the self-diagnostic module, and risk assessment information from the fleet coordination layer and perform risk assessments. The operational mode management module may then be further configured to take as input the diagnostic information from the self-diagnostic module and the risk assessment information from the online risk management module when controlling transitions between operational modes. The operational information from the vessel execution layer may include information derived from data selected from the group consisting of sensor data, object detection data, situational awareness information, motion planning data, and motion control data.

[0011] In a further embodiment of the present invention, the remote control center may further include a mode switch configured to receive user input directing a transition to a remote control mode of operation, and a remote control interface configured to receive user input provided directly to a vessel execution layer of an autonomous vessel to enable remote control of the autonomous vessel by a human operator.

[0012] Various modes of operation are contemplated for an autonomous vessel operating in a system according to the present invention, and the modes of operation in which the autonomous vessel may operate are selected from the group consisting of: local standby, normal autonomous operation, berthing, departing, en route, preparing to berth, berthing, manually controlled, remotely controlled, and a minimum risk state.

[0013] In embodiments of the present invention, the at least one risk management module may use a method selected from the group consisting of influence diagrams, Bayesian belief networks, fuzzy logic, and reinforcement learning.

[0014] According to another aspect of the present invention, a method for operating a plurality of autonomous vessels under supervision by a remote control center is provided. The vessels are configured to serve a plurality of locations, and the system includes a communication interface configured to enable communication between the remote control center and each of the autonomous vessels, a fleet coordination layer implemented in a computer system associated with the remote control center, a ship coordination layer implemented on the plurality of autonomous vessels, and a ship execution layer implemented on the plurality of autonomous vessels. The method includes, at the fleet coordination layer implemented in the remote control center, generating and distributing mission descriptions to each of the autonomous vessels, receiving and collecting information regarding operation of the autonomous vessels, performing a risk assessment based on the collected information, monitoring operational modes of the autonomous vessels, and issuing operational mode transition commands to the autonomous vessels based on the risk assessment. The method further includes, at each of the autonomous vessels, controlling, at the ship coordination layer, transitions between operational modes of the autonomous vessels according to instructions received from the remote control center and transmitting information regarding the operational modes to the remote control center. The method further includes, at each of the autonomous vessels, controlling, at the ship execution layer, operation of the autonomous vessels according to the received mission description, the received sensor data, and the operational mode instructions, and transmitting information regarding the sensor data to the remote control center.

[0015] Some embodiments may further include, at the remote control center in the fleet coordination layer, receiving information regarding the operational mode of each ship from the ship coordination layer and information regarding the sensor data from the ship execution layer, planning optimization of the utilization of the autonomous ships, and issuing mission descriptions to the autonomous ships based on the planned optimization. Each autonomous ship in the ship coordination layer can then receive the mission description from the fleet coordination layer and the situational awareness information from the ship execution layer, plan or re-plan mission execution based on the received information, and issue instructions to the ship execution layer.

[0016] In some embodiments, the method may include, at the vessel coordination layer, receiving operational information from the vessel execution layer, generating diagnostic information based on the received information, receiving risk assessment information from the fleet coordination layer, and performing a risk assessment based on the received operational information, the received risk assessment information, and the generated diagnostic information to generate risk assessment information for the vessel. The generated diagnostic information and generated risk assessment information for the vessel may then be used as inputs when controlling transitions between operational modes.

[0017] Operational information from the vessel execution layer may include, for example, information derived from data selected from the group consisting of sensor data, object detection data, situational awareness information, motion planning data, and motion control data.

[0018] The operating modes in which the autonomous vessel may operate may be selected from the group consisting of: local standby, normal autonomous operation, at anchor, departing, en route, preparing to berth, at anchor, manually controlled, remotely controlled, and a minimum risk state. [Brief explanation of the drawings]

[0019] The above and further advantages of the system and method provided in accordance with the present invention will now be described in more detail, by way of example, with reference to the accompanying drawings, in which: [Figure 1] 1 is an overview of a system consistent with the principles of the present invention. [Figure 2]1 is a block diagram illustrating information flow between layers of a system consistent with the principles of the present invention. [Figure 3A] FIG. 1 shows a block diagram of modules that may be implemented in an embodiment of a vessel execution layer in a system according to the present invention. [Figure 3B] FIG. 1 shows a block diagram of modules that may be implemented in an embodiment of a vessel execution layer in a system according to the present invention. [Figure 3C] FIG. 1 shows a block diagram of modules that may be implemented in an embodiment of a vessel execution layer in a system according to the present invention. [Figure 4] 1 shows a block diagram of modules that may be implemented in an embodiment of a vessel coordination layer in a system according to the present invention; [Figure 5A] FIG. 1 shows a block diagram of modules that may be implemented in an embodiment of a fleet execution layer in a system according to the present invention. [Figure 5B] FIG. 1 shows a block diagram of modules that may be implemented in an embodiment of a fleet execution layer in a system according to the present invention. [Figure 6] FIG. 1 is a state diagram illustrating various states that an autonomous vessel may be in according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0020] The present invention represents a response to the need for more flexible urban transportation systems by focusing on more efficient use of urban waterways, particularly autonomous vessels for the transportation of passengers and goods. While this area of ​​endeavor shares characteristics with similar developments related to autonomous vehicles, such as automobiles, and driver assistance systems for automobiles, the marine environment also has several features unique to the marine environment, which pose unique challenges but also offer unique opportunities. In this disclosure, the term marine vessel is intended to include, but is not limited to, ferries, water taxis, hovercraft, or any other boat or vessel intended for the transportation of people or goods on urban waterways.

[0021] In the following description of various embodiments, reference will be made to the drawings, in which like reference numerals refer to identical or corresponding elements. The drawings are not necessarily to scale. Instead, certain features may be shown enlarged or in a somewhat simplified or schematic manner, and certain conventional elements may be omitted so as not to clutter the drawings with details that do not contribute to an understanding of these principles, in the interest of illustrating the principles of the invention.

[0022] It should be noted that, unless otherwise specified, different features or elements can be combined with each other, regardless of whether they are described together as part of the same embodiment below. Combinations of features or elements in the drawings are intended to facilitate understanding of the invention rather than limit the scope of the invention to a particular embodiment, and are intended to be interchangeable to the extent that alternative elements having substantially the same function are shown in each embodiment. However, for the sake of brevity, no attempt is made to disclose a complete description of all possible permutations of features. Thus, different drawings do not represent separate embodiments in the sense that they are mutually exclusive alternatives. Instead, the drawings may focus on, for example, different aspects or different levels of detail. Alternative embodiments to those shown in the drawings can be arrived at by adding features, removing features, or arranging features in a different arrangement than shown in the exemplary drawings. Unless features are expressly identified as essential or functionally dependent on each other to function, such features can be omitted, rearranged, or interacted with additional features not described herein in any manner within the ability and knowledge of one of ordinary skill in the art upon studying this disclosure. Similarly, if features are described in different levels of detail with reference to different drawings, this does not imply that the embodiment is configured with either less or more detail. Instead, details described with reference to one drawing should be understood as being available, but not necessarily required, in an embodiment; thus, some or all features of a detailed example may be carried over to a less detailed description, or none of the features of a detailed example may be carried over to a less detailed description, unless otherwise specified or clearly dependent on one another for intended operation. In other words, different aspects, including design choices, are largely independent of one another, and where certain choices made for one aspect affect choices that can be made for other aspects, this will be noted below.Where not indicated, the choices are independent, e.g., the choice of which sensors to use does not depend on whether one chooses a modular approach or an end-to-end approach to designing an autonomous system.

[0023] Therefore, those skilled in the art will understand that the present invention can be practiced without many of the details contained in this detailed description. Conversely, some well-known structures or functions may not be shown or described in detail to avoid unnecessarily obscuring the important description of the various embodiments. The terms used in the description presented below are intended to be interpreted in their broadest reasonable manner, even when used in conjunction with detailed descriptions of certain specific embodiments of the present invention.

[0024] The terms used in the description presented below are intended to be interpreted in their broadest reasonable manner, even when used in conjunction with detailed descriptions of certain specific embodiments of the present invention. To the extent that terms such as first, second, top, bottom, left, right, near, far, etc. are used, they are primarily intended to distinguish features from one another and are not intended to define absolute relationships unless otherwise clear from the context. It should also be noted that when some part, component, or module is explicitly stated to transmit (or receive) instructions, information, or data to or from another part, component, or module, this should be interpreted as an explicit statement meaning that the other part, component, or module performs the corresponding receiving (or transmitting) operation.

[0025] Referring initially to FIG. 1 , an overview of a system is shown that includes three autonomous vessels 101. These vessels 101 operate in the same urban waters. In some embodiments, the vessels are configured to operate between the same destinations, and all vessels follow the same overall route between destinations, except for variations caused by weather or moving obstacles (e.g., other boats), and differences in direction so that vessels do not interfere with each other when traveling in opposite directions between two destinations. In other embodiments, the vessels 101 can operate between different destinations, or even update their destinations on demand, and therefore must periodically modify and update their route plans.

[0026] Vessels 101 are configured to operate autonomously and, therefore, include corresponding autonomous systems 102. Each autonomous system 102 is configured to perform basic operations related to autonomy and automation. Such autonomous systems 102 include or are connected to sensors that provide information about the environment and analytical capabilities that can analyze sensor inputs in order to establish situational awareness, i.e., a representation of the operational domain, including dynamic elements such as other maritime traffic and obstacles, and to execute a motion plan according to a planned route or course. Automation includes the ability to execute motion control operations by steering the vessel according to commands generated from the motion plan.

[0027] Additionally, each vessel 101 includes a supervisory control system 103. The supervisory control system 103 monitors system integrity and performs vessel self-diagnostics. Self-diagnostics are not necessarily limited to the autonomous system 102 itself, but may also include monitoring of any other aspects of the vessel 101 and the environment, and may also include manual input, such as alarms triggered by passengers. The supervisory control system 103 performs risk assessment, mission planning and re-planning, and operational mode management, as described in further detail below. The onboard supervisory control system 103 communicates with a corresponding remote supervisory control system 104. The remote supervisory control system 104, which may be located, for example, on land or in a control center on a boat or some other floating facility, can cooperate with the onboard supervisory control system 103 and, in various situations (or various embodiments), assist the onboard supervisory control system 103 in performing its tasks by performing requested actions or providing external data. Similarly, the remote monitoring and control system 104 may, in some situations or embodiments, operate as a master monitoring and control system configured to obtain data from each vessel 101 and other sources and issue instructions to the onboard monitoring and control systems 103. In some embodiments, the remote monitoring and control system 104 may be configured to integrate information from several vessels 101 to provide each vessel 101 with the ability to establish situational awareness based on information obtained by the several vessels 101.

[0028] The remote monitoring system 104 may be part of a remote control center 110. The remote control center 110 may include a remote support computer 105. This computer 105 may implement additional functions and may even replicate some of the functionality of the onboard autonomous system 102 to provide remote redundancy, allowing the remote support computer 105 to provide a degree of remote control of the vessel 101 in the event of a failure of an onboard system. The remote control center 110 may provide additional functions, which are described in more detail below. Some possibilities include fleet optimization, which can assign vessels to the most demanding routes; networked situational awareness, which can utilize network knowledge to enable vessels to make better decisions than they could make if they relied solely on information from their own sensors; reception and operational mode coordination; and remote control functions, which include receiving and displaying situational awareness data from connected vessels and providing various levels of remote control. Additionally, the remote control center 110 may include a human supervisor 106. To this end, the remote control center 110 may include a dashboard or other type of information display and user input device that allows the human supervisor 106 to access system status information and issue instructions to one or more vessels individually or to the entire system.

[0029] The supervisory control system 103 and the remote monitoring system 104 can be implemented using a variety of methods. Examples include, but are not limited to, rule-based control, hybrid control, dynamic decision networks (DDN), and mixed integer model predictive control (MIMPC). These methods are known in the art and will not be described in detail herein. It should be noted that a given system may implement more than one method for monitoring, and the shipboard supervisory control system 103 and the remote monitoring system 104 need not implement the same method. System designers are free to choose from among these and other suitable methods when implementing embodiments of the present invention.

[0030] The vessels 101 and the remote control center 110 are equipped with communication capabilities that allow the vessels to communicate with the remote control center 110. In some embodiments, the vessels 101 may also be able to communicate directly with each other. While the communication capabilities are generalized in the figure as a network cloud 108, it should be understood that several different wireless communication standards may be utilized, alone or in combination. The vessels may, for example, wirelessly communicate directly with the remote control center 110, or may communicate via local or wide area access points, cellular (or mobile) networks, satellite radio links, or any other suitable method known in the art.

[0031] The interaction and cooperation between the vessels 101 and the remote control center 110 will now be described in more detail with reference to FIG. 2 , which is a block diagram illustrating how the overall system operates simultaneously in three different layers representing three levels of functionality. The top layer, also referred to as the fleet coordination layer 201, is implemented in the remote monitoring system 104. This layer controls the entire fleet and can monitor and assist individual autonomous vessels 101 based on information obtained from the entire fleet and external sources. Below the fleet coordination layer 201 is the vessel coordination layer 202. This layer is primarily implemented as part of the monitoring and control system 103 within each vessel 101. The vessel coordination layer 202 is responsible for mission planning and execution, self-diagnosis, and managing operational modes. The vessel coordination layer 202 receives instructions from the fleet coordination layer 201 and reports back to the fleet coordination layer 201.

[0032] The third layer is the vessel execution layer 203, which is primarily implemented as part of the autonomous system 102 on each vessel 101 in the fleet. The vessel execution layer includes functionality related to autonomous operations performed by the vessel, including sensors, interpretation of sensor data, and motion control based on instructions from the vessel coordination layer combined with situational awareness based on sensor data. The vessel execution layer 203 is in constant communication with the vessel coordination layer 202. Additionally, the vessel execution layer can provide sensor data directly to the fleet coordination layer 201.

[0033] In addition to these three layers, the system may include a remote control interface 204. The remote control interface 204 and associated functionality may be implemented on a computer 105 in the remote control center 110. The remote control interface 204 communicates with a mode switch 205 on each vessel 101. The operator 106 can use the remote control 204 to instruct the mode switch 205 on any vessel to change to remote control mode, in which case the vessel coordination layer 202 is instructed to suspend control over the vessel execution layer 203; instead, the vessel execution layer 203 receives control signals directly from the remote control 204.

[0034] The implementation of functions in the three layers described above provides flexibility and high security. The fleet coordination layer can receive information from all autonomous vessels 101 under its supervision and can also receive information from additional sources, such as weather services, environmental sensors not part of any vessel 101, and information about traffic and expected traffic (i.e., transportation demand). Thus, the fleet coordination layer 201 can optimize and direct operations based on information unavailable to individual vessels 101. The vessel coordination layer 202 can implement and report back instructions received from the fleet coordination layer 201. Thus, the vessel coordination layer 202 provides local monitoring functionality common to autonomous vessels and autonomous vehicles, but this layer also responds to external monitoring based on richer information. Similarly, the vessel execution layer 203 performs functions implemented in traditional autonomous vessels and also responds to and benefits from the collective control of a fleet of vessels, where all vessels contribute sensor data and information about their position and operational status.

[0035] The three layers will now be described in more detail. While each figure depicts an exemplary embodiment of a layer, it should be understood that the description should not be understood as a description of specific embodiments in which all features are interdependent and cannot be implemented without each other or in combination with additional features not shown in the figures. Instead, these examples should be understood as embodiments selected for the purpose of providing the reader with a thorough understanding of the invention, including the possibility of adding or deleting features. Thus, individual features are interdependent only to the extent that they provide input to or receive output from other features, that input or output is necessary to perform a particular task, or is otherwise explicitly stated to be dependent on the specific implementation of another feature. Any feature that one skilled in the art would understand could be removed without fundamentally disabling the system from operating according to its principles should be considered an optional feature, unless otherwise described as essential in this specification or the appended claims.

[0036] Reference is now made to Figure 3A, which is a more detailed illustration of vessel execution layer 203. This figure is a block diagram representing modules with associated functionality and their exchange of information and / or instructions. The figure shows only the information exchange within the layer; communication between layers is shown in Figure 2.

[0037] As described above, the vessel execution layer 203 is implemented as part of the autonomous system 102 on each individual vessel 101. Accordingly, the vessel execution layer includes several sensors 301. These sensors may include radar, lidar, RGB cameras, and IR cameras. Embodiments of the present invention may exclude one or more of these types of sensors or include additional types of sensors based on the needs of the system in a given use case context. The sensors are connected to a perception, planning, and execution module 302. The perception, planning, and execution module 302 may be implemented in a modular architecture or a more holistic end-to-end architecture. In embodiments implementing a modular architecture, the sensor module 301 provides input to a chain of self-contained modules that handle distinct tasks and provide data to each other. An example of a modular execution module 302 that includes an object detection module 3021 is shown in FIG. 3B , where individual pipelines of sensors are fed into the object detection module 3021 for object detection and for determining other environmental parameters (e.g., temperature, wave height, wind, current, etc.). While sensor data from individual sensors may be sufficient for object detection, the subsequent situational awareness module 3022 can perform sensor fusion, integrating the individual detection pipelines and performing target tracking and projection, as well as target classification and characteristic identification. The situational awareness established by the situational awareness module 3022 can be forwarded to the motion planning module 3023. The motion planning module 3023 can then make navigational decisions and perform path planning, collision avoidance, auto-berthing, and other high-level control functions. The higher-level instructions generated by the motion planning module 3023 can then be forwarded to the motion control module 3024, which has the necessary functions for controlling position and motion. The motion control module 3024 can be based on a dynamic positioning (DP) system with the necessary functions, such as stationkeeping, thrust allocation, autopilot, and manual joystick control. The motion control module 3024 provides outputs that control the actuator module 303.The actuators may be components such as engines, rudders, thrusters, servos, etc., to control the direction and speed of the vessel, or may physically control them. A modular architecture can be easily monitored and interpreted, and therefore easy to diagnose in case of failure or unexpected behavior. Causes can be localized to a specific module, and corrective actions, adjustments, or repairs can be performed. For this reason, a modular architecture may be advantageous in terms of warranty and certification, compared to an end-to-end architecture.

[0038] As shown in FIG. 3C , some embodiments of the present invention may implement an end-to-end approach instead of a modular approach. In an end-to-end embodiment, the perception, planning, and execution module 302 may be based on a neural network 3025, for example, that receives data from sensors 301 as input and provides output directly to the actuator module 303. The end-to-end architecture is realized by directly implementing the entire execution pipeline, from processing input from the sensors 301 to generating commands to the actuators 303. In neural network 3025 embodiments, machine learning is used to train the execution module 302 to respond to sensor inputs with actuator commands based on training data. This approach results in a simpler architecture, but undesirable behavior and malfunctions are more difficult to diagnose and more difficult to guarantee.

[0039] It will be appreciated that the signal generation, processing, and forwarding pipeline executed in the vessel execution layer 203 enables the autonomous vessel to perform basic autonomous operations based on short-term situational awareness of its surroundings. In the vessel coordination layer 202, shown in more detail in FIG. 4, decisions and actions are made in both the short-term and medium-term. This layer also handles more diverse sources of information. For example, in addition to vessel position, speed, heading, etc., the vessel coordination layer 202 can receive and utilize information regarding weather (including forecasts), traffic and expected traffic, the relative position of vessels in a convoy, etc. In this example, the vessel coordination layer 202 is shown as comprising four modules, all of which can exchange information with the others.

[0040] The first module is the mission planning and re-planning module 401. This module receives situational awareness data and other status information from the vessel execution layer 203, as well as status and diagnostic information from other modules in the vessel coordination layer. The mission planning and re-planning module 401 may be instructed to execute a mission, typically defined in general terms in a mission description by, for example, a definition of the origin and destination, the time frame in which the mission must be executed, and the route or channel along which the vessel is permitted to travel. Based on such parameters, the mission planning and re-planning module 401 maintains and, as necessary, updates a mission execution plan, which in some embodiments may be a local representation of the mission description supplemented with additional information. This execution plan can be used to generate instructions to the vessel execution layer 203 in substantially real time. In other embodiments, the mission execution plan is much more detailed and includes pre-planned instructions to the vessel execution layer 203. The mission planning and re-planning module 401 instructs the vessel execution layer 203, and in particular the maneuver planning module 304, with the updated mission parameters. During operation, based on altered situational awareness, changing operational modes, changing self-diagnostics or risk assessments, this module can re-plan mission parameters and update instructions to the vessel execution layer 203.

[0041] The second module of the vessel coordination layer is the operational mode management module 402. The operational mode management module 402 receives situational awareness, diagnostic information, and risk assessment as inputs and, based thereon, can make decisions regarding the state of operation of the vessel 101. The state may be one of several operational modes associated with performing a mission, such as at anchor or in transit, or it may be an exceptional state associated with a situation requiring special measures, such as an emergency. The emergency may be handled by entering a minimum risk state (MRC).

[0042] The mission planning and re-planning module 401 and the operational mode management module 402 may be implemented using methods such as rule-based control, hybrid control, reinforcement learning, or MIMPC, although the present invention is not limited to these methods.

[0043] The self-diagnostic module 403 may be configured to receive one or more of vessel system data, sensor data, situational awareness data, operational mode information, etc. and perform diagnostics based on this information. The output from the self-diagnostic module 403 may indicate that all systems are operating, or a deviation (e.g., unexpected information given operational conditions) may indicate that a component, subsystem, or the vessel 101 as a whole is malfunctioning or otherwise not performing as expected. The self-diagnostic module 403 may be implemented based on a deep neural network (DNN), a model-based observer, a discrete event system, or other method known in the art.

[0044] The online risk management module 404 is configured to receive one or more of situational awareness information, self-diagnostic data, environmental information, and other relevant data to perform risk assessment. If the risk management module 404 determines that the risk to the vessel 101 is too high, it can share this information with the operational mode management module 402 and / or the mission planning and re-planning module 401, which can decide to re-plan the mission or proceed to a minimal risk state. The risk management module 404 may be implemented using one or more techniques, such as Bayesian belief networks (BBNs), dynamic decision networks, fuzzy logic, and reinforcement learning. These techniques, per se, are well known in the art and will not be described in further detail herein.

[0045] As previously mentioned, modules within the ship coordination layer 202 can receive information from and issue instructions to the ship execution layer 203. Additionally, modules within the ship coordination layer 202 can report their data to and receive instructions from the fleet coordination layer 201. The ship coordination layer 202 can report, for example, risk assessments, as well as diagnostic and operational status, and can also provide planning information to the fleet coordination layer 201.

[0046] As described with reference to FIG. 2, if the vessel 101 is currently controlled by the remote control 204, it may receive instructions from the remote control interface 204 via the mode switch 205. The operational mode management module 402 may be configured to conclude that the vessel 101 needs to proceed to an operational mode in which it is remotely controlled by an autonomous function within the fleet coordination layer 201 or by a human operator 106 using the remote control interface 204. However, the operational mode management module 402 can only request the remote control mode and should not directly switch to the remote control mode. This is because if no operator 106 is available to immediately assume control or if the fleet coordination layer 201 cannot provide the requested guidance, the vessel 101 should not be able to transition to a mode that relies on unavailable input. In such a situation, the operational mode management module 402 should rather transition to another MRC that does not require external input.

[0047] The fleet coordination layer 201 is shown in more detail in FIG. 5. Like the vessel execution layer, this layer can be implemented using a modular architecture or a monolithic approach. With respect to the fleet coordination layer 201, the term monolithic means essentially the same thing as the term end-to-end in terms of technical implementation, but while an end-to-end approach implies a clear pipeline from sensor inputs to actuator outputs, the fleet coordination layer 201 may be more complex, with inputs that may also be influenced, at least to some extent, by outputs; for this reason, the term monolithic is preferred. An end-to-end approach may be understood as a special case of a monolithic approach.

[0048] FIG. 5A illustrates a modular approach in which this layer includes a fleet coordination and optimization module 501, a networked online risk management module 502, and an operational mode transition coordination module 503. This is the highest supervisory layer for a network of ships and may be configured to operate in collaboration with a human safety manager. Modules within this layer exchange information with each other and receive input from the lower layers 202 and 203, as described above. Additionally, modules within the fleet coordination layer may receive additional information regarding environmental and other conditions, such as weather, currents, and expected traffic. Instructions are issued to the vessel coordination layer 202 by modules within the fleet coordination layer 201. Communication between the fleet coordination layer 201 and the vessel execution layer 203 is primarily a matter of providing a flow of sensor data and other status information from the vessel execution layer 203 to the fleet coordination layer 203. Generally, the fleet coordination layer 201 does not issue commands directly to the vessel execution layer 203. However, it is not inconsistent with the principles of the present invention to include autonomous remote control functionality in the remote control center 110, in which case whether such functionality is part of the fleet coordination layer 201 becomes a semantic, not a technical, issue. Thus, nothing in the disclosure herein is intended to preclude or prohibit the direct issuance of instructions or commands from the fleet coordination layer 201 to the vessel execution layer 203. Instead, the key point is that the design contemplates establishing a system in which operational control of the vessel execution layer is primarily handled locally at each individual vessel 101, and fleet coordination and control is primarily handled centrally in the form of instructions given to local (on-board) implementations of the vessel coordination layer 202.

[0049] The fleet coordination and optimization module 501 provides functionality related to overall resource utilization of the fleet of vessels 101. Some embodiments of the present invention may not implement this module or may implement only a subset of possible features. This module may provide functionality related to logistics and capacity utilization, for example. Based on inputs of current traffic loads and estimates of expected traffic, the fleet coordination and optimization module 501 may optimize routes (to which destinations and which routes to take), departure and arrival times, vessel diversions from one route to another, and similar tasks. This planning may result in new or updated missions for individual vessels 101, and instructions to that effect may be communicated to the affected vessels, and the mission planning and re-planning module 401 must perform the necessary re-planning based on the received instructions. Additionally, the fleet coordination and optimization module 501 may be configured to direct several vessels to cooperate, for example, by instructing one vessel to assist another. The fleet coordination module may be implemented based on rules-based control, hybrid control, reinforcement learning, or MIMPC, for example.

[0050] The networked online risk management module 502 may be similar to and configured to communicate with the online risk management modules 404 implemented on each vessel 101. However, the networked online risk management module 502 does not focus on the individual risks of each individual vessel (although the module may be configured to perform such risk assessments as well). Instead, the focus is on aggregate information from the entire fleet and from other sources to identify risks that are not apparent to the online risk management modules 404 on the individual vessels 101. For example, aggregate information from the fleet regarding underwater currents or objects can reveal risks, and the significance of the identified risks to any given vessel can be assessed and shared with each vessel 101 based on, for example, location, mission, and individual capabilities and operational status. Like the online risk management modules 404 described above, the networked online risk management module 502 may be implemented using BBNs, dynamic decision networks, or fuzzy logic, as well as other suitable methods known in the art.

[0051] An operational mode coordination module 503 tracks the operational modes of all vessels 101 in the fleet based on data received from the ship coordination layer 202, typically the operational mode management module 402, of each vessel. The operational mode coordination module 503 may be configured to override mode decisions made by the autonomous systems on individual vessels, for example, based on external risks determined by the networked online risk management module 502 or based on fleet requirements determined by the fleet coordination and optimization module 501. Additionally, the coordination module 503 may be responsible for handling a set of general operational modes, such as normal autonomous operation, docked, on-site standby, manual control, remote control, and minimum risk conditions. Some of these modes may include sub-modes controlled by the on-board operational mode management module 402. For example, the normal autonomous operation mode may include docked, departed, and underway sub-modes. Additionally, the minimum risk condition may include different states based on different levels and nature of conditions that necessitated the minimum risk condition, as determined by the on-board sensors 301 and situational awareness 303. The operational mode adjustment module 503, like the operational mode management module on the vessel adjustment layer 202, may be implemented based on, for example, rule-based control, hybrid control, reinforcement learning, or MIMPC.

[0052] FIG. 5B illustrates a monolithic implementation of the fleet coordination layer 201. In embodiments in which the fleet coordination layer 201 is implemented in this manner, there is only one fleet coordination module 504, which includes a neural network trained using machine learning. As with the vessel execution layer, a single module trained with machine learning can make diagnosing undesirable behavior or malfunctions more difficult. The requirements at this layer may differ from the vessel execution layer, both in terms of the nature of the inputs and the criticality of the outputs. Thus, some embodiments may implement a modular approach in one of these layers and an end-to-end solution in the other.

[0053] 6 shows an example of a state diagram that may represent operational modes in which various embodiments of the present invention may be implemented. It will be understood that this is an example, and other possibilities are consistent with the principles of the present invention. In this exemplary embodiment, the operational mode coordination module 503 in the fleet coordination layer 201 is responsible for several higher-level modes, while the operational mode management module 402 in the ship coordination layer 202 handles specific modes during autonomous operation.

[0054] The first mode of operation is a general or default mode in which the vessel 101 is in a local standby mode 601. The local standby mode 601 may be a mode of operation in which systems are only operable to maintain a current state. For example, when at anchor, the vessel 101 may only activate systems that collect sensor data, establish situational awareness, and maintain communication with onshore systems. When entering this mode of operation while moving, thrusters may be engaged and controlled to maintain a current position.

[0055] From the on-site standby mode 601, the operational mode coordination module 503 can instruct the on-board operational mode management module 402 to enter normal autonomous operation 602. This mode can have several sub-modes that are directly controlled by the operational mode management module 402 and are typically not subject to intervention from the fleet coordination layer 201. These modes can include berthing 6021, departing 6022, en route 6023, preparing to berth 6024, and berthing 6025. In normal operation, the vessel 101 moves through these operational modes successively while servicing two or more locations. However, some embodiments can allow for a direct transition from departing 6022 to preparing to berth 6024 to allow for efficient abort of the departure, for example, in the event of some exception as determined by the situational awareness module 303 or directed by the fleet coordination layer 201.

[0056] In the illustrated embodiment, the operational mode may return from normal autonomous operation 602 to local waiting 601. If this occurs in any other operational submode other than berthing 6021, the vessel may use its thrusters to remain in its current position. The vessel 101 may also transition from normal autonomous operation mode 602 to manual control 603, in which an onboard operator controls navigation using an onboard control system to operate the vessel 101 like a normal, non-autonomous vessel. If the onboard operator relinquishes control, the operational mode returns to normal autonomous operation 602.

[0057] It should be noted that manual control cannot simply be commanded from the operational mode coordination module 503 because manual control mode depends on the actual presence of a human operator able and willing to assume control. Thus, embodiments may allow the operational mode coordination module 503 to request, but not implement, this transition. Some embodiments may allow an onboard operator to directly initiate this operational mode transition, while other embodiments may require approval from the operational mode coordination module 503 in the fleet coordination layer or a shore-based human operator 106.

[0058] From local standby mode 601, the operational mode can also transition to remote controlled operation 604, which is an operational mode in which a human operator 106 controls the vessel 101 using the remote control interface 204. Like manual control operational mode 603, remote controlled operational mode 604 can only be requested by the system and requires acceptance by a human operator.

[0059] The final operating mode in this example is a minimum-risk state 605. Because the autonomous vessel 101, such as a ship, is in motion and subject to both unavoidable and uncontrollable events, there may be no actual safe state for the system. The minimum-risk state 605 is an operating state in which external conditions as well as system states are considered and the system attempts to maintain a state in which all risks are minimized, or at least a state in which the trade-offs between various risks are optimized. This may be highly dependent on the current situation. For example, a minimum-risk state while at anchor may be very different from a minimum-risk state while in motion, and a minimum-risk state when there is bad weather but all systems are operational may be very different from a minimum-risk state when there is stable weather but the thrusters are not operational. As a result, the minimum-risk state 605, like normal autonomous operation 602, may include several different sub-modes. Whether transitions to the minimum-risk state mode 605 and between sub-modes within this state should be controlled by the operational mode adjustment module 503 or the operational mode management module 402 may vary between embodiments. In some embodiments, the system may implement redundancy to ensure that if the vessel 101 loses communication with the fleet coordination layer 201, it will proceed to the minimum risk condition mode 610 and can manage the minimum risk condition mode 610, while the operational mode coordination module 503 can take control and override if it has better information than the vessel coordination layer 202.

[0060] In the example of Figure 6, the minimum risk condition mode 605 can be entered from all other modes. This mode can be initiated by the operational mode adjustment module 503, by the on-board operational mode management module 402, by the operator 106 of the remote control 204, or by the on-board operator from the manual control mode 603.

[0061] The transition from normal autonomous operation mode 602 to remote control mode 604 is shown as passing through local standby mode 601. This is simply a design choice to avoid abrupt changes in motion or navigation parameters. The same may be done between normal autonomous operation 602 and manual control 603, but this example does not consider this to be the case because it can be assumed that on-board manual control is for a sudden emergency requiring immediate action. However, it should be understood that any embodiment of the present invention may be modified to include more or fewer operating modes than those shown in FIG. 6, and that transitions between modes may be subject to different state transitions than those shown.

Claims

1. A system for operating a plurality of autonomous vessels (101) configured to serve a plurality of locations under the supervision of a remote control center (110), comprising: a communication interface (108) configured to enable communication between the remote control center (110) and each of the autonomous vessels (101); a fleet coordination layer (201) implemented in a computer system associated with the remote control center (110), the fleet coordination layer including at least one module (502, 503; 504) configured to collect information regarding the operation of the autonomous vessel (101), perform a risk assessment based on the collected information, monitor the operation mode of the autonomous vessel (101), and issue an operation mode transition instruction to the autonomous vessel (101) based on the risk assessment; a vessel coordination layer (202) implemented on the plurality of autonomous vessels (101), the vessel coordination layer (202) including an operational mode management module (402) on each of the autonomous vessels (101) configured to control transitions between operational modes of the autonomous vessels (101) according to instructions received from the at least one module (503; 504) in the fleet coordination layer (201); a vessel execution layer (203) implemented on the plurality of autonomous vessels (101), the vessel execution layer including sensors (301), a perception, planning, and execution module (302), and actuators (303), the perception, planning, and execution module (302) configured to control the movement of the autonomous vessels (101) using the actuators (303) in accordance with a mission description, received sensor data, and operational mode commands from the operational mode management module (402); Equipped with At least one module in the vessel coordination layer (202) is configured to send information regarding an operational mode to the remote control center (110), and at least one module in the vessel execution layer (203) is configured to send information regarding sensor data to the remote control center (110).

2. 2. The system of claim 1, wherein the at least one module in the fleet coordination layer includes: a networked online risk management module (502) configured to collect information regarding the operation of the autonomous vessel (101) and perform a risk assessment based on the collected information; and an operational mode coordination module (503) configured to monitor the operational mode of the autonomous vessel (101), receive a risk assessment from the networked online risk management module (502), and issue an operational mode transition instruction to the autonomous vessel (101) based on the received risk assessment.

3. The fleet coordination layer (201) further comprises a fleet coordination and optimization module (501) configured to receive risk assessment information from the risk management module (502), information on the operation mode of each of the ships (101) from the ship coordination layer (202), and information on sensor data from the ship execution layer (203), plan optimization of the utilization of the autonomous ships (101), and issue mission descriptions to the autonomous ships (101) based on the planned optimization; 3. The system of claim 1 or 2, further comprising a mission planning and re-planning module (401) in the vessel coordination layer (202) configured to receive mission descriptions from the fleet coordination layer (201) and situational awareness information from the vessel execution layer (203), plan or re-plan mission execution based on the received information, and issue instructions to the perception, planning, and execution module (302).

4. The vessel coordination layer (202) further comprises a self-diagnosis module (403) configured to receive operational information from the vessel execution layer (203) and perform diagnosis based on the received information, and an online risk management module (404) configured to receive operational information from the vessel execution layer (203), diagnostic information from the self-diagnosis module (403), and risk assessment information from the fleet coordination layer (201) and perform risk assessment; 4. The system of claim 1, wherein the operational mode management module (402) is further configured to obtain diagnostic information from the self-diagnostic module (403) and risk assessment information from the online risk management module (404) when controlling transitions between operational modes.

5. 5. The system of claim 4, wherein the operational information from the vessel execution layer (203) includes information derived from data selected from the group consisting of sensor data, object detection data, situational awareness information, motion planning data, and motion control data.

6. 6. The system of claim 1, further comprising: a mode switch configured to receive user input instructing the remote control center (110) to transition to a remotely controlled operational mode (604); and a remote control interface (204) configured to receive user input provided directly to the vessel execution layer (203) of an autonomous vessel (101) to enable remote control of the autonomous vessel (101) by a human operator (106).

7. 7. The system of claim 1, wherein the operational modes in which the autonomous vessel (101) can operate are selected from the group consisting of: local standby (601), normal autonomous operation (602), anchored (6021), departing (6022), en route (6023), preparing to anchor (6024), anchored (6025), manual control (603), remote control (604), and minimal risk state (605).

8. The system of any one of claims 1 to 7, wherein at least one risk management module (404, 502) uses a method selected from the group consisting of influence diagrams, Bayesian belief networks, fuzzy logic, and reinforcement learning.

9. 1. A method for operating a plurality of autonomous vessels (101) configured to serve a plurality of locations under the supervision of a remote control center (110), comprising: The system includes a communication interface (108) configured to enable communication between the remote control center (110) and each of the autonomous vessels (101), a fleet coordination layer (201) implemented on a computer system associated with the remote control center (110), a vessel coordination layer (202) implemented on the plurality of autonomous vessels (101), and a vessel execution layer (203) implemented on the plurality of autonomous vessels (101); The method comprises: In a fleet coordination layer (201) implemented in the remote control center (110), generating and distributing a mission description to each of the autonomous vessels (101); receiving and collecting information regarding the operation of the autonomous vessel (101); Conducting a risk assessment based on the collected information; monitoring the operational mode of the autonomous vessel (101); issuing an operational mode transition command to the autonomous vessel (101) based on a risk assessment; In each of the autonomous vessels (101), In the ship adjustment layer (202), Controlling the transitions between operational modes of the autonomous vessel (101) according to instructions received from the remote control center (110); Sending information about the operating mode to said remote control center (110); In the ship execution layer (203), controlling the operation of the autonomous vessel (101) according to the received mission description, the received sensor data, and the operational mode instructions; Transmitting information about the sensor data to the remote control center A method comprising:

9. In the remote control center (110) in the fleet coordination layer (201), receiving information about the operational mode of each vessel (101) from the vessel coordination layer (202) and information about sensor data from the vessel execution layer (203); Planning an optimization of the utilization of the autonomous vessel (101); issuing a mission description to the autonomous vessel (101) based on the planned optimization; In each of the autonomous vessels (101) in the vessel coordination layer (202), receiving mission descriptions from the fleet coordination layer (201) and situational awareness information from the vessel execution layer (203); planning or re-planning mission execution based on the received information; Issue instructions to the ship execution layer (203) The method of claim 8 further comprising:

10. In the ship adjustment layer (202), receiving operational information from the vessel execution layer (203); generating diagnostic information based on the received information; receiving risk assessment information from the fleet coordination layer (201); and performing a risk assessment based on the received operational information, the received risk assessment information, and the generated diagnostic information to generate risk assessment information for the vessel. It further includes:

10. A method according to claim 8 or 9, wherein generated diagnostic information and generated risk assessment information about the vessel are used as inputs when controlling transitions between operational modes.

11. 11. The method of claim 10, wherein the operational information from the vessel execution layer (203) includes information derived from data selected from the group consisting of sensor data, object detection data, situational awareness information, motion planning data, and motion control data.

12. 12. The method of any one of claims 8 to 11, wherein the operational modes in which the autonomous vessel (101) is operable are selected from the group consisting of: local standby (601), normal autonomous operation (602), at anchor (6021), departing (6022), en route (6023), preparing to anchor (6024), anchored (6025), manually controlled (603), remotely controlled (604), and minimal risk state (605).

Images